Large file truncation and missing files in mirrored anonymous repository

[gss10282023]

Hi, I am using Anonymous GitHub/anonymous.4open.science to host an anonymized version of my repository:
https://anonymous.4open.science/r/daec9db402fadd75d74cc982e7d1cdb2aa2b17a3/README.md

I noticed that some large files appear to be truncated in the anonymous repository. In addition, the anonymous repository does not show all files from the papar_result_packages directory in the linked GitHub repository. Could you please help check whether this is caused by a file size limit, synchronization issue, or repository mirroring problem? I would like the anonymous version to correctly display all files in that directory, or at least receive clarification on which file size or folder limits apply. Thank you!

[tdurieux]

can you provide link to truncked files and missing files?

[gss10282023]

Hi, thanks for your response.
The original GitHub repository is:
https://github.com/gss10282023/benchmarkCount
The corresponding Anonymous GitHub repository is:
https://anonymous.4open.science/r/daec9db402fadd75d74cc982e7d1cdb2aa2b17a3/README.md
The issue is that in the Anonymous GitHub version, the paper_result_packages folder has a large number of missing files compared with the original GitHub repository. It seems that many files in this folder were not properly mirrored or displayed.
For example, in the Anonymous GitHub repository, some subfolders under paper_result_packages are missing entirely, including:
miniwob_case_bundle
appworld_case_bundle_openrouter
tau3_retail_case_bundle
In addition, some folders are present but appear to have missing files compared with the original GitHub repository, such as:
agentdojo_case_bundle_openrouter_gpt54_high
androidworld_both_agents_scored_cases_official_full100
Could you please help check why the files under paper_result_packages are missing or incomplete in the anonymous repository? Thank you!

[gss10282023]

Hi
I think I found the root cause of the missing folders/results.

The GitHub repository itself is complete. The problem seems to happen when anonymous.4open.science indexes the large parent directory paper_result_packages/.

I checked the repo again carefully:

• on GitHub, paper_result_packages/ contains 5 complete bundles;
• but on anonymous.4open.science, the directory API for paper_result_packages/ only returns .DS_Store, agentdojo, and androidworld;
• appworld, miniwob, and tau3 are missing entirely;
• even androidworld is incomplete there.

The key point is that the GitHub Trees API for the subtree paper_result_packages/ returns truncated=true when called with recursive=1. So anonymous_github is getting only a partial tree for that parent directory.

My understanding is that in GitHubStream.ts, when a subtree returns truncated=true, the current logic still accepts data.tree and pushes that partial result into the index, instead of splitting that subtree into smaller recursive requests. So “force update” just repeats the same truncated indexing result.

That would explain exactly what I am seeing:

• the parent directory is too large,
• GitHub truncates the recursive tree response,
• anonymous_github indexes the partial tree as if it were complete,
• so some bundles and files never enter the mirror index.

I think the fix should be:
if a recursive tree request returns truncated=true, do not treat that result as final for indexing. Instead, fall back to a finer-grained traversal:

1. fetch that subtree with recursive=false,
2. enumerate its direct child trees,
3. recursively fetch each child tree separately,
4. repeat until each subtree returns truncated=false.

A simpler summary is:
for truncated subtrees, split one level deeper instead of accepting the partial recursive response.

I also checked that each individual bundle under paper_result_packages/ does not seem to be truncated when fetched separately, so the issue looks specific to the oversized parent directory rather than the leaf bundles themselves.

[tdurieux]

fixed