Architectural Note: Operating on Git LFS Repositories with Thousands of Files

[bwalsh]

Architectural Note: Operating on Git LFS Repositories with Thousands of Files

1. Context and Problem Statement

In large projects, it’s common for a Git repository to track thousands to hundreds of thousands of files via Git LFS. Typical use cases:

• A research study with many samples (VCFs, BAMs, images, etc.)
• A data lake-ish repo where each commit adds more LFS pointers
• Monorepos that aggregate multiple datasets or experiments

In these cases, standard Git LFS introspection commands become painfully slow. A concrete example:

git lfs ls-files --json

On a repo with thousands of LFS pointers, this can take several minutes. That’s a non-starter for:

• Interactive CLI tools
• Editor/IDE integrations
• CI/CD steps that run frequently

This note describes architectural patterns to avoid global enumeration and keep operations fast and predictable as your LFS population grows.

---

2. Why git lfs ls-files is Slow in Large Repos

Conceptually, git lfs ls-files must:

1. Walk the Git index / working tree to identify LFS-tracked files.
2. For each file, resolve and hydrate metadata (pointer, OID, size, etc.).
3. Optionally serialize to JSON.

Even if the LFS objects are local, this is O(N) over every matching file visible to the command. When N = 10,000+, you’re essentially asking Git + Git LFS to do a full scan and re-derive information that:

• Doesn’t change very often, and
• Could be cached or maintained elsewhere.

From an architecture perspective, the problem is:

We’re using git lfs ls-files as a query engine and index, when it’s really just a dumb enumerator over the current state.

---

3. Design Goals

For a repository with many LFS objects, we want:

1. Predictable latency
   Operations that touch “all LFS files” should be rare and explicit; routine commands should be sub-second, even as the repo grows.

2. Incremental updates
   Avoid full scans of N files when only a handful are new or changed.

3. Subset operations by default
   Most tasks only need a subset (by path, tag, type, or commit range), not the full universe.

4. Separation of metadata from Git internals
   Use Git (and Git LFS) as the transport and integrity layer, not as a full-featured metadata store.

---

4. Core Architectural Pattern: External LFS Metadata Index

Instead of deriving everything on demand from git lfs ls-files, maintain a separate index of LFS metadata that is:

• Versioned alongside the repo (e.g., tracked TSV/JSON),
• Derived incrementally from Git/LFS events, and
• Fast to query (path lookup, OID lookup, tags, etc.).

4.1. Example: META/lfs_index.tsv

A simple pattern:

• Maintain a tracked file such as META/lfs_index.tsv with columns like:

  path    oid_sha256                             size    tags    logical_id
  data/a.bam  1a2b3c...                          12345   tumor   sample:XYZ
  data/b.bam  4d5e6f...                          67890   normal  sample:ABC
  

• This TSV becomes your primary, fast, queryable index, not git lfs ls-files.

Pros:

• Constant-time query by path via grep / awk / Python / SQL.
• Easy to join with other metadata tables (specimens, assays, etc.).
• Can be regenerated in a controlled, explicit operation (like make rebuild-index).

4.2. How to Keep It Up-to-Date

You don’t want manual edits. Use automation on “add” paths:

• use a pre-commit hook:

• For newly staged LFS pointer files, update the index before commit.

This shifts expensive work into the write path where it is amortized and expected, and keeps the read path (queries) fast.

---

5. Avoiding git lfs ls-files in Common Operations

5.1. Don’t use ls-files as your data plane

Refactor any tools that currently:

git lfs ls-files --json | jq ...

to instead read from your external index (TSV/JSON/SQLite). For example:

# Old, slow:
git lfs ls-files --json | jq '.[] | select(.name|test("VCF$"))'

# New, fast:
awk -F'\t' '$1 ~ /\.vcf$/ {print $0}' META/lfs_index.tsv

or in Python:

import csv

with open("META/lfs_index.tsv") as f:
    for row in csv.DictReader(f, delimiter="\t"):
        if row["path"].endswith(".vcf.gz"):
            ...

5.2. Use ls-files only for rare “rebuild index” operations

When you first introduce the index, you may need a one-time or occasional rebuild:

git lfs ls-files --all --json > /tmp/lfs_files.json
# transform into META/lfs_index.tsv

This can take minutes in huge repos—and that’s fine, as long as it is rare and documented as a heavy operation (like npm install, docker build, etc.).

---

6. Subset-First Design: Operate on Paths, Tags, or Commits

If you must derive state from Git directly, design your commands to start with a subset, not the full repo.

6.1. Path-based subsets

For example, instead of:

# Scans entire repo
git lfs ls-files --json

use:

# Only data under a project or cohort
git lfs ls-files --include "data/StudyX/**" --json

and structure your tooling around the concept of project subtrees (data/studyA/, data/studyB/, etc.) so most operations are scoped.

6.2. Commit-range subsets

For incremental workflows (ETL, indexing, sync), use git to find changed files:

git diff --name-only <old-commit> <new-commit> \
  | git check-attr --stdin filter \
  | awk '$2 == "lfs"' # or similar

Then only examine LFS metadata for changed files, merging that into your external index.

---

7. Caching and Incremental Computation

If you really want a “git lfs ls-files --json-like view,” you can implement your own cached snapshot:

1. Keep a file like .cache/lfs_snapshot.json keyed by commit hash (HEAD).

2. On invocation:

   • If HEAD has not changed, just read the cache.
   • If HEAD changed, compute the diff from the last snapshot and patch the cached JSON.

This means you only pay full-scan costs when the diff is large, and usually pay a small, incremental cost.

---

8. CI/CD Considerations

In CI, naive patterns like:

- run: git lfs ls-files --json | jq ...

will slow your builds significantly once the LFS population grows.

Better patterns:

• For linting or validation:

  • Operate on META/*.tsv and cross-check with a small sample of pointers.

• For publishing or sync steps:

  • Use git diff between the last deployed commit and current one to identify only the LFS files that changed.

• For health checks:

  • Schedule a periodic “heavy” job (nightly or weekly) that runs git lfs ls-files to verify repo consistency, rather than doing it on every push.

---

9. Git + LFS as Transport, Not Primary Index

The underlying architectural theme:

• Git is an excellent tool for content addressing, branching, merging, and history.
• Git LFS is an excellent tool for large object transport and storage.

Neither is optimized as a high-level metadata query system for tens of thousands of objects.

So:

• Let Git/LFS handle integrity and distribution.
• Let a simple, explicit index (TSV/JSON/SQLite, or an external service like Indexd) handle queries, tags, and relationships.

You can always rebuild your index from Git LFS if needed, but you shouldn’t be doing that implicitly on every command.

---

10. Practical Recommendations / Checklist

When you notice git lfs ls-files --json taking minutes:

1. Audit your tools

   • Search for any use of git lfs ls-files in scripts, CI configs, and CLIs.
   • Replace them with operations over an external index.

2. Introduce a canonical LFS index

   • Add META/lfs_index.tsv (or similar) to the repo.
   • Define columns: path, oid_sha256, size, tags, logical_id, etc.
   • Commit it and treat it as the primary query surface.

3. Automate index maintenance

   • Add a wrapper command or pre-commit hook that updates the index on git add.
   • Provide a “heavy” rebuild-lfs-index command that users run explicitly when necessary.

4. Scope operations by default

   • Design new commands to accept --path, --tag, --study, or --since <commit> flags.
   • Document that global “scan everything” commands are expensive and should be infrequent.

5. Use CI wisely

   • Only operate on changed LFS files between commits.
   • Reserve full LFS integrity checks for scheduled jobs, not every PR.

[bwalsh]

lfs-index-bundle.zip

[bwalsh]

https://source.ohsu.edu/CBDS/EVOTypes/pull/5

[bwalsh]

---

Using Git Sparse Checkout to Improve Git & Git LFS Performance with Thousands of Files

When your repo has thousands of files (especially large ones tracked by Git LFS), a full checkout can be painful:

• Slow git clone and git checkout
• Slow git status, git diff
• Gigantic working trees and local disk usage
• Unnecessary LFS downloads for files you don’t care about

Sparse checkout lets you tell Git:

“Only populate this subset of the tree in my working directory.”

Combined with Git LFS options, this can dramatically reduce time and I/O.

This doc covers:

1. What sparse checkout is (and what it’s not)
2. Basic sparse checkout workflow
3. How it interacts with Git LFS
4. Recommended patterns and pitfalls
5. Example recipes

---

1. What Sparse Checkout Does (and Doesn’t)

Sparse checkout:

• Keeps the full Git history in .git/ as usual
• Restricts which paths appear in your working directory
• Makes Git treat missing paths as “intentionally not present” (not untracked)

It does NOT:

• Reduce the size of the .git object database by itself
• Prevent LFS metadata (pointer files) from being fetched if the commit needs them

To reduce network + disk load for huge repositories, sparse checkout is often combined with:

• Git partial clone (--filter=blob:none)
• Git LFS smudge disabling (GIT_LFS_SKIP_SMUDGE=1)

But even by itself, sparse checkout can substantially improve working tree performance and make Git usable in huge repos.

---

2. Basic Sparse Checkout Workflow

2.1. Enable sparse checkout in a repo

From Git 2.25+ the modern CLI is:

# From inside the repo
git sparse-checkout init --cone

• --cone mode is simpler: you work in terms of directory roots, not arbitrary patterns.
• Git will create and manage .git/info/sparse-checkout for you.

2.2. Choose what you want to see

Say your repo structure looks like:

.
├── apps/
├── workflows/
├── data/
│   ├── raw/
│   └── processed/
└── docs/

You only care about:

• apps/
• workflows/
• docs/

Run:

git sparse-checkout set apps workflows docs

Now your working directory will only contain those paths. Everything else (e.g. data/) stays in Git history but doesn’t exist in the working tree.

You can add more paths later:

git sparse-checkout add data/processed

Or list what’s currently included:

git sparse-checkout list

Disable sparse checkout and go back to a full tree:

git sparse-checkout disable

---

3. Sparse Checkout + Git LFS: How They Interact

For Git LFS, there are two key behaviors:

1. Smudge filter: When you checkout a file tracked by LFS, the pointer is replaced with the real large file (downloaded if necessary).
2. Fetch behavior: git lfs fetch / git lfs pull control which LFS objects are downloaded.

Sparse checkout helps by ensuring you never even checkout most of the LFS-tracked paths, so the smudge filter isn’t triggered for those paths.

To get the most out of this:

3.1. Clone with smudge disabled (recommended)

Avoid downloading any large LFS objects during the initial clone:

GIT_LFS_SKIP_SMUDGE=1 git clone <url> my-repo
cd my-repo

At this point:

• You have the full Git history + LFS pointers.
• No large LFS blobs have been downloaded yet.

3.2. Enable sparse checkout early

Before you touch branches or run builds:

git sparse-checkout init --cone
git sparse-checkout set apps workflows docs

Now your working tree only exposes those paths.

3.3. Selectively fetch & checkout LFS files

When you’re ready to actually use the large files in your sparse set:

# Fetch only the LFS objects for paths in your sparse checkout
git lfs fetch --include="apps/**,workflows/**,docs/**" --exclude=""

# Materialize those LFS objects in the working tree
git lfs checkout

The combination of:

• GIT_LFS_SKIP_SMUDGE=1
• git sparse-checkout set ...
• git lfs fetch --include=...
• git lfs checkout

means:

Only the LFS files under the directories you care about get downloaded and expanded.

If you later extend your sparse checkout to e.g. data/processed/:

git sparse-checkout add data/processed

# Fetch and checkout only what you just added
git lfs fetch --include="data/processed/**" --exclude=""
git lfs checkout data/processed

---

4. Recommended Patterns

4.1. Organize large files into well-defined directories

Sparse checkout + LFS works best if your repo is structured like:

data/
  raw/
  processed/
  qc/
results/
  cohort-a/
  cohort-b/

Then you can say:

git sparse-checkout set src workflows data/processed
git lfs fetch --include="data/processed/**"
git lfs checkout data/processed

Instead of juggling 1000s of path patterns.

---

4.2. Use --cone mode unless you really need patterns

--cone mode works with directory prefixes. It’s simpler and faster:

git sparse-checkout init --cone
git sparse-checkout set src tests docs

If you must do pattern-based inclusion (e.g. only .vcf.gz), you can turn off cone mode and edit .git/info/sparse-checkout directly, but that’s more complex and easier to break.

---

4.3. Use git status --sparse

When using sparse checkout, status can surprise you if some files are “outside” your sparse set.

Git provides a special view:

git status --sparse

This avoids confusing output about paths outside your sparse checkout.

---

4.4. Avoid “global” commands that imply the whole tree

Be careful with commands like:

• git add .
• git grep -n PATTERN
• Build scripts that assume the entire repo is checked out

They might behave differently in a sparse checkout. Prefer:

• git add src/ or git add path/file.ext
• git grep PATTERN src/ workflows/
• Build scripts that refer to specific directories

---

4.5. Combine with partial clone if the repo is truly huge

For extraordinarily big repos, you can reduce the initial object download with partial clone:

GIT_LFS_SKIP_SMUDGE=1 \
git clone --filter=blob:none <url> my-repo

cd my-repo
git sparse-checkout init --cone
git sparse-checkout set src workflows

Then selectively fetch:

# Normal Git blobs for your sparse tree
git fetch origin main
git checkout main

# LFS blobs for specific paths
git lfs fetch --include="src/**,workflows/**" --exclude=""
git lfs checkout

Partial clone + sparse checkout + LFS control can make “monorepo + huge datasets” survivable.

---

5. Example Workflows

5.1. “Developer” workflow (code only, no big data)

Goal: work on app code and workflows; ignore giant data/ trees.

# 1) Clone without auto-downloading LFS objects
GIT_LFS_SKIP_SMUDGE=1 git clone git@github.com:org/big-repo.git
cd big-repo

# 2) Enable sparse checkout in cone mode
git sparse-checkout init --cone

# 3) Only bring in code + config
git sparse-checkout set apps workflows helm charts docs

# 4) Optionally fetch any LFS objects in those paths (if needed)
git lfs fetch --include="apps/**,workflows/**" --exclude=""
git lfs checkout

Result: Developers never even see the huge raw data; Git stays fast.

---

5.2. “Analyst” workflow (just one cohort’s data)

Goal: work only on results/cohort-a/ without touching 20 other cohorts.

GIT_LFS_SKIP_SMUDGE=1 git clone git@github.com:org/big-repo.git
cd big-repo

git sparse-checkout init --cone
git sparse-checkout set results/cohort-a analysis scripts

git lfs fetch --include="results/cohort-a/**" --exclude=""
git lfs checkout results/cohort-a

If they later need cohort-b:

git sparse-checkout add results/cohort-b
git lfs fetch --include="results/cohort-b/**" --exclude=""
git lfs checkout results/cohort-b

---

5.3. Switching “views” without recloning

You can change what you see after the fact:

# Currently working on apps + workflows
git sparse-checkout set apps workflows

# Switch to data work
git sparse-checkout set data/raw data/processed

# Fetch and checkout only those LFS objects
git lfs fetch --include="data/raw/**,data/processed/**" --exclude=""
git lfs checkout data/raw data/processed

The underlying full repo history stays; you just change which slice is visible.

---

6. Pitfalls and Gotchas

• Old Git versions: pre-2.25 Git uses a different sparse-checkout interface. Use a recent Git if possible.
• CI runners: Sparse checkout is a per-clone setting. If CI expects the entire tree, don’t enable sparse checkout there unless you control the job definitions.
• Tooling that assumes a full tree: Some linters/build systems crawl from repo root; you may need to configure them to avoid paths that aren’t in your sparse set.
• LFS and “missing” files: If you see LFS pointer content instead of the real file, it usually means you’ve fetched the commit but not the object. git lfs fetch + git lfs checkout for the right paths fixes this.

[bwalsh]

@matthewpeterkort

Rewinding my comment on ls-files slow 🐌 🐢.

The fact is - it takes many minutes on Jordan's monorepo use case

I said - it recomputes OID, after re-scanning the source

I don't think it does.

Short version: git lfs ls-files never recomputes the SHA-256 of the real data object; it just parses the oid sha256:… out of the LFS pointer text. There isn’t a code path in ls-files that reads the large file and re-hashes it.

Here’s how it’s wired up in the source.

---

1. Where git lfs ls-files gets its OID from

The ls-files subcommand is implemented in commands/command_ls_files.go in the commands package. The core bits are:

1. It creates a GitScanner:

   gitscanner := lfs.NewGitScanner(cfg, callback)

2. For the “normal” case (no --all), it scans the index:

   if err := gitscanner.ScanIndex(ref, "", nil); err != nil {
       // ...
   }

   The stack trace in #3431 shows exactly this path:

   • commands.lsFilesCommand
   • lfs.(*GitScanner).ScanIndex
   • lfs.scanIndex in lfs/gitscanner_index.go ([GitHub][2])

3. The GitScanner has a pointer callback you register (something like):

   gitscanner.AddPointerCallback(func(p *lfs.WrappedPointer, err error) {
       // p.Oid, p.Name, etc.
   })

   ls-files formats its output purely from that WrappedPointer data – it does not go back to the underlying large file.

How the pointer (and thus OID) is produced

ScanIndex, ScanTree, and friends funnel through the pointer-decoding logic in lfs/pointer.go:

• Functions like DecodePointer, DecodeFrom, DecodePointerFromFile are used throughout the codebase (including scanners and checkout) to recognize and parse pointer blobs. ([GitHub][3])

From the pointer-decoding description:

• DecodeFrom(reader):

  • Reads up to a small cutoff (1KB) from the blob.
  • Detects the LFS pointer magic (version https://git-lfs.github.com/spec/v1).
  • Parses key value lines (version, oid, size, any ext-* lines).
  • Validates that oid has the sha256:… form. ([CSDN Blog][4])

Crucially: in this path it does not read or hash the big binary object that the pointer refers to; it just parses text.

So for git lfs ls-files, the OID comes from:

• The pointer text stored in Git blobs or the index, parsed by DecodePointer and friends, and placed into WrappedPointer.Oid.
• ls-files then prints that Oid (often abbreviated) along with the path.

There is no SHA-256 of the actual large file involved in this command.

---

So why is it so very, very slow? 🐢🐌

🙀 Short answer: it’s slow because of Git metadata and walking

Short answer: it’s slow because of Git metadata and walking, not because it’s re-hashing 5 GB blobs.

git lfs ls-files is basically:

1. Ask Git to enumerate all LFS pointers (via index/tree/log rev-list plumbing, depending on flags).

2. For each pointer:

   • Read the blob contents (the tiny pointer file, not the 5 GB data).
   • Parse the ~200 bytes of text.
   • Call your callback (format + print JSON, etc.).

On a repo with thousands (or hundreds of thousands) of LFS files, that means:

• A lot of Git-object lookups.
• A lot of blob reads and pointer parses.
• Possible history walks (with --all, --committed, etc.).
• A ton of string/JSON formatting and I/O.

Even though each pointer is small, doing it tens or hundreds of thousands of times is expensive.

---

1. Git walking and blob IO, not data IO

Even “just” reading pointer blobs has overhead:

• Git has to:

  • Look up each blob by OID.
  • Decompress it (zlib).
  • Hand that buffer to Git LFS.

You don’t pay the 5 GB IO cost per object, but you do pay:

• zlib decompression
• pointer parsing
• allocations for every WrappedPointer
• output formatting

On a repo with, say, > 1K LFS-tracked files, on network attached storage, this is noticeable.

Flags that make it worse

• git lfs ls-files --all
  Walks all refs, not just HEAD/index. That means:

  • Git must traverse a lot of commits.
  • Many blobs are visited.
  • Even with dedup, it’s still more work.

• git lfs ls-files --full or --name-only + extra scripting

  ⛔️☢️😭
  Sometimes people call ls-files in loops or via tooling, so the total cost is multiplied by user scripts.

---

2. JSON output and stdout becoming the bottleneck

If you use:

git lfs ls-files --json

the code:

• Creates structured LsFilesResult structs.
• Encodes them via encoding/json.
• Writes a big JSON array of all entries.

The overhead of:

• Allocating large in-memory slices.
• Marshaling to JSON.
• Writing megabytes of JSON to stdout.

…is all CPU-bound, and grows linearly with number of LFS pointers. When you have thousands of 5 GB files, you don’t pay for the size, but you do pay for the count.

If you’re piping this into another tool (jq, Python script, etc.), that downstream process can also slow the whole thing by backpressuring stdout.

---

3. Repo size/shape and packfiles

On big, long-lived repos:

• Many LFS pointers are stored in packfiles.

• Reading scattered blobs from packs can cause:

  • Extra seeks.
  • Poor cache locality.
  • Decompression of large pack chunks that only contain a few relevant entries for this command.

So even though each pointer is tiny, getting them out of the packfiles is not free, and scales with:

• Number of pointers.
• Fragmentation / history shape.

---

4. History-heavy modes (if you’re using them)

If you ever call:

• git lfs ls-files --all
• git lfs ls-files --recent
• or anything wrapped in tooling that scans multiple refs or commits,

then the cost is dominated by:

• git rev-list-style history traversal
• per-commit tree walks
• deduplicating pointers across revisions

That’s essentially an index of all LFS pointers across history, done on the fly.

---

5. Why it feels like it’s tied to file size

Even though LFS isn’t re-hashing the 5 GB files, big repos with many huge files often correlate with:

• Many pointers (the real cause).
• Deep history and lots of branches.
• Heavier use of JSON tooling and scripts around LFS.
• Slower disks / networked storage where Git object IO is slower.

So “many 5 GB files” is usually a proxy for “many LFS objects & a complicated history,” which is what ls-files has to grind through.

---

6. Practical ways to speed this up

Here are some tactics you can actually use:

A. Scope the command

Instead of:

git lfs ls-files --all

prefer:

## Only HEAD + index
git lfs ls-files

Or limit by path:

git lfs ls-files "data/study-1234/"

Many tools wrap ls-files without scoping; scoping to a directory or ref can massively cut runtime.

---

B. Avoid JSON when you don’t need it

If you only need the filenames, this:

git lfs ls-files | awk '{print $3}'

is usually cheaper than:

git lfs ls-files --json | jq '.[].name'

Because JSON encoding and decoding is heavy. If you must use JSON, consider post-processing a saved file instead of piping in real time.

---

C. Cache results externally

For really big repos (your world):

• Treat git lfs ls-files as a slow index generation step.
• Run it once and cache the result in:
  • .drs/lfs/objects 👈👈👈👈
  • A local DB (SQLite).
  • A file keyed by HEAD or commit hash.
• Only re-run when HEAD changes, or only for changed paths.

This is essentially the “lfs-index” idea you’ve already started exploring: maintain your own index keyed by commit / path / oid instead of letting ls-files recompute a view of the world each time.

---

D. Use sparse checkout (reduce Git’s universe)

For repos with many directories you don’t care about:

1. Enable sparse checkout:

   git sparse-checkout init --cone
   git sparse-checkout set path/you/care/about

2. Now git lfs ls-files only sees LFS pointers in the sparse set, reducing the work Git and LFS must do.

This can dramatically cut down the pointer set in large monorepos.

---

E. Profile what’s actually happening on your machine

Since you’re comfortable with Go & tooling, you can:

• Wrap git lfs ls-files with:

  GIT_TRACE=1 GIT_CURL_VERBOSE=1 git lfs ls-files ...

  to see if it’s unexpectedly touching remotes (it shouldn’t, but sometimes config surprises you).

• Use strace / dtruss / fs_usage (macOS) to see:

  • Heavy reads of .git/objects/pack/*
  • Thundering herd of small reads/writes

• Measure time with and without --json to see how much is encoding vs scanning.

---

7. TL;DR in your terms

So:

• No, git lfs ls-files is not re-hashing those 5 GB blobs.

• It is:

  • Asking Git to enumerate a huge number of pointer blobs.
  • Decompressing and parsing them all.
  • Potentially walking a lot of history and refs.
  • Possibly JSON-encoding megabytes of metadata.

On a big, LFS-heavy monorepo, that’s enough to make it feel glacial.

If you’d like, I can help you sketch a concrete lfs-index design (in Go) that:

• Tracks pointers incrementally per commit.
• Exposes fast queries like “list LFS files for this path / ref.”
• Completely bypasses git lfs ls-files for your automation.

Let’s layer NFS on top of what’s already happening.

Right, so git lfs ls-files is already pretty chatty in terms of Git object reads — and then you put .git and .git/lfs on NFS and everything gets a 10× latency multiplier.

---

1. What git lfs ls-files actually does (quick recap)

For each LFS-tracked file it does all of this via the Git object store:

1. Enumerates files / index entries / trees (depending on flags).

2. For each candidate blob:

   • Look up the blob in .git/objects (often via packfiles).
   • Decompress a small pointer blob.
   • Parse version/oid/size lines.

3. Marshal output (plain text or JSON) and write it to stdout.

This can mean tens of thousands of:

• Object lookups
• zlib decompressions
• Small reads + small writes

Even on a fast local SSD, that’s noticeable at scale.

---

2. What NFS does to that workload

Now imagine all of that, but:

• .git/objects and .git/lfs live on an NFS mount.

• You have:

  • High latency per I/O.
  • Extra metadata RPCs (lookups, getattr, etc.).
  • Possibly multiple clients sharing the same repo.

Concrete pain points:

a. Latency dominates small Git object reads

Git does a ton of small reads:

• Find packfile
• Read pack index
• Seek into pack
• Read object
• Decompress

On local disk, hundreds of microseconds is fine.
On NFS, each step may involve:

• LOOKUP, GETATTR, READ RPCs
• Round-trip latency to the server

Multiply that by, say, 50k pointer blobs, and it’s very easy to end up in “minutes” territory, even though each pointer is only a couple hundred bytes.

b. Metadata calls explode

Git’s access pattern hits:

• .git/objects/info/alternates
• .git/objects/pack/*.idx
• .git/objects/pack/*.pack
• .git/index
• Different directories for loose objects if you have any

Each of these can trigger:

• GETATTR calls
• Directory re-reads
• NFS attribute cache expiration/revalidation

If your NFS mount has conservative attribute caching (e.g., low actimeo), you get even more RPCs.

c. Server load and contention

If multiple users/CI jobs are:

• Running git status, git log, git lfs ls-files, etc.
• All hammering the same NFS-backed .git/objects

Then the NFS server:

• Becomes CPU bound on:

  • zlib decompress for packfiles (if it’s doing dedup/compression)
  • encryption if using Kerberos/SSL

• Experiences IOPS and queue buildup

Your git lfs ls-files might be slow partly because everyone else is also talking to the same server.

d. NFS caching and “random” access

Git’s object access pattern for pointer blobs in packfiles is not sequential streaming:

• It seeks all over the packfile.
• The kernel’s readahead and NFS’s page cache can’t predict this well.
• You don’t get the full benefit of large rsize/wsize; many small-ish reads still incur NFS overhead.

So even though pointer blobs are tiny, accessing 50k of them scattered in a big packfile via NFS is brutal.

---

3. Why big files seem to make it worse on NFS

Even though ls-files isn’t reading your 5 GB blobs:

• Repos with many ≥5GB files tend to:

  • Have lots of LFS pointers.
  • Have long histories and many packfiles.
  • Live on shared NFS (HPC, institutional storage, etc.).

• The .git/lfs/objects tree can be huge; any incidental access to it (e.g., some commands verifying presence) hits NFS metadata again.

So the size doesn’t hurt ls-files directly, but the repository shape and infra realities absolutely do.

---

4. What git lfs ls-files looks like step-by-step on NFS

Roughly:

1. Git LFS asks Git for relevant blobs (index, HEAD, refs, etc.).

2. For each:

   • git cat-file (or similar plumbing) is invoked:

     • NFS LOOKUP / GETATTR for packindex.
     • NFS READ from packindex.
     • NFS READ from packfile at some offset.
     • Local decompression.

3. LFS parses the pointer.

4. Output is written — likely back to a terminal that isn’t on NFS, but Git’s internal temp files and .git/index.lock etc. are on NFS.

If any part of that path is slow or contended, your command suffers.

---

5. Practical mitigations specifically for NFS

Here’s where you can claw back performance in a realistic lab / HPC / shared storage environment.

A. Keep .git local, use NFS for the working tree (ideal)

Best case:

• Clone repo to local SSD on the node.

• Use NFS only for:

  • data directory
  • or final results

• Let Git and .git/lfs/objects live on local disk.

Variants:

• Use a local “scratch” clone that mirrors the NFS repo and push/pull between them.

• For automation: CI job that:

  • git clone to local ephemeral storage.
  • Runs git lfs ls-files locally.
  • Writes only the results back to NFS.

B. If .git must be on NFS: move LFS storage local

You can configure LFS storage to a local path on each node:

### .lfsconfig or git config
[lfs]
    storage = /local-ssd/lfs-storage

Then:

• Git objects (pointer blobs) may still be on NFS.
• But .git/lfs/objects (big binary files) live locally.
• Future commands that need to touch the data will be faster, and you avoid NFS for that part.

This doesn’t fully fix ls-files (since it still reads pointer blobs from .git/objects), but it can reduce some incidental NFS churn.

C. Aggressive scoping + caching

Given NFS reality, treat ls-files as a slow indexer and:

1. Run it less often and more narrowly:

   • git lfs ls-files path/you/care/about
   • Avoid --all unless absolutely needed.

2. Cache results (per commit) in a local DB:

   • Example key: (repo_id, commit_sha, path_prefix) → list of pointers.
   • Stored in SQLite on local disk.

3. For higher-level tools, query your cache instead of re-running ls-files across the whole repo.

This reduces repeated NFS-heavy traversals.

D. Tune NFS mount options (if you control them)

Things to check with your storage / infra folks:

• rsize/wsize: increase to allow bigger I/O chunks.

• noatime: avoids extra writes/metadata updates.

• attribute caching (actimeo):

  • If set too low, you constantly revalidate attributes.
  • For a mostly read-only .git, you can afford more aggressive caching.

• NFSv4 vs v3:

  • v4 gives better semantics in some cases (locks, delegations) but can have its own performance quirks.

You don’t want to break consistency, but for a repo that isn’t being mutated constantly by many writers, you can often lean into caching.

E. Local “object cache” / mirror strategy

If IT policy says “the repo lives on NFS”, you can still:

• Create a local bare clone that mirrors the NFS repo’s remote.

• Configure your working clones to use:

  • GIT_ALTERNATE_OBJECT_DIRECTORIES
  • or a shared object directory on local fast storage

• Periodically sync the local bare clone from the main remote.

Then Git’s object lookups for ls-files can hit the local object directory instead of NFS, while your working tree and any “official” clone still live on NFS.

---

6. How this influences your lfs-index idea

If you build a custom lfs-index service or tool:

• Make sure it runs on nodes with local storage, not directly on NFS.

• Have it:

  • Pull/fetch from the “real” remote.
  • Maintain its own fast, local Git object store.
  • Answer queries about LFS pointers via an API or cached DB.

Then your day-to-day tooling never has to run git lfs ls-files over NFS at human-interactive times; it just hits the index.

---

TL;DR

• git lfs ls-files is slow in big repos because it inputs “lots of pointer blobs via Git,” not because it reads multi-GB objects.

• On NFS, every small Git object read becomes a high-latency network round trip with extra metadata RPCs.

• Thousands of such calls + packfile seeking + zlib + JSON/IO = very slow.

• The core fixes are:

  • Keep .git (and ideally .git/lfs) on local disk where possible.
  • Scope and cache ls-files results.
  • If stuck with NFS, tune mounts and add a local indexing/mirroring strategy.

[matthewpeterkort]

lots of fun details here. We have implemented git-drs ls-files in #232 and it does not do this local I/O that you have mentioned.