git walker includes untracked files

[brycied00d]

Describe the bug

The "git" walker will walk - and thus include and scan/format - files NOT known to git, e.g. untracked files. This seems quite counterintuitive to me, I expect a walker named "git" to only include files known to git (committed or staged). If I needed treefmt to walk files regardless of their relation to git, that seems to be what the "filesystem" walker is intended for, or I would explicitly run treefmt path/to/file.

This behaviour was introduced in #557 / #560 and even announced as a "feature" in v2.2.0.

In #557, the user described adding a file to the git index (via git add) but not yet committed ("staged" in Git terms), and opined that the file they had just "git add"ed was not included in treefmt via git ls-files. I'm unsure what the user's original issue was because I'm unable to reproduce their described behaviour they experienced; in short, a staged file - regardless of if it's ever been committed - DOES already appear in the default and original git ls-files output. I even tested with treefmt 2.1.1, the release prior to the inclusion of #560 and confirmed that the staged file IS processed by treefmt as #557 asserted it should have been [but wasn't].

brycec:/tmp/porcupines $ /tmp/treefmt --version
treefmt v2.1.1$
brycec:/tmp/porcupines $ /tmp/treefmt -vv
2025/06/06 11:01:58 INFO using config file: /tmp/porcupines/treefmt.toml
DEBU walk | git: processing file: /tmp/porcupines/.gitignore
DEBU walk | git: processing file: /tmp/porcupines/Committed
DEBU walk | git: processing file: /tmp/porcupines/Staged
DEBU walk | cache: read 3 files from delegate
DEBU formatter | mylanguage: match: /tmp/porcupines/.gitignore
DEBU formatters signature for key mylanguage: 3cdb31578adf1058d99e43f48d2083f8
DEBU formatter | mylanguage: match: /tmp/porcupines/Committed
DEBU formatter | mylanguage: match: /tmp/porcupines/Staged
traversed 3 files
emitted 3 files for processing
formatted 0 files (0 changed) in 11ms

In addition, the addition of --others in #560 actually caused the regression reported by #565. --exclude-standard would not be needed if --others was not passed to git ls-files

Here is an example of git ls-files that confirms that staged (git add) files are included in the output by default, and that .gitignore files are also excluded from the output by default.

# Create a simple git repo with some files in varying Git states
brycec:~ $ mkdir /tmp/porcupines
brycec:/tmp $ cd /tmp/porcupines
brycec:/tmp/porcupines $ git init
Initialized empty Git repository in /tmp/porcupines/.git/
brycec:/tmp/porcupines $ touch Untracked Staged Committed Ignored
brycec:/tmp/porcupines $ git add Committed
brycec:/tmp/porcupines $ git commit -m "Committed a file"
[main (root-commit) 1f5856f] Committed a file
 1 file changed, 0 insertions(+), 0 deletions(-)
 create mode 100644 Committed
brycec:/tmp/porcupines $ echo IgnoredFile > .gitignore
brycec:/tmp/porcupines $ git add .gitignore
brycec:/tmp/porcupines $ git add Staged
# As of v2.1.1
brycec:/tmp/porcupines $ git ls-files
.gitignore
Committed
Staged
# As of #560 / 9662ee657c9e6c52f1758b13da98ad2391304c48, v2.2.0
brycec:/tmp/porcupines $ git ls-files --cached --others
IgnoredFile
Untracked
.gitignore
Committed
Staged
# As of #565 / cca79a8385c339b492e00fea6125a7056758b5db, v2.2.1
brycec:/tmp/porcupines $ git ls-files --cached --others --exclude-standard
Untracked
.gitignore
Committed
Staged

To Reproduce

Steps to reproduce the behavior:

1. Start with the above sample git repository.
2. And the following treefmt.toml

   [formatter.mylanguage]
   command = "echo"
   options = []
   includes = [ "*" ]
   excludes = []
   priority = 0

3. treefmt -vv
4. Observe that treefmt ran the "mylanguage" formatter against untracked files Untracked, treefmt.toml

   brycec:/tmp/porcupines $ /tmp/treefmt --version
   treefmt v2.3.1$
   brycec:/tmp/porcupines $ /tmp/treefmt --init
   Generated treefmt.toml. Now it's your turn to edit it.
   brycec:/tmp/porcupines $ grep -E '^[^# ]' treefmt.toml
   walk = "git"
   [formatter.mylanguage]
   command = "touch"
   options = []
   includes = [ "*" ]
   excludes = []
   priority = 0
   brycec:/tmp/porcupines $ /tmp/treefmt -vv
   INFO config: no tree root specified
   INFO config: attempting to resolve tree root using git: git rev-parse --show-toplevel
   DEBU tree-root-cmd | stdout: /tmp/porcupines
   
   INFO config: tree root: /tmp/porcupines
   DEBU Resolving path '/tmp/porcupines': git
   DEBU walk | git: processing file: /tmp/porcupines/Untracked
   DEBU walk | git: processing file: /tmp/porcupines/treefmt.toml
   DEBU walk | git: processing file: /tmp/porcupines/.gitignore
   DEBU walk | git: processing file: /tmp/porcupines/Committed
   DEBU walk | git: processing file: /tmp/porcupines/Staged
   DEBU walk | cache: read 5 files from delegate
   DEBU read 5 files
   DEBU formatter | mylanguage: match: /tmp/porcupines/Untracked
   DEBU formatters signature for key mylanguage: 3cdb31578adf1058d99e43f48d2083f8
   DEBU formatter | mylanguage: match: /tmp/porcupines/treefmt.toml
   DEBU formatter | mylanguage: match: /tmp/porcupines/.gitignore
   DEBU formatter | mylanguage: match: /tmp/porcupines/Committed
   DEBU formatter | mylanguage: match: /tmp/porcupines/Staged
   DEBU formatter | mylanguage: executing: /usr/bin/touch Untracked treefmt.toml .gitignore Committed Staged
   INFO formatter | mylanguage: 5 file(s) processed in 1.55886ms
   DEBU file has changed path=Untracked prev_size=0 prev_mod_time="2025-06-06 10:18:35 -0700 PDT" current_size=0 current_mod_time="2025-06-06 10:50:59 -0700 PDT"
   DEBU file has changed path=treefmt.toml prev_size=2196 prev_mod_time="2025-06-06 10:50:57 -0700 PDT" current_size=2196 current_mod_time="2025-06-06 10:50:59 -0700 PDT"
   DEBU file has changed path=.gitignore prev_size=12 prev_mod_time="2025-06-06 10:23:13 -0700 PDT" current_size=12 current_mod_time="2025-06-06 10:50:59 -0700 PDT"
   DEBU file has changed path=Committed prev_size=0 prev_mod_time="2025-06-06 10:18:43 -0700 PDT" current_size=0 current_mod_time="2025-06-06 10:50:59 -0700 PDT"
   DEBU file has changed path=Staged prev_size=0 prev_mod_time="2025-06-06 10:18:41 -0700 PDT" current_size=0 current_mod_time="2025-06-06 10:50:59 -0700 PDT"
   traversed 5 files
   emitted 5 files for processing
   formatted 5 files (5 changed) in 16ms
   DEBU no more files to read
   

Expected behavior

My expectation is that, with walk = "git", only the files included in git ls-files (.gitignore, Committed, and Staged in the above example) will be processed; files Untracked and treefmt.toml would NOT be considered by treefmt. Only "3" files should have been processed/traversed/formatted above.

I propose that the fix is to revert to using git ls-files without any further switches, and documentation that clearly states that the git walker will only include files known to git, including staged files, and observes .gitignore exclusions; if the user needs for treefmt to scan untracked files or ignored files, they should use the "filesystem" walker or pass those files directly to treefmt as arguments.

System information

Observed and confirmed on multiple Linux systems including ArchLinux and NixOS 25.05. treefmt is version 2.3.1.

[jfly]

In #557, the user described adding a file to the git index (via git add) but not yet committed ("staged" in Git terms), and opined that the file they had just "git add"ed was not included in treefmt via git ls-files. I'm unsure what the user's original issue was because I'm unable to reproduce their described behaviour they experienced

@brycied00d, I think you just misread #557? Quoting the original report (emphasis mine):

2. Don't git add it

---

My expectation is that, with walk = "git", only the files included in git ls-files (.gitignore, Committed, and Staged in the above example) will be processed; files Untracked and treefmt.toml would NOT be considered by treefmt. Only "3" files should have been processed/traversed/formatted above.

This is the opposite of what @michaelpj requested in #557. So we cannot make everyone happy here.

I think I prefer the current behavior. In my experience, untracked files are either:

1. Build artifacts that should be gitignored, OR
2. New files that are being created as part of some development, and are about to be git add-ed.

Both of these work fine with treefmt's current behavior: if you gitignore it, treefmt will not format it, and if you're about to git add it, then formatting it seems very reasonable.

Could you share more about why you want this behavior changed? It is just about it being counterintuitive?

[michaelpj]

I kept my ticket generic, but perhaps it also helps to explain more why I care about that. I use the jj VCS, and when backed by a git repository it periodically creates commits. But there are periods where new files known to jj are not-added-or-committed, i.e. untracked. It's annoying if they don't get formatted.

However, it's certainly arguable that jj is using git weirdly, so I think the best argument is just based on git. And I do agree with @jfly that it's desirable behaviour anyway.

Another way of looking at it: most tools that defer to git for working out what files to ignore are reasonably explicit about saying that they defer to .gitignore. So in that regard the new behaviour is more in line with the common one: we ignore the files that git ignores, and otherwise we do everything.

[brycied00d]

Thanks for the replies! And thank you @michaelpj for the extra context, I wasn't familiar with jj.

I think you just misread #557?

@jfly : 🤦 You're absolutely correct, I didn't even see that "don't", my focus was drawn to the formatted code block. That certainly explains why I couldn't reproduce what I'd interpreted had been said to be the issue. Thanks for clearing that up.

Could you share more about why you want this behavior changed? It is just about it being counterintuitive?

Naturally! Put simply, I often have notes, snippets, test scripts, and even simply files that aren't ready to be added yet in a given directory. For example, in this current project, I have a handful of nix files that aren't yet ready, and won't be for a few weeks (as time permits), and deadnix errors out every time treefmt runs on my working directory. I don't believe that these files, especially the ones that will eventually, someday be staged and committed, belong in .gitignore because those exlcusions don't apply to everyone or the whole project, just myself.

and if you're about to git add it, then formatting it seems very reasonable

My view is that if I haven't yet staged the file, then in "git walk" it shouldn't be automatically included by treefmt; I'll manually format it a la treefmt $file or my IDE integration will run treefmt against the file (according to the IDE's configuration) if that's what I want.

Especially when used as a pre-commit hook, formatting files that aren't part of the commit or git tree is a significant pain. For what it's worth, in our use case we modified our pre-commit hook script to run treefmt $(git ls-files) as a workaround to the unexpected behaviour.

Many other tools, linters and formatters, have either a "git/CI mode" that scans only the files known to Git (e.g. markdownlint --git-recurse runs git ls-files under the hood), or can be configured to run a command to obtain the list of files for scanning or exclusion (e.g. rubocop). I expected treefmt's "git" walker to behave the same way. The coworker who added treefmt to the pre-commit hooks believed the "git" walker worked that way, as did my other coworkers expect treefmt to work this way too until I demonstrated that was not the case.

And just to be clear, I don't see the issue/behaviour as anything to do with .gitignore, and I don't want to frame this as "treefmt is handling .gitignore differently from other tools" because ultimately treefmt's .gitignore handling IS just like other tools. What I'm saying is "in git mode, treefmt is walking non-git files"

Thank you for taking the time to read and consider this!

[michaelpj]

Yes, I can see there's an argument for both "piggyback on git to figure out which files to ignore" and "piggyback on git to work out which files to include", and it's not entirely clear which one treefmt should be doing. Maybe it should be configurable? unclear to me at least!

[jfly]

We absolutely care about making treefmt play nicely with precommit hooks. I don't think I understand the issue, though. Precommit hooks are notoriously difficult to write correctly (distinguishing between staged and unstaged changes for instance). Can you share more information about what you tried and how it behaved surprisingly? (feel free to file a separate issue if appropriate)

I don't believe that these files, especially the ones that will eventually, someday be staged and committed, belong in .gitignore because those exlcusions don't apply to everyone or the whole project, just myself.

Sure. I'd expect them to live in a branch though, rather than littering your workspace. I'm not here to judge your git workflow though, it works for you! Just guessing why this might not affect many of our users.