feat(output): implement YAML and JSON output #1800

Dustin-Jiang · 2025-09-30T15:01:56Z

Implement --yaml switch for YAML format output, which allows using yq or nushell to interact with the result.

Actually I was initially working on #1765, but could not reach to a perfect state. JSON afraids trailing commas, making it not so easy to statelessly stream the result without a buffer. Considering a large number of potiential results, it would be memory-consuming to store all lines and print them at last.

On the other side, the YAML format, while infamous for its complexity on parsing, is friendly for streaming output. No need for serializing or extra dependencies, the simple and fast write! is all you need.

There are tools like yq that work just like beloved jq, and nushell supports YAML as well, so I suggest this PR might be able to close #1765.

tmccombs · 2025-09-30T15:37:03Z

I think i would really prefer json output. There are more tools that would support that as input.

As for the streaming problem, I think there are a couple of ways to handle that:

Write the comma at the beginning of each entry instead of the end
Use newline delimited json (ndjson) instead of a json array. Tools like jq can still parse this.
Just don't write the last item until we either have the next one, or have reached the end, so we know if we need a comma or not

Dustin-Jiang · 2025-09-30T18:17:33Z

I think i would really prefer json output. There are more tools that would support that as input.

Yeah definitely, but the original functions in output.rs were stateless, so I have to refactor it to make it stateful.

The good news is, fortunately, that we gain a little bit performance improvement after refactoring, maybe because of less param passing (I guess)?

tmccombs

Should also have some tests, and the man page should be updated.

As well as the things I commented on above.

src/config.rs

src/output.rs

src/walk.rs

doc/fd.1

src/cli.rs

src/output.rs

tmccombs · 2025-10-15T07:21:54Z

src/output.rs

+    // Try to convert to UTF-8 first
+    match std::str::from_utf8(bytes) {
+        Ok(utf8_str) => {
+            let escaped: String = utf8_str.escape_default().collect();


I don't think collecting into an intermediate string is strictly necessary, if PathEncoding included a lifetime based on the path.

collecting into an intermediate string

I'm a little bit confused, since the escape_default always allocates new memory.

In my understanding, to avoid an intermediate string, I should borrow the utf8_str and store in the PathEncoding as Cow<'a, str>, and escape it when write!().

But the PathEncoding is created per file and released after the certain file is printed (or not?), and it is a must (or not?) to .escape_default().collet() into a String when escaping, so I can't see differences between.

But I'm just a newbie to Rust and I'm pretty sure you are right :), so looking forward to your further suggestion.

So, you would define PathEncoding like

enum<'a> PathEncoding<'a> { Utf8(EscapeDefault<'a>), Bytes(&'a [u8]), }

Then this would be defined like:

fn encode_path(path: &Path) -> PathEncoding<'_> { match path.to_str() { Some(utf8) => PathEncoding::Utf8(utf8.escape_default()), None => PathEncoding::Bytes(path.as_os_str().as_encoded_bytes()), } }

And FileDetail would need a lifetime parameter as well.

I would strongly encourage y'all to look at how ripgrep encodes paths. And in particular, you really really really should not be using WTF-8 in your output format. From the WTF-8 spec:

Any WTF-8 data must be converted to a Unicode encoding at the system’s boundary before being emitted. UTF-8 is recommended. WTF-8 must not be used to represent text in a file format or for transmission over the Internet.

You'll also want to check how PathEncoding serializes. Sometimes just serializing a &[u8] leads to something sub-optimal (like an array of integers or something). This is why ripgrep base64 encodes data that isn't valid UTF-8.

src/output.rs

tavianator · 2025-11-04T13:33:17Z

Just want to throw out a reference to libxo as my favourite way for command line programs to support structured output formats. Not sure there's something like that in the Rust ecosystem.

tmccombs

I few minor suggestions, but I think it's close.

tmccombs · 2025-11-11T07:12:10Z

doc/fd.1

 due to OS restrictions on the maximum length of command lines.
+.TP
+.BI "\-\-json "
+Specify JSONL (as known as NDJSON) format to use for the output. 


It would probably be good to list the fields that are included in the output.

tmccombs · 2025-11-11T07:14:12Z

src/walk.rs

 }

-impl<'a, W: Write> ReceiverBuffer<'a, W> {
+impl<'a, W: Write + 'static> ReceiverBuffer<'a, W> {


Probably doesn't matter too much, but this could probably be

Suggested change

impl<'a, W: Write + 'static> ReceiverBuffer<'a, W> {

impl<'a, W: Write + 'a> ReceiverBuffer<'a, W> {

tmccombs · 2025-11-11T07:16:15Z

src/walk.rs

    /// Flush stdout if necessary.
    fn flush(&mut self) -> Result<(), ExitCode> {
-        if self.stdout.flush().is_err() {
+        if self.printer.stdout.flush().is_err() {


Should we add a flush method on to printer, so this doesn't need to worry about the stdout field?

tmccombs · 2025-11-11T07:17:19Z

CHANGELOG.md


 ## Features

+- Add `--yaml` flag for YAML format output. 


This should be changed to

Suggested change

- Add `--yaml` flag for YAML format output.

- Add `--jsonl` flag for JSONL format output.

tmccombs · 2025-11-12T06:11:58Z

src/output.rs

+        write!(self.stdout, "}}")
+    }
+
+    fn print_entry_detail(&mut self, format: OutputFormat, entry: &DirEntry) -> io::Result<()> {


nit: We could potentially combine this with the print_entry_json_obj method now, since it is only used for jsonl.

OTOH, it may be useful in the future if we add a yaml format later, or maybe a tabular format?

Dustin-Jiang added 4 commits September 30, 2025 22:13

feat(output): add yaml output

ffec773

docs: update CHANGELOG

97d6bfb

fix: escape path, and disable permission display on Windows

cdb7bc0

fix(ci): fix warnings in cargo clippy

c9cbb25

Dustin-Jiang added 4 commits October 1, 2025 01:47

refactor: make output stateful

192ca92

feat(output): add json output

602b389

fix(ci): fix warnings in cargo clippy

eb2f10d

fix: fix function calling in Windows

4ea0ed3

Dustin-Jiang changed the title ~~feat(output): implement YAML output~~ feat(output): implement YAML and JSON output Sep 30, 2025

fix: fix reference mutable type annotation in Windows

e225946

tmccombs requested changes Oct 2, 2025

View reviewed changes

tmccombs requested a review from sharkdp October 2, 2025 07:54

Dustin-Jiang added 8 commits October 10, 2025 23:33

fix: resolve suggested changes

977ee0e

fix: move JSON array printing to Printer

6434ee5

feat: implement NDJSON output

b2d385f

fix(ci): fix warnings in cargo clippy

703b32f

tests: add tests for --output flags

650e86c

docs: update manpage for --output flags

2e463c7

Merge branch 'master' into feature-yaml

e46cce0

tests: fix invalid utf8 base64 test

10570e9

Dustin-Jiang requested a review from tmccombs October 15, 2025 07:22

tmccombs reviewed Oct 15, 2025

View reviewed changes

Dustin-Jiang added 3 commits October 15, 2025 16:39

docs: update manpage to change "ndjson" to "jsonl"

949a5aa

fix: change FileDetail creating logic and base64 import

cb3ef97

fix: change ndjson flag to commonly used jsonl

60ecc09

tmccombs reviewed Oct 28, 2025

View reviewed changes

src/output.rs Outdated Show resolved Hide resolved

Dustin-Jiang added 2 commits October 29, 2025 23:37

fix: replace String to &str with lifetime, adopt as_encoded_bytes

7c9f1d8

feat: add --json flag for JSONL output

2c1bdb5

Dustin-Jiang added 4 commits October 29, 2025 23:49

docs: add --json flag to manpage

e831976

Merge branch 'master' into feature-yaml

30c9e86

fix(clippy): collapse if blocks

49654f4

Merge remote-tracking branch 'origin/master' into feature-yaml

e4741c0

Dustin-Jiang added 3 commits November 8, 2025 20:07

Merge remote-tracking branch 'origin/master' into feature-yaml

4a0ecc5

fix: remove the --output flag

56d347e

tests: fix --output tests to --json

6db4409

tmccombs requested changes Nov 12, 2025

View reviewed changes

	impl<'a, W: Write + 'static> ReceiverBuffer<'a, W> {
	impl<'a, W: Write + 'a> ReceiverBuffer<'a, W> {

	- Add `--yaml` flag for YAML format output.
	- Add `--jsonl` flag for JSONL format output.

Uh oh!

feat(output): implement YAML and JSON output #1800

Are you sure you want to change the base?

feat(output): implement YAML and JSON output #1800

Uh oh!

Conversation

Dustin-Jiang commented Sep 30, 2025

Uh oh!

tmccombs commented Sep 30, 2025

Uh oh!

Dustin-Jiang commented Sep 30, 2025

Uh oh!

tmccombs left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

tavianator commented Nov 4, 2025

Uh oh!

tmccombs left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants