feature/progress-event

[bwalsh]

Custom Transfer Agent Reports Byte-Level Progress to Git LFS

Implement JSON progress events in cmd/transfer for both send and recieve

Context

We operate a Git LFS custom transfer agent (upload and/or download). Users expect Git LFS’s terminal UI to show live upload progress (bytes sent, overall percent, ETA-like behavior) similar to the built-in HTTP adapter.

In practice, the agent cannot control Git’s progress UI; it can only provide progress telemetry to git-lfs, which then updates its own progress display.

Git LFS’s custom transfer protocol explicitly supports line-delimited JSON “progress” events with byte counters (bytesSoFar, bytesSinceLast) and requires that the last progress event reflects completion. ([Debian Sources]1)

---

Decision

Implement byte-accurate progress reporting in the custom transfer agent by emitting line-delimited JSON messages on stdout:

• Emit one or more {"event":"progress", ...} messages during each transfer.
• Ensure the final progress message for a successful transfer has bytesSoFar == size.
• Emit a {"event":"complete", "oid": ...} message when done (or complete with an embedded error).
• Flush stdout after each JSON line (to avoid buffered/stale progress).

Protocol requirements (LDJSON, progress fields, and completion expectations) are defined by git-lfs. ([Debian Sources]1)

---

Rationale

• Aligns with the officially-supported Git LFS custom transfer protocol.
• Works with Git LFS’s built-in progress renderer across concurrent transfers.
• Keeps stdout machine-readable and allows stderr for human logs.

---

Non-Goals

• Directly updating Git’s native “upload progress” text (not supported by the agent).
• Implementing a separate custom TUI/TTY progress renderer in the agent.

---

Implementation Notes

Progress event format (stdout; one JSON object per line):

{ "event": "progress", "oid": "<oid>", "bytesSoFar": 1234, "bytesSinceLast": 64 }

• bytesSoFar: total bytes transferred so far for that oid
• bytesSinceLast: delta since last progress message
  The last progress message for a successful transfer must have bytesSoFar == size. ([Debian Sources]1)

Process I/O hygiene

• stdout: protocol JSON only (LDJSON).
• stderr: logs/diagnostics.
• flush stdout after each JSON line (recommended by the protocol). ([Debian Sources]1)

---

Alternatives Considered

1. Write to stderr with a custom progress bar
   Rejected: doesn’t integrate with git-lfs’s progress aggregation; likely noisy, breaks tooling, and won’t match LFS’s UX expectations.

2. Disable concurrency and do internal concurrency (lfs.customtransfer.<name>.concurrent=false)
   Optional optimization, but not a replacement for progress reporting; progress events still required for good UX.

3. No progress reporting
   Rejected: poor UX, appears hung on large objects, increases support load.

---

Consequences

Positive

• Users see accurate progress and throughput in standard git lfs push/pull output.
• Works with lfs.concurrenttransfers since git-lfs aggregates per-object progress.

Negative / Risks

• Over-reporting progress (too frequent messages) can add overhead.
• Incorrect counters (non-monotonic bytesSoFar, wrong final value) can cause confusing UI.

---

Acceptance Tests

AT-1: Progress events update LFS UI during upload

Given a repo with an LFS-tracked file of size ≥ 100MB
And lfs.customtransfer.<name>.path points to the agent
When the user runs git lfs push origin <ref>
Then the terminal output shows incrementing upload progress over time
And the transfer completes successfully.

Pass criteria: progress visibly updates multiple times before completion.

---

AT-2: Final progress equals object size

Given an upload of object oid=X with size=S
When the agent completes successfully
Then the final progress event emitted for oid=X has bytesSoFar == S ([Debian Sources]1)
And the agent emits a complete event for oid=X.

Pass criteria: captured stdout logs show the final progress equals S and a subsequent complete.

---

AT-3: LDJSON protocol compliance

Given an upload request
When the agent writes protocol messages
Then each JSON message is on a single line terminated by \n ([Debian Sources]1)
And no non-JSON text is written to stdout.

Pass criteria: a line-by-line parser can decode every stdout line as JSON.

---

AT-4: Error handling does not kill the agent

Given a batch of multiple uploads
And one object upload fails (simulated 403/timeout/remote failure)
When the agent reports failure
Then it returns a complete message containing an error for that oid ([Debian Sources]1)
And continues processing subsequent transfers without exiting early.

Pass criteria: later objects still upload; process exit code remains 0 unless a fatal (non-transfer-specific) error occurs.

---

AT-5: Concurrency does not break progress

Given lfs.concurrenttransfers=8 and agent concurrency enabled
When pushing ≥ 16 LFS objects
Then progress updates are observed for multiple OIDs
And overall push completes with all objects uploaded.

Pass criteria: at least 2 OIDs show progress interleaved; push succeeds.

---

Test Harness Ideas (Optional)

• Instrument the agent to optionally mirror protocol stdout to a file (or run under a wrapper capturing stdout) to assert:

  • monotonic bytesSoFar
  • final bytesSoFar == size
  • bytesSinceLast sums to size

• Use a “slow” transport mode (rate-limited copy) to force multiple progress ticks.

---

References

• Git LFS Custom Transfer Protocol (LDJSON, progress fields, completion semantics). ([Debian Sources]1)