feature/add-url-md5-sha256

[bwalsh]

ADR: Alternate Object Identifier Strategy for Git LFS Without Content SHA256

Status

Proposed

---

Use Case: Managing Remote-Only Large Files Without Content Hashing

Title

Enable Git LFS workflows for remote-only or very large files without requiring SHA256 content hashing during git add.

---

Primary Actor

Research data steward / data engineer using Git LFS integrated with DRS.

---

Scenario

A research team maintains large genomic files (e.g., BAM, CRAM, FASTQ) that are:

• Already stored in an object store (S3 / Ceph / GCS)
• Registered in DRS
• Multi-GB or TB in size
• Not always locally downloaded

The user wants to:

• Reference these files in a Git repository
• Track them via Git LFS
• Maintain reproducibility and metadata linkage
• Avoid computing SHA256 hashes locally (too slow or impossible)

---

User Story

As a research data steward managing large, remote DRS-registered files,
I want to add files to a Git LFS repository without computing a full content SHA256 hash,
So that I can efficiently reference remote objects while maintaining compatibility with Git LFS and DRS workflows.

---

Functional Expectations

• During git add, the clean filter:

  • Does not require downloading or hashing full file contents.
  • Generates a stable alternate object identifier.

• During git lfs push:

  • Remote existence checks use DRS or metadata services.

• During git checkout:

  • Files are resolved via DRS ID.

• No additional metadata files are committed to Git.

• Integrity and deduplication are delegated to DRS.

---

Acceptance Criteria

• User can run git add on remote-managed large files without content hashing delays.
• Repository remains fully compatible with Git LFS commands.
• git lfs push does not attempt redundant uploads when DRS already contains the object.
• git checkout correctly restores files via DRS resolution.
• No extra files beyond standard LFS pointers are committed.
• Workflow works for multi-GB files without significant local CPU or I/O overhead.

---

Business / Architectural Value

• Eliminates expensive SHA256 operations on large files.
• Enables remote-first, metadata-addressable architecture.
• Aligns Git workflows with DRS and Indexd object identity.
• Supports scalable genomics and bioinformatics data management.

[bwalsh]

ADR: Alternate Object Identifier Strategy for Git LFS Without Content SHA256

Status

Proposed

Date

2026-02-11

---

Context

Git LFS normally uses a SHA256 hash of file contents as the object identifier (oid). This provides strong integrity guarantees and enables content-addressable deduplication.

In our architecture, these assumptions don’t always hold:

• Objects may be remote-only (already stored in S3-compatible object storage and/or registered in DRS/Indexd).
• Objects may be multi-GB / TB, making local hashing expensive.
• During git add / the clean phase, content may be unavailable, or hashing may be prohibitively slow.

We therefore need a practical alternate identity strategy that:

• Avoids content hashing during clean
• Remains compatible with clean/smudge/transfer flows
• Keeps Git commits free of extra “state files”

---

Problem

If we do not compute sha256(content) during the clean phase, we must still:

• Produce a valid Git LFS pointer with an oid
• Support push/pull through a custom transfer agent
• Define remote existence semantics without relying on content SHA256
• Avoid committing auxiliary metadata files to Git

---

Decision Drivers

• Practical performance for very large files
• Remote-first workflows (DRS/S3-compatible storage)
• Compatibility with Git LFS pointer format and lifecycle
• No extra files committed to the repo (beyond standard pointers)
• Clear, testable behavior during clean → transfer → smudge

---

Considered Options

Option 1 — SHA256 of Relative Path

oid = sha256(relative_path)

Pros

• Cheap and deterministic

Cons

• Renames change identity
• Not content-addressable
• No deduplication
• Weak integrity guarantees

---

Option 2 — SHA256 of ETag (Selected)

oid = sha256(etag)

Pros

• Cheap to compute (no file reads)
• Tends to be stable per uploaded object version (ETag changes when content changes)
• Works for remote-only objects
• Enables dedup-ish behavior if ETags are stable for identical content

Cons

• ETag semantics vary by backend and upload method (not universally “MD5 of content”)
• Multipart uploads often produce composite ETags (not equal to MD5(content))
• Integrity shifts from Git-native to storage-native guarantees

---

Decision

Adopt a model aligned with Option 2 (SHA256 of ETag):

• The LFS pointer oid is computed as sha256(etag) when content SHA256 is not available or too expensive.
• The custom transfer agent resolves objects using storage metadata (ETag, size, and/or DRS/Indexd records).
• Full content SHA256 is optional and may be computed later (see Promotion Strategy).
• No additional files beyond standard LFS pointers are committed to Git.
• Dynamic state (ETag ↔ content SHA256, remote locations, etc.) is stored in:
  • .git/lfs (local cache), and/or
  • external registries (DRS/Indexd)

---

Practical Clean / Transfer / Smudge Flow

Clean (git drs add-url)

Goal: Produce a standard LFS pointer without reading full content.

Behavior:

1. Determine object identity inputs (remote URL/DRS ID, bucket/key, etc.).
2. Retrieve the object ETag (e.g., HEAD request).
3. Compute oid = sha256(etag).
4. Emit a standard pointer:
   • version
   • oid sha256:<sha256(etag)>
   • size (if available)

Note: Git LFS “clean” will drop non-standard pointer fields. If additional metadata is needed (drs id, remote url), store it outside the pointer (e.g., Indexd or a local cache), or use a separate, explicitly-managed metadata file that is not committed.

---

Transfer Agent (git lfs push/pull)

Goal: Define remote existence and transfer semantics without content SHA256.

Behavior:

• Existence check: use ETag-derived oid (plus size) to determine whether the remote already has the object.
• Upload: if required, upload via signed URL or direct S3 API, then record resulting ETag.
• Download: resolve to a location (DRS/Indexd, bucket/key) and verify returned ETag matches expected.

Recommended:

• Treat (etag, size) as the practical identity tuple for existence checks.
• Persist mappings:
  • oid(sha256(etag)) → etag → (bucket,key,versionId?) → drsId/indexdId

---

Smudge (git checkout)

Goal: Materialize content into working tree.

Behavior:

1. Read pointer oid.
2. Resolve oid via registry/cache to obtain a fetch location (DRS access URL, bucket/key).
3. Download content.
4. Verify:
   • returned ETag matches expected ETag for this oid (and optionally size)
5. (Optional) If content is now local, compute content SHA256 and promote (below).

---

Promotion Strategy: Upgrade to True Content SHA256 When Available

Sometimes content becomes locally available later (e.g., after smudge, after a workflow run, or after a cache warm-up). We want a strategy to compute and preserve true content SHA256 without breaking Git history or requiring pointer rewrites unless explicitly requested.

Goals

• Preserve “fast path” behavior (ETag-based OID) for remote-only workflows.
• Allow “integrity upgrade” when content is available.
• Keep repo clean: no extra committed state artifacts.

Proposed Strategy

1. Compute content SHA256 opportunistically when the full file exists locally (e.g., post-smudge or explicit command).
2. Record the mapping in external/local state:
   • oid_etag = sha256(etag)
   • oid_content = sha256(content)
   • Store: oid_etag → oid_content along with etag, size, and remote location identifiers.
3. Use the mapping for:
   • Stronger verification (detect storage corruption)
   • Cross-backend portability (ETag semantics differ; content SHA256 is portable)
   • Optional dedup across backends
4. Optional pointer promotion (explicit):
   • Provide a command (e.g., g3t lfs promote / git lfs promote) that:
     • Rewrites pointers from oid_etag to oid_content
     • Updates any external metadata registry to preserve lineage
     • Requires a normal Git commit (user-controlled)
   • Default behavior: do not rewrite committed pointers automatically.

---

Ceph RGW ETag Behavior (Backend Analysis)

Our backend is Ceph RGW (S3-compatible). ETag behavior matters because we are using sha256(etag) as the effective LFS OID.

Key Observations

• S3-style ETags are commonly MD5(content) for single-part uploads, but multipart uploads typically produce composite ETags (often with a -<parts> suffix), and ETag should not be treated as a universal checksum:
  • https://docs.aws.amazon.com/AmazonS3/latest/userguide/tutorial-s3-mpu-additional-checksums.html
• Ceph RGW implements the S3 API and supports multipart upload operations:
  • https://docs.ceph.com/en/latest/radosgw/s3/objectops/
• RGW’s internal representation of large/multipart objects can involve multiple underlying objects (head/tail), which supports why an ETag may not be a simple MD5 of the full logical object:
  • https://ceph.io/en/news/blog/2025/rgw-deep-dive-2/
• Practical, real-world discussions of RGW multipart ETag behavior:
  • https://forum.proxmox.com/threads/ceph-object-gateway-radosgw-multi-part-uploads-and-etag.144687/

Practical Implications for sha256(etag)

• sha256(etag) is a good remote-identity proxy if:
  • the ETag returned by HEAD is stable for a given stored object, and
  • changes when content changes.
• For multipart uploads, ETag will still typically change when any part changes, but:
  • it is not a portable content checksum, and
  • it may vary based on part sizing / upload strategy.

Recommended Operational Guidance

• Prefer a consistent upload strategy (fixed multipart part size) to stabilize ETag patterns.
• Treat (etag, size) as the practical identity for existence checks.
• Use the promotion strategy to compute true content SHA256 for portability/strong integrity when needed.

---

Risk Analysis

R1: ETag is not a universal content checksum

• Description: ETag may be MD5(content) in some cases, composite in multipart cases, and otherwise implementation-dependent.
• References:
  • https://docs.aws.amazon.com/AmazonS3/latest/userguide/tutorial-s3-mpu-additional-checksums.html
• Impact: Integrity assumptions may be weaker than content SHA256.
• Mitigation:
  • Use (etag, size) together
  • Promote to content SHA256 when content becomes available
  • For high-integrity contexts, require checksum fields from DRS or explicit checksum headers

R2: Multipart upload variability

• Description: ETag can depend on part size/count; same content uploaded differently may yield different ETags.
• References:
  • https://docs.aws.amazon.com/AmazonS3/latest/userguide/tutorial-s3-mpu-additional-checksums.html
  • https://forum.proxmox.com/threads/ceph-object-gateway-radosgw-multi-part-uploads-and-etag.144687/
• Impact: Dedup across clients may be reduced.
• Mitigation:
  • Standardize multipart part size in tooling
  • Use promotion mapping (oid_etag → oid_content) to unify identity later

R3: Backend-specific edge cases (Ceph RGW / clients)

• Description: Some clients may observe missing or unexpected ETag behavior in certain multipart implementations or integrations.
• References:
  • https://forum.proxmox.com/threads/ceph-object-gateway-radosgw-multi-part-uploads-and-etag.144687/
• Impact: Clean phase may fail to compute OID if ETag not obtainable.
• Mitigation:
  • Define a fallback: sha256(path) or require remote registration via DRS that returns a stable identifier
  • Add retries and clear error messages

R4: Collision / mistaken-identity risk

• Description: SHA256 collision risk is negligible; the main risk is ETag collision, reuse, or mismatch due to backend behavior.
• Impact: Wrong object may be referenced.
• Mitigation:
  • Verify size
  • Optionally store additional invariants (versionId, checksums from DRS)
  • Promote to content SHA256 where appropriate

---

Sequence Diagram: Clean → Transfer → Smudge

sequenceDiagram
  autonumber
  actor U as User
  participant G as Git
  participant C as Clean Filter
  participant A as Transfer Agent
  participant S as Object Store (Ceph RGW / S3)
  participant R as Registry (DRS/Indexd or local cache)
  participant M as Smudge Filter

  U->>G: git drs add-url my-data/file.bam
  G->>C: clean(file stream)
  C->>S: HEAD object (or resolve remote) to obtain ETag
  S-->>C: ETag, size
  C->>C: oid = sha256(etag)
  C-->>G: LFS pointer (oid=sha256(etag), size)

  U->>G: git push
  G->>A: transfer(upload/existence check for oid)
  A->>R: lookup oid -> remote location?
  R-->>A: location info (if known)
  A->>S: HEAD/exists? (verify ETag/size)
  alt object missing
    A->>S: PUT/Multipart upload
    S-->>A: ETag
    A->>R: record oid->etag->location mapping
  else object exists
    A->>R: ensure mapping present / refresh
  end
  A-->>G: success

  U->>G: git checkout
  G->>M: smudge(pointer oid)
  M->>R: resolve oid -> location
  R-->>M: fetch URL / bucket+key
  M->>S: GET object
  S-->>M: bytes + ETag
  M->>M: verify ETag matches oid expectation
  M-->>G: write file to working tree

  opt promotion
    M->>M: compute sha256(content)
    M->>R: store mapping oid_etag -> oid_content
  end

Loading

---

Consequences

Positive

• Eliminates expensive hashing during git add
• Enables remote-only workflows
• Scales to multi-GB objects
• Keeps Git history clean (no extra committed state files)

Negative

• Identity depends on ETag semantics (backend/upload dependent)
• Not universally content-addressable without promotion

---

Future Work

• Document backend-specific assumptions (Ceph RGW versions, encryption, multipart policies)
• Define a standard multipart part size policy for stable ETags
• Add explicit CLI to “promote” pointers to content SHA256 when desired
• Optionally incorporate DRS checksum fields for stronger integrity checks

[bwalsh]

ADR: Handling Foreign URL Objects with Standalone Git LFS Transfer Agent

Status

Proposed

Context

We support adding remote (“foreign”) objects into a Git repository using:

git drs add-url s3://my-bucket/my-file.txt my-data/my-file.txt

This command creates a valid Git LFS pointer file using:

oid = sha256(etag)

We are currently using:

lfs.standalonetransferagent=drs

During git push, Git LFS reports:

(missing) data/simple_test_file.txt (<oid>)

and the standalone transfer agent never receives an "upload" event.

Problem

When using lfs.standalonetransferagent, Git LFS:

• Does not negotiate via the Batch API.
• Requires the referenced LFS object to exist locally in:
  .git/lfs/objects/<oid[0:2]>/<oid[2:4]>/
• Marks objects as missing if local content is absent.
• Will not enqueue an upload request if the object does not exist locally.

Therefore, the standalone transfer agent cannot implement "reference mode" unless local object hydration occurs before push.

Alternative 1: Download "on add"

When using lfs.standalonetransferagent, we COULD implement a pre-push hydration strategy:

1. Detect LFS OIDs about to be pushed.
2. For any OID missing locally:
   • Resolve oid -> foreign_url using local registry and/or DRS/Indexd.
   • Download the object from the foreign source.
   • Store it in the canonical LFS object path:
     .git/lfs/objects/<oid[0:2]>/<oid[2:4]>/
3. Allow normal git push to proceed.
4. The standalone transfer agent will then receive valid "upload" events.

We explicitly do not rely on Batch API “already present” semantics when using standalone mode.

Rationale

Standalone transfer agents bypass Batch API negotiation.

Therefore:

• “Already present” cannot be signaled server-side.
• LFS must see a local object before it attempts upload.
• Hydration ensures compatibility with upstream Git LFS behavior.
• No extra metadata files are committed to the repository.

This preserves:

• git drs add-url pointer-only workflow
• Standard Git LFS push semantics
• Compatibility with existing tooling

Architecture

Local Registry (Untracked)

When git drs add-url is run:

• Write pointer file to working tree.

• Store mapping in local state:

  .git/lfs/foreign/registry.jsonl

Example record:

{
"oid": "sha256:<sha256(etag)>",
"etag": "",
"size": 12345,
"source_url": "s3://my-bucket/my-file.txt"
}

This registry is never committed.

Sequence Diagram

sequenceDiagram
  autonumber
  actor U as User
  participant G as Git
  participant L as Git LFS
  participant H as Pre-Push Hydration
  participant R as Local Registry
  participant S as Foreign Object Store
  participant A as Standalone Transfer Agent
  participant C as Canonical LFS Storage

  U->>G: git push
  G->>L: pre-push hook invoked
  L->>H: identify OIDs to push
  H->>R: lookup oid -> source_url
  alt object missing locally
      H->>S: download object
      S-->>H: bytes + etag
      H->>H: store in .git/lfs/objects/<oid>
  end
  H-->>L: hydration complete
  L->>A: upload event (oid, size, path)
  A->>C: upload/copy object
  C-->>A: success
  A-->>L: complete event
  L-->>G: push succeeds

Loading

Alternatives Considered

1. Batch API “already present” negotiation

Rejected because:

• lfs.standalonetransferagent bypasses Batch API.
• No server-side negotiation step is available.

2. Fake upload completion without local object

Rejected because:

• LFS fails earlier with “(missing)” before agent sees upload.
• Violates LFS expectations and risks corruption.

Acceptance Criteria

• git drs add-url creates a valid LFS pointer.
• git push succeeds without "(missing)" errors.
• Standalone transfer agent receives "upload" events.
• No extra committed metadata files.
• Fresh clones succeed if canonical storage contains object.

Consequences

Positive

• Compatible with standalone mode
• Preserves pointer-only add workflow
• Keeps upstream LFS behavior unchanged

Negative

• Requires hydration download before first push
• Cannot implement true reference-only semantics without Batch API

Future Work

• Evaluate switching to negotiated lfs.customtransfer.* model for true reference mode.
• Add concurrent hydration support.
• Add retry/backoff policy for foreign downloads.

[bwalsh]

I tried the sentinel object approach, now I get the following error 

"""
Git LFS upload failed: 0% (0/1), 0 B | 0 B/s (corrupt) data/simple_test_file.txt (b86b6c25340cbb2126d7878cb8b6fae81cc8d7ea765cd83c883f67ebf3495b36) 
"""

Yep — that result is exactly what you’d expect from the sentinel approach.

What “(corrupt)” means here

Git LFS found something in .git/lfs/objects/.../<oid>, so it’s no longer “missing”, but when it tried to use/validate it, it determined the local object is not a valid LFS object for that OID.

In practice, “corrupt” usually means one (or more) of these checks failed:

• the object file size doesn’t match what LFS expects
• the object file content doesn’t hash to the OID (for standard LFS, OID = sha256(content))
• the object file is incomplete / truncated
• LFS tried to read it and couldn’t

With your sentinel, at least the hash check will always fail (because the file bytes are not the real content, and you’re using an OID derived from ETag anyway).

So: writing dummy bytes into .git/lfs/objects cannot work reliably because LFS treats that store as authoritative and validates it.

---

How to fix it (standalone transfer agent)

Fix A (recommended): Pre-push hydration with real bytes

Before git push, ensure the real content is written into the LFS object store at:

.git/lfs/objects/<oid[0:2]>/<oid[2:4]>/<oid>

Then LFS won’t call it corrupt/missing, and it will proceed to call your upload path.

Even if you keep your OID = sha256(etag), you must ensure the local object bytes match whatever validation LFS performs. If LFS is doing a sha256(content) verification, then your ETag-derived OID will never validate.

Which brings us to the key point:

⚠️ In standard Git LFS, oid MUST equal sha256(content)

If you deviate from that, Git LFS will label objects corrupt as soon as it validates local content.

That means: ETag-derived OIDs are not compatible with stock Git LFS integrity checks unless you are also replacing the parts of LFS that assume OID=sha256(content) (i.e., using a negotiated custom transfer flow plus server semantics that don’t require local verification).

---

Fix B: Keep OID as sha256(content), and use ETag only for lookup

This is the “works with Git LFS” option:

• Pointer oid remains sha256(content) (true LFS spec)

• You store etag, source_url, etc. externally (Indexd/DRS/local registry)

• You avoid hashing at add time by:

  • deferring SHA256 computation until hydration time (pre-push or first checkout)
  • or computing SHA256 server-side during mirroring

This preserves all LFS invariants and eliminates “corrupt”.

---

Fix C: Stop using standalone and switch to negotiated custom transfers + Batch semantics

If your real goal is: “don’t require local content, don’t require sha256(content), just reference remote objects”, you need to stop fighting LFS’s assumptions.

• Remove lfs.standalonetransferagent
• Use lfs.customtransfer.*
• Implement an LFS Batch API that returns “already present / no upload action”
• Ensure downloads resolve through DRS/Indexd

This is the only path that cleanly supports “reference-only” without content hashing while remaining sane.

---

How to confirm

1. Confirm what OID is in the pointer:

git show HEAD:data/simple_test_file.txt

2. Confirm what’s in .git/lfs/objects/.../<oid>:

OID=b86b6c25340cbb2126d7878cb8b6fae81cc8d7ea765cd83c883f67ebf3495b36
ls -l .git/lfs/objects/${OID:0:2}/${OID:2:2}/$OID
shasum -a 256 .git/lfs/objects/${OID:0:2}/${OID:2:2}/$OID

If that SHA doesn’t equal the pointer’s OID, LFS will call it corrupt.

$ git show HEAD:data/simple_test_file.txt
version https://git-lfs.github.com/spec/v1
oid sha256:b86b6c25340cbb2126d7878cb8b6fae81cc8d7ea765cd83c883f67ebf3495b36
size 12
$ cat .git/lfs/objects/b8/6b/b86b6c25340cbb2126d7878cb8b6fae81cc8d7ea765cd83c883f67ebf3495b36 
git-drs:SENTINEL DATA

---

Bottom line

• The sentinel approach gets you from missing → corrupt, because LFS validates.
• With a standalone transfer agent, you can’t “trick” LFS with placeholder bytes.
• If you keep stock Git LFS: the oid must be sha256(content).