DataLoader subprocess deadlock: MainThread hangs forever on queue.get after shared-memory unlink failure

[mihow]

Symptom

Worker process appears RUNNING (supervisor, process listing) but stops making progress. Last Batch N: ... log line is a normal-looking processing entry — no exception, no SIGKILL, no OOM. Uptime keeps accumulating. Log file mtime goes stale.

When processing resumes on other workers (NATS redelivers after ack_wait) the hung worker remains silent until manually restarted.

Root cause (evidenced by py-spy native dump)

With AMI_NUM_WORKERS=1, torch DataLoader uses a subprocess to prefetch batches. The subprocess communicates with the main process via multiprocessing.Queue. During normal operation the subprocess serializes each batch tensor with torch.multiprocessing.reductions.reduce_storage, which uses _share_fd_cpu_ / shm_unlink under the hood.

Under pressure (large images, rapid batching, or when the subprocess was previously signalled) the subprocess can die while attempting to share storage, logging to stderr:

RuntimeError: could not unlink the shared memory file /torch_<pid>_<hash>_<n> : No such file or directory (2)

The subprocess dies. MainThread is still blocked in _try_get_data, polling the now-abandoned multiprocessing.Queue with no timeout:

poll (libc.so.6)
select (selectors.py:415)
wait (multiprocessing/connection.py:948)
_poll (multiprocessing/connection.py:440)
poll (multiprocessing/connection.py:257)
get (multiprocessing/queues.py:113)
_try_get_data (torch/utils/data/dataloader.py:1310)
_get_data (torch/utils/data/dataloader.py:1483)
_next_data (torch/utils/data/dataloader.py:1524)
__next__ (torch/utils/data/dataloader.py:741)
_preload (trapdata/antenna/datasets.py:529)
__next__ (trapdata/antenna/datasets.py:552)
_process_job (trapdata/antenna/worker.py:458)

The torch DataLoader does have a timeout= parameter for exactly this situation, but the default is 0 (wait forever). We don't set it.

Reproduction

Not yet reproduced on demand. Observed twice in production within an 8-hour window on two different H100 MIG slices (12 GiB and 24 GiB), both running feature/tuning-knobs-138 and feature/uv-migration. Appears to correlate with large-image processing (3840×2160 and 4096×2160 mixed-resolution batches).

Workaround applied

Set AMI_NUM_WORKERS=0 on the affected workers. Eliminates the subprocess entirely — DataLoader runs inline in the main process. Worker has been stable for the remainder of the observation window. Trade-off: no data-load/GPU-compute overlap, estimated ~20–40% throughput loss on H100 MIG slices where S3 image download is a non-trivial fraction of per-batch time.

Proposed fixes (not decided — discussion welcome)

Ordered roughly from cheapest to cleanest:

1. Set an explicit DataLoader(timeout=...) — e.g. 120 seconds. On stall, DataLoader raises instead of hanging. Worker process can exit / supervisor restarts it / NATS redelivers. Cheapest possible guard. Does not fix the unlink crash.

2. Catch the shared-memory unlink error and restart the DataLoader — wrap the inner loop in a try/except on RuntimeError matching "could not unlink the shared memory file", tear down the DataLoader, rebuild it, resume. More complex but preserves in-process restart.

3. Switch to multiprocessing_context="spawn" or "forkserver" — may or may not help, depending on whether the root cause is fork-related. Worth trying on a reproducer.

4. Use persistent_workers=True + a fixed worker pool — keeps the subprocess alive across batches, may avoid the per-batch subprocess churn that triggers the race.

5. Upgrade torch — we're on 2.10.0+cu128 in the venv that hangs. Recent torch versions have had several fixes in torch.multiprocessing.reductions for SHM cleanup races. Worth checking the torch changelog against 2.10.0 for known fixes.

Evidence

• py-spy native dump of the hung main thread (see traceback above)
• Supervisor log shows steady Batch N progression then silence, no exit
• stderr log contains the could not unlink the shared memory file RuntimeError
• No dmesg OOM, no CUDA error, GPU memory allocation unchanged during the hang

What we still need to verify

• Reproducibility: whether the hang is deterministic given a specific batch content (e.g. the mixed-resolution warning immediately before the hang in two of our cases), or time-dependent
• Whether setting DataLoader(timeout=120) alone is sufficient for operational recovery, or whether the unlink error also needs to be handled
• Throughput delta at NUM_WORKERS=0 vs a fixed NUM_WORKERS=1 once stable — to confirm whether restoring subprocess prefetch is worth the complexity
• Whether a current-torch upgrade would silently fix this (check torch release notes for SHM / _share_fd_cpu_ fixes since 2.10.0)

Related

• Refactor _process_job() in worker.py #121 — _process_job() refactor. The hang occurs inside _process_job at the DataLoader iteration site. A refactor could introduce the timeout guard naturally.
• Clarify and document tuning parameters #138 — tuning-parameter documentation. AMI_NUM_WORKERS is one of the parameters; its current behaviour (hangs the worker if > 0 under load) needs to be called out.

[mihow]

Recurrence observed 2026-04-30 — silent DataLoader hang on a GPU worker, matches this issue's signature.

Symptom

Worker process running, supervisor reports RUNNING, but no log output for ~23h. CPU time on parent worker process grew only ~55s since the start of the silent window (i.e. effectively idle). DataLoader subprocess (forked child) accumulated ~22s CPU over the same window — also idle. Other worker proc on the same host was actively processing batches throughout, so no host-wide cause (network, NVIDIA, OOM).

Restarting just the wedged supervisor program cleared the hang and it picked up work within seconds.

py-spy dump of parent (idle, holding the DataLoader iterator)

Thread (idle): "MainThread"
    wait (threading.py:359)
    get (queue.py:180)
    _try_get_data (torch/utils/data/dataloader.py:1310)
    _get_data (torch/utils/data/dataloader.py:1473)
    _next_data (torch/utils/data/dataloader.py:1524)
    __next__ (torch/utils/data/dataloader.py:741)
    _preload (trapdata/antenna/datasets.py:455)
    __next__ (trapdata/antenna/datasets.py:478)
    _process_job (trapdata/antenna/worker.py:477)
    ...
Thread (idle): "Thread-... (_pin_memory_loop)"
    select (selectors.py:415)
    wait (multiprocessing/connection.py:1136)
    _poll → poll → get
    do_one_step (torch/utils/data/_utils/pin_memory.py:28)
    _pin_memory_loop (torch/utils/data/_utils/pin_memory.py:52)
Thread (idle): "QueueFeederThread"
    wait (threading.py:355)
    _feed (multiprocessing/queues.py:251)

py-spy dump of DataLoader child worker

Thread (idle): "MainThread"
    select (selectors.py:415)
    wait (multiprocessing/connection.py:1136)
    _poll → poll → get
    _worker_loop (torch/utils/data/_utils/worker.py:318)
    ... (multiprocessing.process bootstrap)
    __init__ (torch/utils/data/dataloader.py:1192)
    _get_iterator (torch/utils/data/dataloader.py:433)
    __iter__ (torch/utils/data/dataloader.py:495)
    __init__ (trapdata/antenna/datasets.py:447)
    _process_job (trapdata/antenna/worker.py:470)

Both ends sitting on multiprocessing.Queue.get indefinitely — parent waiting for a batch that the child will never send, child waiting for a task it will never receive. Same deadlock shape as the original report.

Notes

• Worker was running with --num_workers > 0 (DataLoader subprocesses present).
• pin_memory_loop thread alive in parent → worker subprocess started successfully and the deadlock arose mid-stream rather than at startup.
• No prior error / warning in stderr or stdout before the silence began.
• File-descriptor count not noted at observation time; can capture next recurrence.

This makes the hang visible in production with a 5-minute "no batch progress" alarm; restart loop clears it but the deadlock root cause remains. Cross-reference: the FD-leak issue tracked separately appears related but distinct (this hang occurred well before fd exhaustion would be expected on a fresh-ish process).

[mihow]

Claude says: New production recurrence — adding evidence and a fix proposal.

Plain-language summary (for non-deep-in-the-weeds readers)

On 2026-05-19 a routine 408-image production job lost about half its images (~201/408 failed) because the GPU workers ran out of host memory and got killed by the kernel mid-job. The root cause was the same DataLoader shared-memory leak this issue tracks. Each batch leaves behind shared-memory pages that PyTorch never reclaims, and over enough batches the host runs out of RAM and the OS kills the worker — which then drops all the NATS messages it was holding, which then makes the affected images look like "failed" to the server.

We've known about this since late April but no fix has shipped. This comment proposes two small DataLoader config changes (switch the multiprocessing context from fork to forkserver, and add a timeout so the silent hang becomes a restartable error) that should address the root cause rather than just slow it down.

Recurrence: 2026-05-19

A 408-image production job (quebec_vermont_moths_2023 pipeline) was pulled in parallel by two prod worker processes (numprocs=2, AMI_NUM_WORKERS=1) on each of two H100 24GB hosts (~68 GiB host RAM, zero swap, Arbutus cloud). Both hosts hit the kernel global OOM-killer within ~5 minutes of each other (~17:32–17:36 UTC).

Per-process RSS at OOM (host A snapshot):

process	RSS	shmem-rss
ami prod_00	17.6 GB	16.6 GB
ami prod_01	23.7 GB	22.3 GB
pt_data_worker (one of)	19.5 GB	(anon-rss 14.1 GB)
pt_data_worker (one of)	7.7 GB	—
sum (ami + pt_data_worker)	~71.7 GB	exceeds host RAM

Host B was more extreme: single ami proc at 35.3 GB shmem-rss, sibling at 14.8 GB, plus DataLoader children → ~69 GB sum.

dmesg confirms oom-kill: constraint=CONSTRAINT_NONE ... global_oom (host-wide, not cgroup). Multiple cascading kills as autorestarted workers re-bloated. App log on host A shows the matching RuntimeError: could not unlink the shared memory file /torch_*: No such file or directory (2) traceback in torch/multiprocessing/reductions.py reduce_storage — the exact #140 signature.

Settings on the hosts: pin_memory=True, prefetch_factor=4, num_workers=1, persistent_workers=True, batch_size=1 (per datasets.py:411–446). Stock fork multiprocessing context.

Downstream damage

Killed workers held NATS msgs they never acked. Messages hit max_deliver=2 and exhausted. Server-side health-check then flipped ~201/408 images to failed. Job marked SUCCESS with 49% failure rate. Same cascade described in #143 and on the antenna side.

Why the existing mitigations didn't help

• persistent_workers=True is already on (datasets.py:444) — but shmem still accumulates per batch. Either persistent_workers isn't actually persistent under our conditions, or the leak is in collation/transfer not subprocess churn. Worth resolving in this fix.
• PR Fix memory leak in worker batch processing #119 added torch.cuda.empty_cache() between batches and cut growth ~52%, but residual ~3.4 MB/batch × hundreds of batches × 2 parallel workers easily reaches host-RAM limits within minutes on big jobs. Audit-B in LRU eviction for idle pipeline models + audit residual RSS growth after #118 #144 is now urgent — not deferrable.
• AMI_NUM_WORKERS=1 was the old-cloud mitigation when hosts had one worker process. With numprocs=2, doubled worst-case RSS exceeds the new hosts.

Proposed fix (this incident is concrete enough to act on)

Combine the cheapest two interventions that aren't in the codebase yet, in one PR:

1. multiprocessing_context="forkserver" (or "spawn") on the DataLoader in datasets.py:411–446. fork shares the parent's heap by COW; under high parent RSS this is how stale CUDA / pinned-memory state ends up in every subprocess and gets reported as shmem-rss. forkserver re-execs from a clean state per subprocess. Listed in both DataLoader subprocess deadlock: MainThread hangs forever on queue.get after shared-memory unlink failure #140 and DataLoader pipe FD leak: ~10k pipes/day accumulate, worker hits EMFILE silently after ~6 days uptime #145 fix sketches; never tried.
2. timeout=300 on the DataLoader. Converts the silent infinite hang on shm_unlink failure into a supervisor-restartable error. Zero risk to the happy path.

These are 5–10 LoC. PR #139's knob plumbing is orthogonal — can land independently.

What this incident does NOT settle

• Whether pin_memory=False (per the Clarify and document tuning parameters #138 analysis) would have helped here — needs an isolated ablation, since GPU utilization fixes #122 introduced pin_memory=True for measured GPU-utilization gains. The two facts haven't been compared on a small-image / HTTP-sourced workload yet.
• Whether forkserver will break large-model state pickling — needs a quick smoke test on each registered pipeline before rollout. Risk is low (DataLoader subprocs don't import models, only the dataset class) but worth verifying.

Happy to take the fix PR; this is a real fire and the incident gives a concrete reproducer (large-image batch on 68 GiB host with two parallel workers).