Adding Euclid test data and infrastructure

[EiffL]

Summary

Adds the full shine.euclid package — a forward-modelling pipeline for Bayesian shear inference on Euclid VIS quadrant-level multi-exposure data. Given science exposures, a PSF grid, background maps, and a MER source catalog, the pipeline renders a differentiable scene model in JAX and runs MAP (or NUTS) inference to estimate per-source morphology and a shared gravitational shear (g1, g2).

What's included

New package: shine/euclid/ (~1,800 lines)

• config.py — Pydantic configuration: data paths, source selection (SNR, det_quality_flag filtering, size cuts), prior distributions, multi-tier stamp sizes, inference method
• data_loader.py — Reads quadrant FITS files (SCI/RMS/FLG extensions), PSF grids with bilinear interpolation, background maps, and MER catalogs. Computes per-source per-exposure pixel positions via WCS, local Jacobians, PSF stamps, and visibility flags. Filters sources outside all exposure footprints. Stores sky coordinates and exposure footprint corners for diagnostics
• scene.py — NumPyro generative model rendering Sersic galaxies convolved with spatially-varying PSFs via JAX-GalSim. Multi-tier stamp sizes (64/128/256 px) assigned per source by half-light radius, each tier rendered with its own jax.vmap to preserve parallelism. Standalone render_model_images() for post-inference visualization
• plots.py — 3-panel diagnostic figure (observed | SHINE model | chi residual) with separate pixel-level and residual masks (bright stars, ghosts, saturated stars)

Notebook: notebooks/euclid_vis_map.ipynb

End-to-end MAP fitting on 3 dithered VIS exposures (~630 sources), with:

• Model/residual comparison plots per exposure
• Source map colored by stamp tier
• Sky-coordinate view of exposure footprints and dither pattern

Tests: tests/test_euclid/ (15 tests)

• Data loader: PSF grid interpolation, exposure reading, WCS transforms, source selection (SNR, flag, size filters), ExposureSet assembly
• Scene: single-exposure rendering, multi-exposure model, multi-tier stamp rendering, render_model_images()
• Integration: end-to-end MAP inference on real data

Data: data/EUC_VIS_SWL/ (Git LFS)

3 dithered VIS exposures + background maps + PSF grid + MER catalog for quadrant 3-4-F.

Key design decisions

• Multi-tier stamps: Sources are assigned to 64/128/256 px stamp tiers based on 2 * (3 * HLR_pix + PSF_half). Each tier gets a separate vmap+scan rendering pass — no wasted computation on oversized stamps, full parallelism within each tier
• Catalog filtering: Uses MER det_quality_flag to exclude saturated, border, bright-star-contaminated, extended-object, and deblending-skipped sources (mask 0x78C); keeps blended/neighbor sources. Sources larger than the max stamp or outside all footprints are also excluded
• Background subtraction at load time: Background maps are subtracted during prepare_image_data(), not modelled during inference
• Consistent flux: flux_detection_total used everywhere (SNR filtering, sorting, ADU conversion)
• Residual masking: Pixel-level bad pixel mask (bit 0) on observed panel; broader mask including GHOST (bit 5), STARSIGNAL (bit 18), SATURATEDSTAR (bit 19) on residual panel only

[claude]

Code review

Found 1 issue:

configs/euclid_vis.yaml (lines 42-43)

Config field name mismatch

The YAML configuration uses field names that don't match the Pydantic model:

Issue:

• YAML uses galaxy_stamp_size (singular) at line 42, but the model expects galaxy_stamp_sizes (plural, a List[int]) - see config.py#L213
• YAML includes fft_size: 128 at line 43, but this field doesn't exist on EuclidInferenceConfig (FFT sizes are auto-computed as 2 * stamp_size)

Impact:
Since EuclidInferenceConfig uses Pydantic's default extra="ignore" behavior, these mismatched fields will be silently discarded if someone loads this YAML. The user would get the default galaxy_stamp_sizes: [64, 128, 256] instead of their intended single size of 64.

Suggested fix:
Replace lines 42-43 with:

galaxy_stamp_sizes: [64]

(Remove fft_size since it's auto-computed as 2 * stamp_size for each tier)

[EiffL]

@claude the tests are failing on github, probably because we need to change the github action workflows to download the git lfs part of the data. Can you look into t hat and fix it?

[claude]

Claude finished @EiffL's task in 2m 11s —— View job

---

CI test failures diagnosed - Git LFS configuration needed

Root cause

The tests are failing because Git LFS files aren't being downloaded during checkout. Instead of actual FITS files, the workflow gets small text "pointer files" containing:

version https://git-lfs.github.com/spec/v1
oid sha256:c2cc4da8b6d68d8212c69437b9...

When astropy tries to read these as FITS files, it fails with:

OSError: No SIMPLE card found, this file does not appear to be a valid FITS file.

This affects 12 tests that need the Euclid data in data/EUC_VIS_SWL/.

The fix

Add LFS support to .github/workflows/test.yml line 17-19:

steps:
- uses: actions/checkout@v4
  with:
    lfs: true

This tells the checkout action to run git lfs pull automatically.

Why I can't push this fix

GitHub requires special workflows permission for Apps to modify workflow files (security measure). The push was rejected with:

refusing to allow a GitHub App to create or update workflow `.github/workflows/test.yml` 
without `workflows` permission

How to fix it

Option 1 - Quick manual fix:

1. Edit .github/workflows/test.yml on GitHub
2. Change lines 17-19 from:

   - uses: actions/checkout@v4

   to:

   - uses: actions/checkout@v4
     with:
       lfs: true

3. Commit directly to the euclid branch

Option 2 - Grant workflows permission:
If you update the GitHub App permissions to include workflows: write, I can push fixes to workflow files automatically in the future.

---