Skip to content

Dataset-saving infrastructure phase 1#688

Merged
Cadair merged 68 commits intoDKISTDC:mainfrom
SolarDrew:filesaving
Mar 31, 2026
Merged

Dataset-saving infrastructure phase 1#688
Cadair merged 68 commits intoDKISTDC:mainfrom
SolarDrew:filesaving

Conversation

@SolarDrew
Copy link
Copy Markdown
Contributor

Closes #671

@codspeed-hq
Copy link
Copy Markdown

codspeed-hq bot commented Feb 9, 2026

Merging this PR will not alter performance

✅ 14 untouched benchmarks
🆕 2 new benchmarks

Performance Changes

Mode Benchmark BASE HEAD Efficiency
🆕 Simulation test_dataset_save[inversion] N/A 3 s N/A
🆕 Simulation test_dataset_save[large_visp_dataset] N/A 665.5 ms N/A

Comparing SolarDrew:filesaving (cf95e9b) with main (b280f21)1

Open in CodSpeed

Footnotes

  1. No successful run was found on main (126f423) during the generation of this report, so b280f21 was used instead as the comparison base. There might be some changes unrelated to this pull request in this report.

@SolarDrew
Copy link
Copy Markdown
Contributor Author

I'm sure there are other issues remaining but the bulk of the current error are due to slicing bugs in Inversion, which should be solved by #690

@SolarDrew
Copy link
Copy Markdown
Contributor Author

I've merged #690 and #695 for now so that the tests are actually testing the changes in this PR. Once those are merged I'll rebase and tidy up the history.

@SolarDrew
Copy link
Copy Markdown
Contributor Author

pre-commit.ci autofix

Copy link
Copy Markdown
Member

@Cadair Cadair left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

(Blocking because of git dependency)

Comment thread pytest.ini Outdated
Comment thread pyproject.toml Outdated
Comment thread dkist/dataset/dataset.py Outdated
Comment thread dkist/dataset/dataset.py Outdated
Comment thread dkist/dataset/dataset.py Outdated
Comment thread dkist/dataset/tiled_dataset.py Outdated
Comment thread dkist/io/asdf/converters/dataset.py Outdated
Comment thread dkist/io/asdf/resources/schemas/file_manager-1.1.0.yaml
Comment thread dkist/io/asdf/resources/schemas/tiled_dataset-1.4.0.yaml
Comment thread dkist/io/asdf/converters/dataset.py Outdated
Comment thread dkist/io/asdf/converters/dataset.py Outdated
Comment thread dkist/io/asdf/converters/file_manager.py Outdated
Comment thread dkist/io/asdf/entry_points.py
Comment thread dkist/io/dask/striped_array.py Outdated
Comment thread dkist/io/utils.py Outdated
Comment thread dkist/io/utils.py Outdated
Comment thread pyproject.toml Outdated
Comment thread dkist/io/utils.py Outdated
Comment thread pyproject.toml Outdated
Copy link
Copy Markdown
Member

@Cadair Cadair left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Think it still needs the 2nd changelog entry but not that important.

@SolarDrew SolarDrew enabled auto-merge (squash) March 31, 2026 14:43
@Cadair Cadair disabled auto-merge March 31, 2026 14:44
@Cadair Cadair merged commit 06643cd into DKISTDC:main Mar 31, 2026
32 of 34 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Phase 1 - save modified datasets without modified data

2 participants