Skip to content

Use dagster for reproducible orchestration / fetch all targets from official sources #267

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
nwoodruff-co wants to merge 4 commits into
base: main
Choose a base branch
from
Open

Use dagster for reproducible orchestration / fetch all targets from official sources #267

nwoodruff-co wants to merge 4 commits into from

Conversation

@nwoodruff-co
Copy link
Collaborator

@nwoodruff-co nwoodruff-co commented Jan 28, 2026

  • Moves to a dagster pipeline (we'll use Modal as the backend)
  • Moves all targets to be automatically fetched from official sources via either APIs (DWP Stat-Xplore, e.g.) or from downloading XLSX/CSV files from government websites

@MaxGhenis MaxGhenis requested a review from baogorek January 28, 2026 19:34
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@baogorek baogorek Awaiting requested review from baogorek

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@nwoodruff-co @nikhilwoodruff