feat(prepro): assign segment/subtype using nextclade sort (3/n) #5402
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
anna-parker
changed the title
3 tasks
anna-parker
force-pushed
the
sort-multi-path
branch
from
4966265 to
95813b2
Compare
anna-parker
force-pushed
the
multi-segment-submission-2
branch
from
ffac38a to
87fbe02
Compare
anna-parker
commented
Nov 17, 2025
9 tasks
anna-parker
commented
Nov 18, 2025
| <<: *preprocessingConfigFile | ||
| log_level: INFO | ||
| nextclade_dataset_name: community/hodcroftlab/enterovirus/enterovirus/linked | ||
| minimizer_index: "https://raw.githubusercontent.com/alejandra-gonzalezsanchez/loculus-evs/master/evs_minimizer-index.json" |
Contributor
Author
There was a problem hiding this comment.
@corneliusroemer idea: add a flag that we are multipath and that the genes need to have the subtype as a prefix
anna-parker
commented
Nov 18, 2025
3 tasks
anna-parker
changed the title
anna-parker
added a commit
that referenced
this pull request
Nov 19, 2025
resolves # to be merged into #5402 when it works ### Screenshot As prepro now assigns the segments the website does not know which entry in the original data corresponds to which segment - this updates prepro to also return a map with this information. ### PR Checklist - [ ] All necessary documentation has been adapted. - [ ] The implemented feature is covered by appropriate, automated tests. - [ ] Any manual testing that has been done is documented (i.e. what exactly was tested?) 🚀 Preview: https://improve-endpoints.loculus.org
anna-parker
force-pushed
the
sort-multi-path
branch
2 times, most recently
from
77f4b06 to
5e0cd35
Compare
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
anna-parker
force-pushed
the
sort-multi-path
branch
from
fbb92f0 to
8ecc3af
Compare
corneliusroemer
changed the title
anna-parker
force-pushed
the
multi-segment-submission-2
branch
from
87fbe02 to
a61ec9b
Compare
anna-parker
force-pushed
the
sort-multi-path
branch
2 times, most recently
from
6f073fc to
b3cd475
Compare
resolves # When submitting via the single page submission users now need to submit with the fastaHeader in the format `submissionId_segment` - i.e. the same requirements we have for the multi segment submission. - [ ] All necessary documentation has been adapted. - [ ] The implemented feature is covered by appropriate, automated tests. - [ ] Any manual testing that has been done is documented (i.e. what exactly was tested?) 🚀 Preview: https://edit-page-anya-2.loculus.org
… same as in ingest)
anna-parker
force-pushed
the
sort-multi-path
branch
from
ab2b5db to
c9b5f6c
Compare
anna-parker
merged commit c9afc61
into
multi-segment-submission-2
40 of 42 checks passed
anna-parker
added a commit
that referenced
this pull request
Dec 5, 2025
… refactor multi segment submission in backend and edit page and have prepro assign segments (#5382) resolves #4999 #4708, #4734, #5511 partially resolves #5392, #5185 (comment) includes work done in #5398 and #5402 This PR additionally fixes submission, subtype assignment and search for EVs and other multi-path organisms. ### BREAKING CHANGES When users submit to multi-segmented organisms and want to group multiple segments under one metadata entry they are now required to add an additional `fastaIds` column with a space -separated list of the `fastaId`s (fasta header IDs) of the respective sequences. If no `fastaIds` column is supplied the `submissionId` will be used instead and the backend will assume that (as in the single-segmented case) there is a one-to-one mapping of metadata `submissionId` to `fastaId`. This new submission structure was voted for in microbioinfo: https://microbial-bioinfo.slack.com/archives/CB0HYT53M/p1760961465729399 and discussed in https://app.nuclino.com/Loculus/Development/2025-10-20-Weekly-6d5fe89f-8ded-4286-b892-d215e0a498f6 (and in other meetings) Nextclade sort (uses a minimizer index for fast local alignment) or nextclade align (full sequence alignment to reference) will be used to assign segments/subtypes for all multi-segmented and multi-pathogen sequences (this is also done in ingest for grouping segments): ``` segment_classification_method: "minimizer" or "align" minimizer_url: <url_to_minimizer_index_used_by_nextclade_sort> ``` For organisms without a nextclade dataset we still allow the fasta headers to be used to determine the segment/subtype - entries must have the format `<submissionId>_<segmentName>` (as in current set up). As preprocessing now assigns segments it will return a map from the segment (or subtype) to the fastaId in the processedData, the map is called: `sequenceNameToFastaId`. This allows us to surface the segment assignment on the edit page. ### Nextclade Preprocessing pipeline config changes Instead of having a dictionary for the nextclade datasets and servers we make `nucleotideSequences` a dictionary where each item includes all information required to run nextclade. I.e. we change from: ``` nextclade_dataset_name: L: nextstrain/cchfv/linked/L M: nextstrain/cchfv/linked/M S: nextstrain/cchfv/linked/S nextclade_dataset_server: https://raw.githubusercontent.com/nextstrain/nextclade_data/cornelius-cchfv/data_output genes: [RdRp, GPC, NP] ``` to: ``` nextclade_sequence_and_datasets: - name: L nextclade_dataset_name: nextstrain/cchfv/linked/L nextclade_dataset_tag: <optional - was previously incorrectly placed on an organism level> nextclade_dataset_server: <optional overwrites nextclade_dataset_server for this seq> accepted_sort_matches: <optional, used for classify_with_nextclade_sort and require_nextclade_sort_match, if not given nextclade_dataset_name and name are used> gene_prefix: <optional, prefix to add to genes produced by nextclade run, e.g. nextclade labels genes as `AV1` but we expect `EV1_AV1`, here `EV1` would be the prefix > genes: [RdRp] - name: M nextclade_dataset_name: nextstrain/cchfv/linked/M genes: [GPC] - name: S nextclade_dataset_name: nextstrain/cchfv/linked/S genes: [NP] nextclade_dataset_server: https://raw.githubusercontent.com/nextstrain/nextclade_data/cornelius-cchfv/data_output segment_classification_method: <optional, default for multi segmented viruses is align - if you assign segments in ingest for grouping use the same option here as you use there e.g. "minimizer" or "align"> minimizer_url: <optional, url_to_minimizer_index_used_by_nextclade_sort> ``` ### Ingest Pipeline Config changes `minimizer_index` is changed to `minimizer_url` for consistency (can be used in ingest and preprocessing and should both be the same) ### Optional additional Config changes Limit the number of sequences the backend will accept per submission by using - should be added for multi-segmented organisms: ` submissionDataTypes: &defaultSubmissionDataTypes consensusSequences: true maxSequencesPerEntry: 1 ` ### Testing You can use pathoplexus/example_data#16 and pathoplexus/dev_example_data#2 for testing. ### PR Checklist - [x] Update values.schema.json and other READMEs - [x] add fastaId to commonMetadata (ensure it is downloaded in templates): #5561 - [x] Fix how genes are returned (will cause a config update): #5563 - [x] Improve prepro code (less duplication and more tests): #5554 - [x] ingest EVs as single segmented to ensure search works: #5511 - [x] keep tests for alignment NONE case - [x] Create a minimizer for tests using: https://github.com/loculus-project/nextclade-sort-minimizer-creator - [x] Any manual testing that has been done is documented: submission of EVs from test folder were submitted with the same fastaHeader as the submissionId -> this succeeded, additionally the submission of CCHF with a fastaID column in the metadata was tested (also in folder above), additionally revision of a segment was tested - [x] Have preprocessing send back a segment: fastaHeader mapping - ~add integration testing for full EV submission user journey~ -> will be done in a later PR - [x] improve CCHF minimizer (some segments are again not assigned) - [x] discuss if the originalData dictionary should be migrated (persistent DB has segmentName as key, now we have fastaHeader as key) -> decided against - [x] update PPX docs with new multi-segment submission format -> test PR here: pathoplexus/pathoplexus#759 - [x] update example data for demo 🚀 Preview: https://edit-page-anya.loculus.org --------- Co-authored-by: Cornelius Roemer <[email protected]> Co-authored-by: Fabian Engelniederhammer <[email protected]> Co-authored-by: Theo Sanderson <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
resolves #4847
Screenshot
Improves #4821, comes after #5398
You can use pathoplexus/dev_example_data#2 for testing.
Nextclade sort will be used to assign segments/subtypes for all aligned sequences:
For organisms without a nextclade dataset we still allow the fasta headers to be used to determine the segment/subtype - entries must have the format _ (as in current set up).
As preprocessing now assigns segments it will return a map from the segment (or subtype) to the fastaHeader in the processedData:
sequenceNameToFastaHeaderMap. This allows us to surface this assignment on the edit page.Prepro config changes
Instead of having a dictionary for the nextclade datasets and servers we make
nucleotideSequencesa list of sequences:Note the templates now also generate the genes list from the merged config.
PR Checklist
Future Work
🚀 Preview: https://sort-multi-path.loculus.org