feat: Add support for repodata patching in rattler-index, fix silent failures #1129
Conversation
pavelzw
changed the title
| )) | ||
| }) | ||
| .collect::<Vec<_>>(); | ||
| try_join_all(tasks).await?; |
There was a problem hiding this comment.
this was silently failing before because it only checked whether the join handles were okay but not whether the actual return values of the tasks were
| pb.abandon_with_message(format!( | ||
| "{} {}", | ||
| console::style("Failed to index").red(), | ||
| console::style(subdir.as_str()).dim() | ||
| )); |
There was a problem hiding this comment.
this is not perfect as the progress bar can get overwritten by other tasks that are still executed after this failure. haven't seen a nice way to avoid this behavior
i think we can keep it for now like this since the error messages are printed anyway and i expect this script to be executed in non-interactive environments anyway most of the time
| /// The maximum number of packages to process in-memory simultaneously. | ||
| /// This is necessary to limit memory usage when indexing large channels. | ||
| #[arg(long, default_value = "128", global = true)] | ||
| #[arg(long, default_value = "32", global = true)] |
There was a problem hiding this comment.
when testing it out with 128 on aws s3, i regularly got errors that weren't there when limiting ourselves to 32 maximum concurrent downloads
|
the failing JS tests don't look related, right? |
|
@wolfv @baszalmstra this is ready for review |
|
Just FYI, a PatchInstructions v2 file will be landing soon: conda/ceps#108 |
|
Will keep this in mind when implementing sharding in rattler-index 👍🏻 |
2 participants
Description
xref #1096