Optimizations using batch mapping#485

Closed

aryanorpe wants to merge 10 commits intoinstructlab:mainfrom

aryanorpe:batch-map

Commits on May 3, 2025

Using batch mapping for the load_and_validate_dataset function which uses data.map in process_messages_into_input_ids function in data_process.py
aryanorpe
committed
Ran 'make fix' command to fix linter error.
aryanorpe
committed
Added batch mapping for the `process_samples` function in `process_messages_into_input_ids`.
aryanorpe
committed

Commits on May 17, 2025

Update src/instructlab/training/data_process.py

aryanorpe
and
JamesKunstle
authored

Commits on Jul 15, 2025

Merge branch 'main' into batch-map
RobotSail
authored