Currently, when requesting columns or rows, the full result is returned as a slice, which may result in high memory use for large datasets.
Proposal:
- Replace returning a complete slice with an iterator or on-demand function to yield columns or rows progressively.
- This approach avoids loading all data at once into memory, conserving resources and improving performance when working with large CSV files.
- The new API should:
- Be easy to adopt for clients expecting the current slice-style output.
- Preserve or improve performance characteristics.
- Prevent out-of-memory crashes on large or streaming inputs.
Benefits:
- Reduced peak memory usage
- Faster initial responses when using only part of the result
- Scales better for large or streaming CSV sources
Acceptance Criteria:
- Replace or supplement the current slice-returning interface with an efficient iterator-based or on-demand approach for columns or rows extraction.
- Adequate tests and documentation for new interface and migration guidance.