Skip to content

Replace slice of columns or rows return with iterator or on-demand function for memory efficiency #11

@DISTREAT

Description

@DISTREAT

Currently, when requesting columns or rows, the full result is returned as a slice, which may result in high memory use for large datasets.

Proposal:

  • Replace returning a complete slice with an iterator or on-demand function to yield columns or rows progressively.
  • This approach avoids loading all data at once into memory, conserving resources and improving performance when working with large CSV files.
  • The new API should:
    • Be easy to adopt for clients expecting the current slice-style output.
    • Preserve or improve performance characteristics.
    • Prevent out-of-memory crashes on large or streaming inputs.

Benefits:

  • Reduced peak memory usage
  • Faster initial responses when using only part of the result
  • Scales better for large or streaming CSV sources

Acceptance Criteria:

  • Replace or supplement the current slice-returning interface with an efficient iterator-based or on-demand approach for columns or rows extraction.
  • Adequate tests and documentation for new interface and migration guidance.

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions