Skip to content

Conversation

@adriangb
Copy link
Contributor

Summary

This PR enhances the physical-expr projection handling with several improvements needed for better projection management in datasources.

Changes

  1. Add trait implementations:

    • Added PartialEq and Eq for ProjectionExpr
    • Added PartialEq and Eq for ProjectionExprs
  2. Add project_batch() method:

    • Efficiently projects RecordBatch with pre-computed schema
    • Handles empty projections correctly
    • Reduces schema projection overhead for repeated calls
  3. Fix update_expr() bug:

    • Bug: Previously returned None for literal expressions (no column references)
    • Fix: Now returns Some(expr) for both Unchanged and RewrittenValid states
    • Impact: Critical for queries like SELECT 1 FROM table where no file columns are needed
  4. Change from_indices() signature:

    • Changed from &SchemaRef to &Schema for consistency
  5. Add comprehensive tests:

    • test_merge_empty_projection_with_literal() - Reproduces roundtrip issue
    • test_update_expr_with_literal() - Tests literal handling
    • test_update_expr_with_complex_literal_expr() - Tests mixed expressions

Part of

This PR is part of #18627 - a larger effort to refactor projection handling in DataFusion.

Testing

All tests pass:

  • ✅ New projection tests
  • ✅ Existing physical-expr test suite
  • ✅ Doc tests

@github-actions github-actions bot added the physical-expr Changes to the physical-expr crates label Nov 14, 2025
@adriangb adriangb changed the title Enhance physical-expr projection handling Misc improvements to ProjectionExprs Nov 14, 2025
@adriangb adriangb force-pushed the projection-source-physexpr branch from 6100daf to 3f0dd4c Compare November 15, 2025 00:46
This PR adds trait implementations, a project_batch() method, and fixes
a bug in update_expr() for literal expressions. Also adds comprehensive tests.

Part of apache#18627
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

physical-expr Changes to the physical-expr crates

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant