Skip to content

[experimental] Columnar query engine initial implementation #1394

@albertlockett

Description

@albertlockett

In #1342 we implemented a version of our columnar query engine (which I'm still somewhat hacking on). I thought it would be good to have an issue to track future work on this including feature gaps, performance optimizations, future ideation/experimentation, etc.

(note: list below is still a work in progress. what's documented below is not a comprehensive list of TODOs. More may be added to it as future and for some of these we can/will make dedicated issues).

Feature gaps:

  • Additional signal support
    • Metrics
    • Traces
  • Attribute transformations:
    • Set attribute
      • from literals(extend attributes["X"] ="Y")
      • from other expressions (things like extend attributes["event"] = <some_field> / some_func(<some_field>) / <etc.>)
    • Rename attributes (project-rename)
    • Drop attributes (project-away)
  • Filtering:
    • Literal on RHS of binary expression not supported (where "WARN" == severity_text does not work)
    • Filter by body

Plan Construction & Execution: (I will add issues to expand on the bullet points in near-future)

Further Exploration

  • supporting nested streams pipelines (iterating metrics datapoints)
  • filtering by nested streams (filter metrics by datapoints)
  • support fork (copy data and send down two pipelines)

Sub-issues

Metadata

Metadata

Assignees

No one assigned

    Labels

    query-engineQuery Engine / Transform related tasksquery-engine-columnarColumnar query engine which uses DataFusion to process OTAP Batches

    Projects

    Status

    No status

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions