feat!: version 13 - dataset evaluators#9642

Draft

mikeldking wants to merge 39 commits intomainfrom

Commits on Nov 30, 2025

feat: dataset evaluators

mikeldking
authored and
RogerHYang
committed
ci: add ci for version-13

mikeldking
authored and
RogerHYang
committed
feat: dataset evaluators

mikeldking
authored and
RogerHYang
committed
fix: drop support for python 3.9 (#9818)
RogerHYang
committed
feat(evaluators): db migration for evaluator tables (#9960)

committed
feat(evaluators): mutations for playground evaluator selector (#10042)

axiomofjoy
and
RogerHYang
committed
feat: Create evaluator mutations with optional dataset_id (#10065)

cephalization
authored and
RogerHYang
committed
fix: ensure fields of polymorphic evaluator orm types are eagerly loaded (#10068)

axiomofjoy
authored and
RogerHYang
committed
feat: Evaluators creation page (#10054)

cephalization
authored and
RogerHYang
committed
fix(evaluators): persist choices (#10076)

axiomofjoy
authored and
RogerHYang
committed
feat: Collect all json path segments when flattening example keys (#10075)

cephalization
authored and
RogerHYang
committed
feat(evaluators): add evaluator select (#10063)

yfrigui2
authored and
RogerHYang
committed
feat: Reorganize new evaluator form (#10081)

cephalization
authored and
RogerHYang
committed
chore: Dummy playground evaluation payloads (#10102)

authored and
RogerHYang
committed
feat: Add examples route with examples table (#10123)

cephalization
authored and
RogerHYang
committed
feat: Add optional description field to new evaluator creation (#10132)

cephalization
authored and
RogerHYang
committed
feat: Improve rendering of dataset evals on playground (#10136)

cephalization
authored and
RogerHYang
committed
feat(evaluators): assign evaluator to dataset UI (#10135)

authored and
RogerHYang
committed
feat: add metadata to evaluator db table (#10139)
RogerHYang
committed
fix(evaluators): return annotation name in output config resolver (#10152)

axiomofjoy
authored and
RogerHYang
committed
feat(evaluators): evaluators update and delete mutations (#10128)

axiomofjoy
authored and
RogerHYang
committed
feat(evaluators): add annotation name to eval menu (#10156)

yfrigui2
authored and
RogerHYang
committed
feat: Add evaluators table to dataset evaluators page (#10157)

cephalization
authored and
RogerHYang
committed
fix: Fix import error on evaluator page (#10185)

cephalization
authored and
RogerHYang
committed
feat(evaluators): load in a default template for the evaluator that is useful (#10187)

mikeldking
authored and
RogerHYang
committed
fix: eslint errors

mikeldking
authored and
RogerHYang
committed
ci: add ci for 12 (#10196)

mikeldking
authored and
RogerHYang
committed
feat: persist tools with eval (#10220)

cephalization
authored and
RogerHYang
committed
fix(evaluators): add validation for llm evaluator prompts (#10193)

axiomofjoy
authored and
RogerHYang
committed
feat: Refactor evaluator form for usage in create and edit workflows (#10253)

cephalization
authored and
RogerHYang
committed
feat: playground eval select updates (#10163)

yfrigui2
authored and
RogerHYang
committed
only include dataset-specific evaluators in playground eval select (#10292)

yfrigui2
authored and
RogerHYang
committed
feat: add eval outputs to playground (#10263)

authored and
RogerHYang
committed
feat: Create distinct slideovers for evaluator use cases (#10303)

authored and
RogerHYang
committed
fix(evaluators): clean up evaluators rebase

axiomofjoy
authored and
RogerHYang
committed
feat: Implement builtin evaluators (#10308)

authored and
RogerHYang
committed
feat: enable deleting dataset evaluator (#10354)

yfrigui2
authored and
RogerHYang
committed
fix: fix evaluator config dialog layout (#10366)

yfrigui2
authored and
RogerHYang
committed
feat: Add input mapping support to built-in evaluators (#10355)

authored and
RogerHYang
committed