RFC: Configurable maturity scoring system

[stuart-gano]

RFC: Configurable Maturity Scoring System

Status: Draft — requesting feedback
Author: Stuart Gano
Branch: configurable-maturity-scoring

---

Problem

The maturity scoring logic is hardcoded in scanner.py and score_all_spaces.py. If an admin wants to adjust scoring weights, disable a criterion, or change stage thresholds, it requires a code change and redeployment. Different workspaces may have different priorities (e.g., some care more about Unity Catalog compliance, others about SQL coverage).

Proposal

Make the maturity scoring system fully configurable by workspace admins through:

1. YAML default config (maturity_config_default.yaml) — ships with the app, defines the 5-stage model with 16 criteria, point weights, and stage thresholds
2. Lakebase admin overrides — partial overrides stored in Lakebase, merged at scoring time
3. Admin UI — new "Scoring Config" tab in Admin Dashboard for editing without touching code
4. Registered check functions — Python functions keyed by criterion ID, decoupled from config

Architecture

maturity_config_default.yaml    ← Source of truth (shipped)
        │
        ▼
  get_default_config()          ← Loads YAML once, deep copies
        │
        ├──► merge_config()     ← Merges with Lakebase overrides
        │         │
        ▼         ▼
  get_active_config()           ← Returns merged config
        │
        ▼
  calculate_score(space, config) ← Iterates criteria, runs _CHECKS[id]
        │
        ▼
  { score, maturity, breakdown, criteria_results, findings, next_steps }

Merge Strategy

• Stages: Replaced wholesale if overridden (they're a coherent set)
• Criteria: Merged by id — override fields win, unmentioned fields preserved, new IDs appended
• Empty overrides = use defaults (reset path)

What Admins Can Configure

Setting	How	Example
Point weights	Change points per criterion	tables_attached: 10 → 15
Enable/disable	Toggle enabled flag	Disable sql_functions
Stage thresholds	Change range bounds	Basic starts at 25 instead of 30
Custom criteria	Add new criterion ID	Requires matching @_register in code

Security

Check functions are registered Python functions, not eval'd expressions. The YAML config only contains metadata (IDs, points, descriptions). Adding a new custom criterion ID requires a corresponding @_register("id") decorator in scanner.py — there's no arbitrary code execution path.

Changes

Backend

File	Change
maturity_config_default.yaml	New — 5 stages, 16 criteria with weights/types/descriptions
services/maturity_config.py	New — Config loader, merge logic, Lakebase persistence
services/scanner.py	Rewritten — Config-driven engine with @_register check functions
models.py	Added CriterionResult, updated ScoreBreakdown to 5-stage keys
routers/admin.py	Added GET/PUT /maturity-config, POST /maturity-config/reset

Frontend

File	Change
MaturityConfigEditor.tsx	New — Full config editor with stage thresholds, criteria table, save/reset
AdminDashboard.tsx	Added tab navigation (Overview / Scoring Config)
IQScoreTab.tsx	Updated breakdown bars to 5-stage model
types/index.ts	Added MaturityConfig, MaturityConfigStage, MaturityConfigCriterion
lib/api.ts	Added getMaturityConfig, updateMaturityConfig, resetMaturityConfig

Other

File	Change
jobs/score_all_spaces.py	New — Standalone batch scoring notebook
docs/genie-space-maturity.md	New — Documents stages, criteria, admin API
tests/test_maturity_scoring.py	New — 41 tests: scoring engine, checks, integration
tests/test_maturity_config.py	New — 19 tests: YAML integrity, merge logic

Open Questions

1. Stage threshold validation — Currently the UI lets you set non-contiguous ranges (e.g., Nascent 0-25, Basic 30-49 leaving a gap). Should we enforce contiguous ranges in the UI, the API, or both?

2. Custom criteria workflow — An admin can add a new criterion ID in the UI, but it won't score anything until a @_register function is added in code. Should we surface a warning for unregistered criterion IDs? Or is this an acceptable admin-only footgun?

3. Score history after config change — If an admin changes weights, historical scores become stale. Should we re-score all spaces on config change? Add a "config version" to scan results? Or just let the next scan pick up the new config?

4. Batch job config — score_all_spaces.py currently has its own inline scoring logic (mirrors scanner.py). Should it import from the app's scanner instead, or stay self-contained for notebook portability?

Test Plan

• uv run pytest tests/test_maturity_scoring.py tests/test_maturity_config.py — 60/60 passing
• npm run build — frontend compiles clean
• Manual: Admin Dashboard → Scoring Config tab loads config
• Manual: Adjust criterion points → Save → re-scan → score reflects change
• Manual: Reset Defaults → config returns to original
• Manual: Disable a criterion → re-scan → criterion excluded from results

[hiydavid]

Opened a new epic to track this feature (see #199 )