Support recursive design: additive multi-file MTHDS libraries

[lchoquel]

Summary

Adds support for recursive / additive multi-file MTHDS library construction: a same-domain library can now be authored as separate, additive .mthds files (forward-declared headers + separate definitions) instead of one monolithic file. This unblocks parallel, top-down method construction.

Merge-time rules

• Pipe signature ↔ concrete reconciliation. A PipeSignature (forward declaration / "header") and a concrete pipe with the same code reconcile — the concrete satisfies the signature instead of raising a duplicate-code error. Contracts are compared by normalized concept identity (bare ≡ domain-qualified, Text ≡ native.Text, structural multiplicity), replacing the old raw-string contract_equals. Two concrete pipes, or genuinely different contracts, still error.
• Cross-file concept resolution. Concept references resolve against the merged library instead of per file, so a concept declared once can be referenced by bare code from sibling files of the same domain (same directory batch or across separate load batches). A concept declared twice still raises ConceptLibraryError; an unresolvable reference now surfaces as a structured validation error instead of a raw traceback.
• Order-independent domain-metadata merge. Domain description / system_prompt now merge order-independently across same-domain files: a membership-only file that omits them defers to whichever file declares them, instead of warning and racing on filesystem load order. The root header always wins over siblings that carry only domain = "...". Two files declaring different non-empty values still keep the first and warn. Implemented as a shared merge_domain_metadata_field helper wired into both the crate factory and the runtime DomainLibrary.

Validate surface

• pending_signatures on a successful lenient validate. A successful validate bundle (notably --allow-signatures) now reports the library-wide set of pipes still declared as PipeSignature — surfaced as a pending_signatures array in the agent-CLI / builder JSON envelope and a "Pending signatures" markdown section, so a top-down build sees exactly what remains.

Test plan

• make agent-check — clean (ruff, plxt, pyright, mypy).
• make agent-test — full suite green.
• New unit coverage under tests/unit/pipelex/libraries/ for signature/concrete reconciliation, normalized contract conformance, cross-file concept references, and the domain-metadata merge matrix (both load orders, same-value, conflict; across description + system_prompt, warnings asserted via caplog).

Documentation

Docs-site coverage added this cycle (reference level — the full top-down walkthrough is deferred, see Documentation Debt):

• docs/building-methods/pipes/signature-pipes.md — new note: a signature and its concrete definition can live in separate sibling files of the same domain; loading merges them and they reconcile when contracts match. Points to the validate output.
• docs/tools/cli/validate.md — new "Pending signatures" tip under Validate Bundle: a successful run reports pending_signatures (library-wide list of still-unimplemented PipeSignature headers), most useful on --allow-signatures.
• docs/tools/cli/agent-cli.md — extended the signature note with the pending_signatures envelope field (JSON array + markdown section).

CHANGELOG.md [Unreleased] already carries the full feature entry (additive construction, reconciliation, cross-file concept resolution, domain-metadata merge, pending_signatures). pipelex/cli/agent_cli/CLAUDE.md documents the pending_signatures envelope field. Design records live in wip/recursivity/.

Documentation Debt

• Critical → deferred by decision. No end-to-end "build a method top-down across multiple files" how-to / tutorial on the docs site yet. The runtime capability is complete, but the orchestrator that drives the workflow (mthds-vibe / mthds-signature-expander in mthds-plugins) is not built yet, so a detailed walkthrough now would risk drifting from the real flow. Write it once that orchestrator lands. Suggest a docs-debt label.
• Reference-only coverage (acceptable for now): signature↔concrete reconciliation, cross-file concept resolution, and the domain-metadata merge rule are covered in CHANGELOG + wip/recursivity/ but have no dedicated how-to. Same trigger to expand: when the orchestrator ships.

🤖 Generated with Claude Code

---

Summary by cubic

Adds recursive, additive multi-file .mthds libraries for the same domain. Signatures reconcile with concretes, cross-file refs resolve across files/batches, and validate bundle returns pending_signatures (intentionally omitted by validate-all); works for dependencies and the concepts-only loader.

• New Features

  • Signature ↔ concrete reconciliation with normalized contract matching (bare ≡ domain-qualified, Text ≡ native.Text, multiplicity). Conflicting concretes or mismatched contracts still error.
  • Cross-file same-domain concept/pipe reference resolution against the merged library (within a batch and across batches). Unresolvable refs return a structured validation error.
  • Order-independent merge of domain description and system_prompt. Membership-only files defer; conflicting non-empty values keep the first and warn.
  • validate bundle returns pending_signatures in JSON and renders a "Pending signatures" section; validate-all omits it by design.

• Bug Fixes

  • Removed the per-file pipe-reference validator so qualified same-domain refs resolve post-merge like bare refs; missing refs are caught by merged validation.
  • Concepts-only loader now validates at the library level and batches sibling .mthds files so same-domain refs resolve regardless of filename order.
  • Dependency loading routes through LibraryCrateFactory, reconciling multi-file PipeSignature + concrete packages; genuine duplicates raise.
  • Canonical native-concept check in concept-ref validation (Text or native.Text only), fixing false skips for qualified refs like my_domain.Text.
  • Recorded deferred P3 polish in wip/recursivity/review-followups.md (docs only).

^{Written for commit b77726b. Summary will update on new commits.}

[chatgpt-codex-connector]

You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard.

[greptile-apps]

Greptile Summary

This PR adds additive multi-file .mthds library construction. The main changes are:

• Reconciles PipeSignature headers with concrete pipe definitions.
• Resolves same-domain concept references against the merged library.
• Merges domain metadata without depending on file load order.
• Reports remaining pending_signatures from successful bundle validation.

Confidence Score: 5/5

This looks safe to merge.

• No blocking issues found in the changed code.

_{Reviews (5): Last reviewed commit: "fix(library): use canonical native-conce..." | Re-trigger Greptile}

[cubic-dev-ai]

No issues found across 28 files

<sub>Tip: cubic can generate docs of your entire codebase and keep them up to date. Try it here.

Re-trigger cubic</sub>

[lchoquel]

Review round 1 — addressed

greptile P1 (load_libraries_concepts_only one-file batches → order-dependent concept-ref validation): fixed in afd9217. load_libraries_concepts_only now collects all sibling .mthds files and runs a single load_concepts_only_from_blueprints batch — mirroring load_concepts_only_from_directory — so same-domain concept references resolve against the merged set regardless of directory load order. Added an order-independence regression test (test_load_concepts_only_library_dir_sibling_cross_reference, where the referencing file sorts before its declaration, so the old per-file loader would fail).

cubic: no issues found — thanks.

Also rewrote TODOS.md into a reviewer-facing guide describing the four merge-time changes, the file map, and the design decisions.

make agent-check clean, make agent-test green.

@greptileai @cubic-dev-ai please re-review.

[cubic-dev-ai]

Review round 1 — addressed

greptile P1 (load_libraries_concepts_only one-file batches → order-dependent concept-ref validation): fixed in afd9217. load_libraries_concepts_only now collects all sibling .mthds files and runs a single load_concepts_only_from_blueprints batch — mirroring load_concepts_only_from_directory — so same-domain concept references resolve against the merged set regardless of directory load order. Added an order-independence regression test (test_load_concepts_only_library_dir_sibling_cross_reference, where the referencing file sorts before its declaration, so the old per-file loader would fail).

cubic: no issues found — thanks.
...

@lchoquel I have started the AI code review. It will take a few minutes to complete.

[cubic-dev-ai]

1 issue found across 32 files

Prompt for AI agents (unresolved issues)

Check if these issues are valid — if so, understand the root cause of each and fix them. If appropriate, use sub-agents to investigate and fix each issue separately.


<file name="pipelex/builder/operations/validate_ops.py">

<violation number="1" location="pipelex/builder/operations/validate_ops.py:72">
P2: validate_all is missing pending_signatures in its return dict, inconsistent with the other three validation surfaces that now include it. The CLI JSON envelope for --all won't report pending signatures.</violation>
</file>

<sub>Tip: cubic can generate docs of your entire codebase and keep them up to date. Try it here.

Re-trigger cubic</sub>

[cubic-dev-ai]

P2: validate_all is missing pending_signatures in its return dict, inconsistent with the other three validation surfaces that now include it. The CLI JSON envelope for --all won't report pending signatures.

Prompt for AI agents

Check if this issue is valid — if so, understand the root cause and fix it. At pipelex/builder/operations/validate_ops.py, line 72:

<comment>validate_all is missing pending_signatures in its return dict, inconsistent with the other three validation surfaces that now include it. The CLI JSON envelope for --all won't report pending signatures.</comment>

<file context>
@@ -69,6 +69,7 @@ async def validate_bundle_file(
         "bundle_path": str(bundle_path),
         "validated_pipes": build_validated_pipes(result.dry_run_result),
         "total_pipes": len(result.dry_run_result),
+        "pending_signatures": result.pending_signatures,
     }
 
</file context>

[lchoquel]

Review round 2 — addressed

cubic P2 (validate_all missing pending_signatures): intentional, not a gap — won't add it. pending_signatures is a per-bundle, top-down-build nudge ("which forward-declared headers are still unimplemented in this bundle"), surfaced only by validate bundle. This scoping is already documented in pipelex/cli/agent_cli/CLAUDE.md ("a non-blocking nudge on a successful lenient --allow-signatures run"). validate --all is a whole-library sweep, not a build step (and validate pipe omits it for the same reason). Adding it there would also force reshaping the shared acquire_and_validate, which tears its library down before returning. Recorded the rationale as a comment at both validate-all surfaces in f19277b so it isn't re-flagged.

greptile: 5/5, no blocking issues — thanks.

@greptileai @cubic-dev-ai please re-review.

[cubic-dev-ai]

Review round 2 — addressed

cubic P2 (validate_all missing pending_signatures): intentional, not a gap — won't add it. pending_signatures is a per-bundle, top-down-build nudge ("which forward-declared headers are still unimplemented in this bundle"), surfaced only by validate bundle. This scoping is already documented in pipelex/cli/agent_cli/CLAUDE.md ("a non-blocking nudge on a successful lenient --allow-signatures run"). validate --all is a whole-library sweep, not a build step (and validate pipe omits it for the same reason). Adding it there would also force reshaping the shared acquire_and_validate, which tears its library down before returning. Recorded the rationale as a comment at both validate-all surfaces in f19277b so it isn't re-flagged.

greptile: 5/5, no blocking issues — thanks.
...

@lchoquel I have started the AI code review. It will take a few minutes to complete.

[cubic-dev-ai]

1 issue found across 32 files

Prompt for AI agents (unresolved issues)

Check if these issues are valid — if so, understand the root cause of each and fix them. If appropriate, use sub-agents to investigate and fix each issue separately.


<file name="pipelex/builder/operations/validate_ops.py">

<violation number="1" location="pipelex/builder/operations/validate_ops.py:72">
P2: validate_all is missing pending_signatures in its return dict, inconsistent with the other three validation surfaces that now include it. The CLI JSON envelope for --all won't report pending signatures.</violation>
</file>

<sub>Tip: cubic can generate docs of your entire codebase and keep them up to date. Try it here.

Re-trigger cubic</sub>

[lchoquel]

Review round 3 — addressed

cubic P1 (concept_reference_validation.py:59 native check too broad): valid — fixed in f39ce6f. The check used ref.local_code in native_codes, so a same-domain qualified ref sharing a native local code (e.g. my_domain.Text) was wrongly treated as native and skipped, suppressing the undeclared-reference error — even though that ref actually fails at resolution (ConceptLibrary.get_required_concept_from_concept_ref_or_code uses the canonical check). Switched to NativeConceptCode.is_native_concept_ref_or_code (accepts only bare Text / native.Text), now consistent with both resolution and contract_match. Added a regression test (test_same_domain_qualified_ref_sharing_native_local_code_is_flagged).

Good catch. make agent-check clean, make agent-test green.

@greptileai @cubic-dev-ai please re-review.

[cubic-dev-ai]

Review round 3 — addressed

cubic P1 (concept_reference_validation.py:59 native check too broad): valid — fixed in f39ce6f. The check used ref.local_code in native_codes, so a same-domain qualified ref sharing a native local code (e.g. my_domain.Text) was wrongly treated as native and skipped, suppressing the undeclared-reference error — even though that ref actually fails at resolution (ConceptLibrary.get_required_concept_from_concept_ref_or_code uses the canonical check). Switched to NativeConceptCode.is_native_concept_ref_or_code (accepts only bare Text / native.Text), now consistent with both resolution and contract_match. Added a regression test (test_same_domain_qualified_ref_sharing_native_local_code_is_flagged).

Good catch. make agent-check clean, make agent-test green.
...

@lchoquel I have started the AI code review. It will take a few minutes to complete.

[cubic-dev-ai]

No issues found across 32 files

<sub>Tip: cubic can generate docs of your entire codebase and keep them up to date. Try it here.

Re-trigger cubic</sub>

[lchoquel]

Pre-landing review (gstack /review) — verdict: ship

Ran an independent, fresh-context review pass on top of the bot rounds: three reviewers (adversarial / chaos-engineer, test-coverage, maintainability) each read the full diff against dev. (Cross-model Codex pass skipped — local service_tier config error, unrelated to this PR.)

Scope: CLEAN. Intent (additive multi-file libraries: reconciliation, cross-file concept resolution, domain-metadata merge, pending_signatures) matches the diff exactly — no scope creep, no missing requirements.

Correctness: no bugs found. The adversarial reviewer explicitly cleared the high-risk areas: the ConceptLibraryError/PipeLibraryError → LibraryError reparenting doesn't shadow any other except arm (PipeNotFoundError re-raised first; PipeRunError/DryRunError/SignaturesNotAllowedError/PipeFactoryError/PipelexInterpreterError are not LibraryErrors); fingerprint is stable across load orders; load_from_crate ordering and the dependency path are sound; removing the per-file validators leaves no silent hole (dangling refs still caught at load/dry-run). Verdict: ship.

Tests: strong coverage. Reconciliation matrix, contract-match normalization (incl. the [] vs [1] / True == 1 guard), cross-batch + order-independent concepts-only resolution, domain-metadata merge matrix, error reparenting → structured ValidateBundleError, and pending_signatures content are all exercised.

Maintainability: clean, no over-engineering. The new modules (contract_match, collision_messages, domain_metadata_merge, concept_reference_validation) are small, single-purpose, pulled out at the right seams.

Only P3 polish surfaced (docstring accuracy, an advisory-description silent-drop note, one low-confidence test-coverage gap) — no bugs. Per "no over-engineering / defer doubts," these are recorded in wip/recursivity/review-followups.md rather than churned in now.

PR is ready to land once CI is green on the latest commit.