fix(orchestration): restore agent attribution in director summary#554
fix(orchestration): restore agent attribution in director summary#554ashutoshrana wants to merge 2 commits into
Conversation
Fixes THU-MAIC#511 — three symptoms (off-topic replies, role confusion, premature END) traced to a single root cause: summarizeConversation() collapsed all messages to generic [User]/[Assistant] labels, stripping the [AgentName]: prefix that convertMessagesToOpenAI() adds when re-encoding peer agent turns as role:'user'. The director could not distinguish a substantive human challenge from a brief agent acknowledgment, causing it to emit END on unresolved questions. Changes: - conversation-summary.ts: detect [AgentName]: prefix on role:'user' messages and label them [Agent: Name] instead of [User]; label genuine human messages [Student (Human)]; add extractLastHumanMessage() to surface the last unaddressed human turn - director-prompt.ts: inject an Open Student Question section when a human message is present, blocking premature END while the question is unresolved - director-graph.ts: pass openaiMessages to buildDirectorPrompt (reuses the already-computed variable, no extra cost) - director/system.md: wire {{openStudentQuestionSection}} placeholder so the section is actually injected into the director prompt - tests: 38 new tests for conversation-summary, 3 regression tests for template injection; 331 total passing
|
Thanks for working on this fix and for adding tests around the attribution issue. I do see a problem with the current approach, though: it relies on regex-matching the rendered message text ([Name]: ...) to infer whether a message came from a human or from another agent. In the real runtime path, that is not a reliable source of truth. I think the fix should use the real message role/source from structured metadata (originalRole, agentId, etc.) instead of parsing display text. In other words, the director should make this decision based on the actual speaker identity, not on a |
|
Hi @cosarah — you're right, and the issue is more fundamental than just occasional misclassification. After tracing the full data flow: The fix uses metadata as you suggested:
Happy to push an updated commit if this direction looks right — or adjust the approach if you'd prefer a different pattern. |
|
This direction looks right to me. Keeping sender prefixes in the conversation context is still useful for attribution, but the human vs. agent distinction should come from the real structured role/source metadata rather than regex-matching formatted text. Using the pre-conversion message metadata for Open Student Question and limiting prefix handling to presentation/readability feels like the correct fix. |
Summary
Fixes #511
Three reported symptoms — off-topic replies, role confusion, and premature END — all trace to one root cause:
summarizeConversation()collapsed everyrole: 'user'message to the same generic label, stripping the[AgentName]:prefix thatconvertMessagesToOpenAI()adds when re-encoding peer agent turns. The director LLM had no way to distinguish a substantive human challenge from a brief agent acknowledgment, so it would emitENDon unresolved student questions.Root Cause
```
Teacher: "The Tiananmen gate is axisymmetric..."
Xiao Ming: "[Xiao Ming]: Yes, symmetric from the front!" ← re-encoded as role:'user'
Student: "Wait — the gate is 3D. Can 3D objects really be axisymmetric?"
```
Before this fix, the director summary showed both as identical
[User]lines. The director read the second as a follow-up that appeared resolved and emittedEND, leaving the student's real challenge unanswered.Changes
lib/orchestration/summarizers/conversation-summary.tsAGENT_PREFIX_REto detect the[AgentName]:prefix onrole:'user'messagessummarizeConversation()now labels peer agent turns[Agent: Name]and genuine human messages[Student (Human)]— the director can distinguish themextractLastHumanMessage()to retrieve the last unaddressed human turn (skips agent-prefixed messages)lib/orchestration/director-prompt.tsbuildOpenStudentQuestionSection()— surfaces the most recent unresolved human question explicitly in the director's system prompt with a clear instruction not to emitENDwhile it remains unansweredopenAIMessagesparameter (10th, backward-compatible)lib/orchestration/director-graph.tsopenaiMessagesvariable tobuildDirectorPrompt()— no extra computation, one-line changelib/prompts/templates/director/system.md{{openStudentQuestionSection}}placeholder so the new section is actually injected into the director prompt. Without this,interpolateVariables()silently drops any variable with no matching placeholder — the fix would be a no-op.Tests
tests/orchestration/conversation-summary.test.ts— 38 new tests covering role label correctness, the exact Multi-agent dialogue: off-topic replies, role confusion, premature discussion end #511 scenario replay, content truncation,maxMessagesslicing, and allextractLastHumanMessagecasestests/prompts/templates.test.ts— 3 regression tests confirming{{openStudentQuestionSection}}is injected when a human message is present and absent otherwise331 tests passing, zero lint errors, zero new TypeScript errors.
Scope
This fix is limited to the director summary and prompt layer. No changes to graph topology, LangGraph state,
convertMessagesToOpenAI(), or any agent-facing logic.