Skip to content

fix(orchestration): restore agent attribution in director summary#554

Open
ashutoshrana wants to merge 2 commits into
THU-MAIC:mainfrom
ashutoshrana:fix/multi-agent-dialogue-attribution
Open

fix(orchestration): restore agent attribution in director summary#554
ashutoshrana wants to merge 2 commits into
THU-MAIC:mainfrom
ashutoshrana:fix/multi-agent-dialogue-attribution

Conversation

@ashutoshrana
Copy link
Copy Markdown

Summary

Fixes #511

Three reported symptoms — off-topic replies, role confusion, and premature END — all trace to one root cause: summarizeConversation() collapsed every role: 'user' message to the same generic label, stripping the [AgentName]: prefix that convertMessagesToOpenAI() adds when re-encoding peer agent turns. The director LLM had no way to distinguish a substantive human challenge from a brief agent acknowledgment, so it would emit END on unresolved student questions.

Root Cause

```
Teacher: "The Tiananmen gate is axisymmetric..."
Xiao Ming: "[Xiao Ming]: Yes, symmetric from the front!" ← re-encoded as role:'user'
Student: "Wait — the gate is 3D. Can 3D objects really be axisymmetric?"
```

Before this fix, the director summary showed both as identical [User] lines. The director read the second as a follow-up that appeared resolved and emitted END, leaving the student's real challenge unanswered.

Changes

lib/orchestration/summarizers/conversation-summary.ts

  • Added AGENT_PREFIX_RE to detect the [AgentName]: prefix on role:'user' messages
  • summarizeConversation() now labels peer agent turns [Agent: Name] and genuine human messages [Student (Human)] — the director can distinguish them
  • Added extractLastHumanMessage() to retrieve the last unaddressed human turn (skips agent-prefixed messages)

lib/orchestration/director-prompt.ts

  • Added buildOpenStudentQuestionSection() — surfaces the most recent unresolved human question explicitly in the director's system prompt with a clear instruction not to emit END while it remains unanswered
  • Added optional openAIMessages parameter (10th, backward-compatible)

lib/orchestration/director-graph.ts

  • Passes the already-computed openaiMessages variable to buildDirectorPrompt() — no extra computation, one-line change

lib/prompts/templates/director/system.md

  • Added {{openStudentQuestionSection}} placeholder so the new section is actually injected into the director prompt. Without this, interpolateVariables() silently drops any variable with no matching placeholder — the fix would be a no-op.

Tests

  • tests/orchestration/conversation-summary.test.ts — 38 new tests covering role label correctness, the exact Multi-agent dialogue: off-topic replies, role confusion, premature discussion end #511 scenario replay, content truncation, maxMessages slicing, and all extractLastHumanMessage cases
  • tests/prompts/templates.test.ts — 3 regression tests confirming {{openStudentQuestionSection}} is injected when a human message is present and absent otherwise

331 tests passing, zero lint errors, zero new TypeScript errors.

Note: lib/export/latex-to-omml.ts has a pre-existing TS2307: Cannot find module 'mathml2omml' error present on main before this branch — not introduced here.

Scope

This fix is limited to the director summary and prompt layer. No changes to graph topology, LangGraph state, convertMessagesToOpenAI(), or any agent-facing logic.

Fixes THU-MAIC#511 — three symptoms (off-topic replies, role confusion, premature END)
traced to a single root cause: summarizeConversation() collapsed all messages
to generic [User]/[Assistant] labels, stripping the [AgentName]: prefix that
convertMessagesToOpenAI() adds when re-encoding peer agent turns as role:'user'.
The director could not distinguish a substantive human challenge from a brief
agent acknowledgment, causing it to emit END on unresolved questions.

Changes:
- conversation-summary.ts: detect [AgentName]: prefix on role:'user' messages
  and label them [Agent: Name] instead of [User]; label genuine human messages
  [Student (Human)]; add extractLastHumanMessage() to surface the last
  unaddressed human turn
- director-prompt.ts: inject an Open Student Question section when a human
  message is present, blocking premature END while the question is unresolved
- director-graph.ts: pass openaiMessages to buildDirectorPrompt (reuses the
  already-computed variable, no extra cost)
- director/system.md: wire {{openStudentQuestionSection}} placeholder so the
  section is actually injected into the director prompt
- tests: 38 new tests for conversation-summary, 3 regression tests for template
  injection; 331 total passing
@cosarah
Copy link
Copy Markdown
Collaborator

cosarah commented May 11, 2026

Thanks for working on this fix and for adding tests around the attribution issue.

I do see a problem with the current approach, though: it relies on regex-matching the rendered message text ([Name]: ...) to infer whether a message came from a human or from another agent. In the real runtime path, that is not a reliable source of truth.
For example, actual user messages can already be prefixed during conversion, so this logic can misclassify a real human message as an agent message, which means Open Student Question may never be populated when it actually should be.

I think the fix should use the real message role/source from structured metadata (originalRole, agentId, etc.) instead of parsing display text. In other words, the director should make this decision based on the actual speaker identity, not on a
regex over the formatted content.

@ashutoshrana
Copy link
Copy Markdown
Author

Hi @cosarah — you're right, and the issue is more fundamental than just occasional misclassification.

After tracing the full data flow: use-chat-sessions.ts sets senderName: t('common.you') on every human message. message-converter.ts then applies a [senderName]: prefix to all role:'user' content unconditionally (line 92–93). So in the director path, all role:'user' messages carry a [Name]: prefix — AGENT_PREFIX_RE matched them all as agent turns, meaning extractLastHumanMessage() always returned null and the Open Student Question section never fired.

The fix uses metadata as you suggested:

extractLastHumanMessage() now takes StatelessChatRequest['messages'] (pre-conversion) and uses msg.metadata?.originalRole === 'user' as the discriminator — set by use-chat-sessions.ts on every genuine human message. Text is extracted from msg.parts directly, no content parsing.

summarizeConversation() uses msg.role as the discriminator. In the director path (convertMessagesToOpenAI called without currentAgentId), peer agent messages stay as role:'assistant' — they are never re-encoded as role:'user'. So role is reliable here. The [senderName]: prefix is stripped from user content cosmetically for readability, not used for logic.

buildDirectorPrompt() 10th param changed from OpenAIMessage[] to StatelessChatRequest['messages']; director-graph.ts passes state.messages (already in scope at line 164) instead of the converted openaiMessages.

Happy to push an updated commit if this direction looks right — or adjust the approach if you'd prefer a different pattern.

@cosarah
Copy link
Copy Markdown
Collaborator

cosarah commented May 11, 2026

This direction looks right to me. Keeping sender prefixes in the conversation context is still useful for attribution, but the human vs. agent distinction should come from the real structured role/source metadata rather than regex-matching formatted text. Using the pre-conversion message metadata for Open Student Question and limiting prefix handling to presentation/readability feels like the correct fix.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Multi-agent dialogue: off-topic replies, role confusion, premature discussion end

2 participants