BUG TAPAttack attack execution error

[3leg0ocat]

from pyrit.executor.attack import AttackExecutor

AttackExecutor().execute_multi_turn_attacks_async(TAPAttack())

The code above will throw an error when run:
Attack strategy TreeOfAttacksWithPruningAttack must use MultiTurnAttackContext or a subclass of it.

# Validate that the attack uses MultiTurnAttackContext
if hasattr(attack, "_context_type") and not issubclass(attack._context_type, MultiTurnAttackContext):
    raise TypeError(
        f"Attack strategy {attack.__class__.__name__} must use MultiTurnAttackContext or a subclass of it."
    )

I'd like to know if this design intentionally excludes TAP? Could the type checking be relaxed to allow TAPAttackContext as well?

[romanlutz]

@3leg0ocat did you figure it out?

[3leg0ocat]

@romanlutz Thanks, that part is clear now. However, there is still a related issue that consistently reproduces when using TAPAttack together with scorers other than the simple example ones (e.g. SelfAskCategoryScorer).

The error occurs when printing the attack result stage:
ConsoleAttackResultPrinter().print_result_async()
It fails inside group_conversation_message_pieces_by_sequence with:

ValueError: All message pieces must be from the same conversation.
Expected conversation_id='...', but found '...'.

This indicates that message_pieces contains entries from multiple conversation_ids.

The failure happens when querying memory entries by conversation_id with join_scores=True:

memory_entries: Sequence[PromptMemoryEntry] = self._query_entries(
    PromptMemoryEntry,
    conditions=and_(*conditions) if conditions else None,
    join_scores=True,
)
message_pieces = [memory_entry.get_message_piece() for memory_entry in memory_entries]
return sort_message_pieces(message_pieces=message_pieces)

Although the query is filtered by conversation_id, additional entries seem to be included.
Looking at the query implementation, this seems to be the relevant logic:

query = session.query(Model)
if join_scores and Model == PromptMemoryEntry:
    query = query.options(joinedload(PromptMemoryEntry.scores))
if conditions is not None:
    query = query.filter(conditions)
return query.all()

Based on this, it seems that enabling join_scores=True may change what gets loaded alongside the filtered entries. Could this also be including message_pieces from the scoring stages?

I also compared SelfAskCategoryScorer with FloatScaleThresholdScorer, and one
notable difference is that SelfAskCategoryScorer passes attack_identifier into _score_value_with_llm, for example:

unvalidated_score = await self._score_value_with_llm(
    ...
    scored_prompt_id=message_piece.id,
    objective=objective,
    attack_identifier=message_piece.attack_identifier,
)

Initially I suspected this might affect how scoring records are associated or later retrieved. However, even after explicitly setting
attack_identifier=None, the same conversation mismatch error still occurs.

Given this, could you advise on the recommended way to handle this?

Is SelfAskCategoryScorer expected to be incompatible with TAPAttack, or is
there a supported workaround or configuration to use it without hitting the
conversation mismatch error during result printing?

[romanlutz]

Reopening this because it requires some thought and perhaps an investigation.

[akkupratap323]

@romanlutz Thanks for reopening this. I dug deep into the issue and finally tracked down the root cause.

The problem comes from how SQLAlchemy's joinedload handles the PromptMemoryEntry.scores relationship during TAPAttack's conversation duplication.

Here's the chain of events:

TAPAttack creates branching conversations. When the attack tree branches, duplicate_conversation() (memory_interface.py:574-595) makes new message pieces with fresh conversation IDs but keeps the same original_prompt_id for tracking.

SelfAskCategoryScorer generates scores stored with prompt_request_response_id pointing to the PromptMemoryEntry's id.

The relationship definition doesn't match up properly. In memory_models.py:187-192, the scores relationship uses:

primaryjoin="ScoreEntry.prompt_request_response_id == PromptMemoryEntry.original_prompt_id" This joins on original_prompt_id instead of id.

When ConsoleAttackResultPrinter calls get_conversation(), it triggers get_message_pieces() (memory_interface.py:490-572) with join_scores=True. This calls _query_entries() with joinedload(PromptMemoryEntry.scores).

The LEFT OUTER JOIN pulls in scores from cross-conversation entries. Since duplicated conversations share the same original_prompt_id, it grabs ScoreEntries from other conversation branches.

group_conversation_message_pieces_by_sequence() (message.py:295-300) chokes when it gets MessagePieces with mixed conversation IDs, throwing a ValueError.

This only happens with certain scorers because SelfAskCategoryScorer and other LLM-based scorers create persistent ScoreEntry records in the database. Simpler scorers like basic threshold ones don't, so joinedload doesn't pull extra entries.

Fix options:

Option A (Recommended): Modify get_conversation() to pass join_scores=False, or filter the loaded scores by conversation_id afterward. The printer doesn't actually need scores from other conversations anyway.

Option B: Update the relationship primaryjoin in PromptMemoryEntry.scores to use id instead:

primaryjoin="ScoreEntry.prompt_request_response_id == PromptMemoryEntry.id" Option C: Add a conversation_id filter in the joinedload or use subqueryload with explicit filtering.
Workaround for users: Until it's fixed, avoid calling ConsoleAttackResultPrinter().print_result_async() with TAPAttack results, or stick to scorers that don't create persistent ScoreEntry records.