Implement trace validation to distinguish environmental artifacts from induced faults

[n1k0ver3E]

Description

In the test environment, running kubectl get pods -n docker or kubectl get all -n docker sometimes returns:

No resources found in docker namespace.

The detection agent interprets this as a K8s issue and marks it as such.
However, in the production env this namespace always contains resources, so this case should not be treated as an issue. This leads to false positives and inflates detection accuracy metrics.

Trace

    "agent": "openrouter",
    "session_id": "292a425f-0b28-45f5-96bc-e4f02fa42623",
    "problem_id": "flower_model_misconfig-detection",
    "start_time": 1756135764.7457962,
    "end_time": 1756135765.7294734,
    "trace": [
        {
            "role": "assistant",
            "content": "```\nexec_shell(\"kubectl get pods -n docker\")\n```"
        },
        {
            "role": "env",
            "content": "No resources found in docker namespace.\n"
        },
        {
            "role": "assistant",
            "content": "```\nexec_shell(\"kubectl get all -n docker\")\n```"
        },
        {
            "role": "env",
            "content": "No resources found in docker namespace.\n"
        },
        {
            "role": "assistant",
            "content": "```\nsubmit(\"Yes\")\n```"
        },
        {
            "role": "env",
            "content": "1"
        }
    ],
    "results": {
        "Detection Accuracy": "Correct",
        "TTD": 0.9836771488189697,
        "steps": 3,
        "in_tokens": 15,
        "out_tokens": 33
    }

Open Question
Would it make sense to introduce a supervisor agent to validate conversations before marking them as issues?
The idea is that if no actual problem is evidenced, the supervisor would return NoIssue or Inconclusive instead of allowing a false positive.

One option could be adding a lightweight GPT-5–based supervisor to evaluate detection tasks.
Any suggestions or alternative approaches are welcome. 🤔

[gaganso]

This is a great find, thank you! I think this could apply to other problems too and highlights the need for a better evaluation method that considers the full agent trace.

Building on your idea, this new evaluation agent could have access to the problem description and analyze the trajectory. Based on its feedback, the primary AIOps agent could then continue its search for the problem within its budget. This creates a dynamic that mimics a user interacting with and guiding an AIOps agent.

Please let me know what you think.

"The idea is that if no actual problem is evidenced"

Small question/clarification: I think the problem exists, but the agent doesn't detect the actual problem and reports success when it sees a different issue?

[HacksonClark]

@n1k0ver3E This is just a misunderstanding of what the detection task is. In anomaly detection, the agent is trying to figure out if there is something wrong in the cluster (a fault). It can look at any resource in the cluster and submit an answer (Yes, or no). If you're running any problem (besides no op) the answer will be yes as there is a fault injected into the system (that's what a problem is in AIOpsLab).

So, in your example, this is not a false positive as you are currently running a problem (a fault is injected). That's why the answer is "Yes". AIOpsLab knows the ground-truth related to the fault, how the agent arrives at the answer does not matter. Though, adding some sort of LLM as a judge to look at the trajectory would be interesting.

[gaganso]

Hi @HacksonClark, I agree with you that there was a misunderstanding on it being a false positive. But I also think the agent got "lucky" in a way as it didn't detect the fault induced but probably reported "Yes" due to a setup or environment issue. So the comment on such scenarios inflating an agent's accuracy is true. Do you think we should have a work item on this?

[HacksonClark]

@gaganso we could, but we would essentially have to replace all of detection and localization with LLM as a judge. Which is honestly a good idea.