assign_to_non_existent_node_social_net-analysis-1: the agent can analyze the problem correctly, but the expected fault type "dependency" is unclear

[KaminariOS]

It was able to gather enough info to tell that it is a node issue.
The agent response "misconfig" actually makes sense.

The eval failure is mostly caused by unclear definition of "denpency" and "misconfig", not the agent itself.

[gaganso]

Hi @KaminariOS, This is a good point. What do you think would be a good general solution for this?

The fault type taxonomy could be subjective. For an infrastructure engineer, this is a misconfiguation. For an application engineer, this is an issue in one of the dependencies of the application.

[KaminariOS]

The general problem is the lack of a partial scoring system.

The current eval metric is binary based on the final submission only, so it is very sensitive to how we define the correct answer.

Ideally, the scoring should be continuous and based on the trajectory, like you mentioned in #108.

I think other LLM agent benchmarks also faced a similar problem and I suggest tp have a look at their approaches.