Tool execution infinite loop on the same action

[Lovenoreus]

I places two calls to the container app which were not able to be processed, having these error messages and looping

At 01:15 it starts saying the same sentence in different ways

container_log.txt

container_error_intelligence.mp4

[clemlesne]

I see that the LLM repeated itself with the same action over and over:

2025-01-21T14:48:25.499591Z [info     ] Playing TTS: I have updated the location to Stockholm Central. call.channel=voice call.id=5baf74ca-db60-4795-a83e-99c9ba65c03c call.message=yes this happened in stockholm central call.phone_number=+4570726734 tool.args={"updates": [{"field": "incident_location", "value": "Stockholm Central"}], "customer_response": "I have updated the location to Stockholm Central."} tool.name=updated_claim
2025-01-21T14:48:25.603500Z [info     ] Executed function updated_claim ({'updates': [{'field': 'incident_location', 'value': 'Stockholm Central'}], 'customer_response': 'I have updated the location to Stockholm Central.'}): # Updated fields
- U..."Stockholm Central". call.channel=voice call.id=5baf74ca-db60-4795-a83e-99c9ba65c03c call.message=yes this happened in stockholm central call.phone_number=+4570726734 tool.args={"updates": [{"field": "incident_location", "value": "Stockholm Central"}], "customer_response": "I have updated the location to Stockholm Central."} tool.name=updated_claim
2025-01-21T14:48:26.180376Z [info     ] Continuing chat, 2 remaining   call.channel=voice call.id=5baf74ca-db60-4795-a83e-99c9ba65c03c call.message=yes this happened in stockholm central call.phone_number=+4570726734

[...]

2025-01-21T14:48:27.976715Z [info     ] Playing TTS: I have updated the location to Stockholm Central. call.channel=voice call.id=5baf74ca-db60-4795-a83e-99c9ba65c03c call.message=Thank you for the information. I have updated the location to Stockholm Central. call.phone_number=+4570726734 tool.args={"updates": [{"field": "incident_location", "value": "Stockholm Central"}], "customer_response": "I have updated the location to Stockholm Central."} tool.name=updated_claim
2025-01-21T14:48:28.086788Z [info     ] Executed function updated_claim ({'updates': [{'field': 'incident_location', 'value': 'Stockholm Central'}], 'customer_response': 'I have updated the location to Stockholm Central.'}): # Updated fields
- U..."Stockholm Central". call.channel=voice call.id=5baf74ca-db60-4795-a83e-99c9ba65c03c call.message=Thank you for the information. I have updated the location to Stockholm Central. call.phone_number=+4570726734 tool.args={"updates": [{"field": "incident_location", "value": "Stockholm Central"}], "customer_response": "I have updated the location to Stockholm Central."} tool.name=updated_claim
2025-01-21T14:48:28.492717Z [info     ] Continuing chat, 1 remaining   call.channel=voice call.id=5baf74ca-db60-4795-a83e-99c9ba65c03c call.message=Thank you for the information. I have updated the location to Stockholm Central. call.phone_number=+4570726734

[...]

2025-01-21T14:48:30.171047Z [info     ] Playing TTS: I have updated the location to Stockholm Central. call.channel=voice call.id=5baf74ca-db60-4795-a83e-99c9ba65c03c call.message=Thank you for the information. I have updated the location to Stockholm Central. I have updated the location to Stockholm Central. call.phone_number=+4570726734 tool.args={"updates": [{"field": "incident_location", "value": "Stockholm Central"}], "customer_response": "I have updated the location to Stockholm Central."} tool.name=updated_claim
2025-01-21T14:48:30.276048Z [info     ] Executed function updated_claim ({'updates': [{'field': 'incident_location', 'value': 'Stockholm Central'}], 'customer_response': 'I have updated the location to Stockholm Central.'}): # Updated fields
- U..."Stockholm Central". call.channel=voice call.id=5baf74ca-db60-4795-a83e-99c9ba65c03c call.message=Thank you for the information. I have updated the location to Stockholm Central. I have updated the location to Stockholm Central. call.phone_number=+4570726734 tool.args={"updates": [{"field": "incident_location", "value": "Stockholm Central"}], "customer_response": "I have updated the location to Stockholm Central."} tool.name=updated_claim
2025-01-21T14:48:30.816240Z [info     ] Continuing chat, 0 remaining   call.channel=voice call.id=5baf74ca-db60-4795-a83e-99c9ba65c03c call.message=Thank you for the information. I have updated the location to Stockholm Central. I have updated the location to Stockholm Central. call.phone_number=+4570726734

I see two possibilities:

• Call context is not updated across LLM recursion calls
• LLM didn't understand the situation and he's simply stupid

Can you reproduce this issue?

Also, found a bug in your logs: #399, but not related to our main conversation.

[clemlesne]

As a context, LLM traces are stored in App Insights. Can you search for this particular call and send the Azure OpenAI API call trace? It will include tags with details on what tools and message history that have been sent. With that, we'll be able to see if this is an application bug or a LLM bug.

[Lovenoreus]

As a context, LLM traces are stored in App Insights. Can you search for this particular call and send the Azure OpenAI API call trace? It will include tags with details on what tools and message history that have been sent. With that, we'll be able to see if this is an application bug or a LLM bug.

I never know which services I can leave up and which to delete between each usage of the app to not cause any problems, so I just delete everything to be safe. Earlier I had problems leaving some services up before running the "make deploy" command again, meaning that the LLM traces are gone.

[Lovenoreus]

As a context, LLM traces are stored in App Insights. Can you search for this particular call and send the Azure OpenAI API call trace? It will include tags with details on what tools and message history that have been sent. With that, we'll be able to see if this is an application bug or a LLM bug.

I was able to replicate a sentence looping 3 times, I clicked on "search" in app.insights but I'm not able to navigate this window and select the right thing, you can call me on Teams if you want to show me at any time