Refactor model field parsing in AnthropicChatCompletionStreamingProcessor #137926

Jan-Kazlouski-elastic · 2025-11-11T18:15:43Z

This PR is needed to fix Google Model Garden Anthropic message_start event parsing.
During demo call, it was observed that PERFORM CHAT COMPLETION INFERENCE call returns parsing error when parsing provider's message_start event.

{
    "error": {
        "message": "Received an error response for request from inference entity id [google-model-garden-anthropic-chat-completion]. Error message: [{\"type\":\"message_start\",\"message\":{\"model\":\"claude-3-5-haiku-20241022\",\"id\":\"msg_vrtx_01CBLbBxDTGGQ2768LtpdXnH\",\"type\":\"message\",\"role\":\"assistant\",\"content\":[],\"stop_reason\":null,\"stop_sequence\":null,\"usage\":{\"input_tokens\":12,\"cache_creation_input_tokens\":0,\"cache_read_input_tokens\":0,\"output_tokens\":5}}  }]",
        "type": "anthropic_error"
    }
}

Investigation has shown that during initial testing of Google Model Garden support message_start event was being returned like this:

{
    "type": "message_start",
    "message": {
        "id": "msg_vrtx_01FYVdAsnENM78FNkDy6vQoW",
        "type": "message",
        "role": "assistant",
        "model": "claude-3-5-haiku-20241022",
        "content": [],
        "stop_reason": null,
        "stop_sequence": null,
        "usage": {
            "input_tokens": 445,
            "cache_creation_input_tokens": 0,
            "cache_read_input_tokens": 0,
            "output_tokens": 10
        }
    }
}

Now message_start event is being returned like this:

{
    "type": "message_start",
    "message": {
        "model": "claude-3-5-haiku-20241022",
        "id": "msg_vrtx_01XTaeM2111A1r9tCnM3PCh3",
        "type": "message",
        "role": "assistant",
        "content": [],
        "stop_reason": null,
        "stop_sequence": null,
        "usage": {
            "input_tokens": 13,
            "cache_creation_input_tokens": 0,
            "cache_read_input_tokens": 0,
            "output_tokens": 1
        }
    }
}

Anthropic changed the order of the fields movig model field to the top of message dictionary. That is causing current implementation to fail during events parsing. In order to provide the fix for this behavior - order of field parsing is changed so model field is parsed first.

For future proof, fix can be achieved by complex refactoring of whole AnthropicChatCompletionStreamingProcessor class, but that is likely to be a complex task that may take a lot of resources and time. Because during GMG support implementation usage of ConstructingObjectParser was not proven to be optimal for this task. For now - I'm proposing quick fixing this by changing the parsing order and discuss the possibility of further refactoring later.

Testing result:

event: message
data: {"id":"msg_vrtx_01VSpa1Mwq3GYeh2rpqq5VZC","choices":[{"delta":{"role":"assistant"},"index":0}],"model":"claude-3-5-haiku-20241022","object":null,"usage":{"completion_tokens":1,"prompt_tokens":330,"total_tokens":331}}

event: message
data: {"id":null,"choices":[{"delta":{"content":""},"index":0}],"model":null,"object":null}

event: message
data: {"id":null,"choices":[{"delta":{"content":"I"},"index":0}],"model":null,"object":null}

event: message
data: {"id":null,"choices":[{"delta":{"content":"'ll"},"index":0}],"model":null,"object":null}

event: message
data: {"id":null,"choices":[{"delta":{"content":" help"},"index":0}],"model":null,"object":null}

event: message
data: {"id":null,"choices":[{"delta":{"content":" you find the current price"},"index":0}],"model":null,"object":null}

event: message
data: {"id":null,"choices":[{"delta":{"content":" of a scarf."},"index":0}],"model":null,"object":null}

event: message
data: {"id":null,"choices":[{"delta":{"content":" I"},"index":0}],"model":null,"object":null}

event: message
data: {"id":null,"choices":[{"delta":{"content":"'ll"},"index":0}],"model":null,"object":null}

event: message
data: {"id":null,"choices":[{"delta":{"content":" use"},"index":0}],"model":null,"object":null}

event: message
data: {"id":null,"choices":[{"delta":{"content":" the get_current_"},"index":0}],"model":null,"object":null}

event: message
data: {"id":null,"choices":[{"delta":{"content":"price function"},"index":0}],"model":null,"object":null}

event: message
data: {"id":null,"choices":[{"delta":{"content":" to"},"index":0}],"model":null,"object":null}

event: message
data: {"id":null,"choices":[{"delta":{"content":" retrieve"},"index":0}],"model":null,"object":null}

event: message
data: {"id":null,"choices":[{"delta":{"content":" this"},"index":0}],"model":null,"object":null}

event: message
data: {"id":null,"choices":[{"delta":{"content":" information."},"index":0}],"model":null,"object":null}

event: message
data: {"id":null,"choices":[{"delta":{"tool_calls":[{"index":0,"id":"toolu_vrtx_017pa6BRMXSw61bPKWjfWmaw","function":{"arguments":"{}","name":"get_current_price"},"type":null}]},"index":1}],"model":null,"object":null}

event: message
data: {"id":null,"choices":[{"delta":{"tool_calls":[{"index":0,"function":{"arguments":""},"type":null}]},"index":1}],"model":null,"object":null}

event: message
data: {"id":null,"choices":[{"delta":{"tool_calls":[{"index":0,"function":{"arguments":"{\"item"},"type":null}]},"index":1}],"model":null,"object":null}

event: message
data: {"id":null,"choices":[{"delta":{"tool_calls":[{"index":0,"function":{"arguments":"\": \"scarf"},"type":null}]},"index":1}],"model":null,"object":null}

event: message
data: {"id":null,"choices":[{"delta":{"tool_calls":[{"index":0,"function":{"arguments":"\"}"},"type":null}]},"index":1}],"model":null,"object":null}

event: message
data: {"id":null,"choices":[{"delta":{},"finish_reason":"tool_use","index":0}],"model":null,"object":null,"usage":{"completion_tokens":85,"prompt_tokens":0,"total_tokens":85}}

event: message
data: [DONE]

- Have you signed the contributor license agreement?
- Have you followed the contributor guidelines?
- If submitting code, have you built your formula locally prior to submission with gradle check?
- If submitting code, is your pull request against main? Unless there is a good reason otherwise, we prefer pull requests against main and will backport as needed.
- If submitting code, have you checked that your submission is for an OS and architecture that we support?
- If you are submitting this code for a class then read our policy for that.

…ssor

…gProcessor

Jan-Kazlouski-elastic · 2025-11-12T00:06:09Z

After discussion in slack I updated the solution. Improved parsing logic, type checks, documentation and test coverage.

Testing results:

Completion Non-Streaming (regression)

RQ

{
    "input": "The sky above the port was the color of television tuned to a dead channel.",
    "task_settings": {
        "max_tokens": 20
    }
}

RS

{
    "completion": [
        {
            "result": "This is the famous opening line from William Gibson's seminal cyberpunk novel \"Neurom"
        }
    ]
}

Completion Streaming (regression)

RQ

{
    "input": "The sky above the port was the color of television tuned to a dead channel.",
    "task_settings": {
        "max_tokens": 20
    }
}

RS

event: message
data: {"completion":[{"delta":"This"}]}

event: message
data: {"completion":[{"delta":" is the"},{"delta":" famous"}]}

event: message
data: {"completion":[{"delta":" opening"}]}

event: message
data: {"completion":[{"delta":" line from William"},{"delta":" Gibson"}]}

event: message
data: {"completion":[{"delta":"'s groun"}]}

event: message
data: {"completion":[{"delta":"dbreaking cyb"}]}

event: message
data: {"completion":[{"delta":"erpunk novel \""},{"delta":"Neu"}]}

event: message
data: [DONE]

Basic Chat Completion (fixed)

RQ

{
    "messages": [
        {
            "role": "user",
            "content": "What is deep learning?"
        }
    ],
    "max_completion_tokens": 100
}

RS

event: message
data: {"id":"msg_vrtx_0123G7M1VN7R63EQQsaRpKA5","choices":[{"delta":{"role":"assistant"},"index":0}],"model":"claude-3-5-haiku-20241022","object":null,"usage":{"completion_tokens":5,"prompt_tokens":12,"total_tokens":17}}

event: message
data: {"id":null,"choices":[{"delta":{"content":""},"index":0}],"model":null,"object":null}

event: message
data: {"id":null,"choices":[{"delta":{"content":"Deep learning is a subset"},"index":0}],"model":null,"object":null}

event: message
data: {"id":null,"choices":[{"delta":{"content":" of machine learning that uses"},"index":0}],"model":null,"object":null}

event: message
data: {"id":null,"choices":[{"delta":{"content":" artificial"},"index":0}],"model":null,"object":null}

event: message
data: {"id":null,"choices":[{"delta":{"content":" neural networks with multiple layers"},"index":0}],"model":null,"object":null}

event: message
data: {"id":null,"choices":[{"delta":{"content":" ("},"index":0}],"model":null,"object":null}

event: message
data: {"id":null,"choices":[{"delta":{"content":"hence"},"index":0}],"model":null,"object":null}

event: message
data: {"id":null,"choices":[{"delta":{"content":" \""},"index":0}],"model":null,"object":null}

event: message
data: {"id":null,"choices":[{"delta":{"content":"deep\") to progress"},"index":0}],"model":null,"object":null}

event: message
data: {"id":null,"choices":[{"delta":{"content":"ively extract"},"index":0}],"model":null,"object":null}

event: message
data: {"id":null,"choices":[{"delta":{"content":" higher-level features from"},"index":0}],"model":null,"object":null}

event: message
data: {"id":null,"choices":[{"delta":{"content":" raw"},"index":0}],"model":null,"object":null}

event: message
data: {"id":null,"choices":[{"delta":{"content":" input"},"index":0}],"model":null,"object":null}

event: message
data: {"id":null,"choices":[{"delta":{"content":"."},"index":0}],"model":null,"object":null}

event: message
data: {"id":null,"choices":[{"delta":{"content":" Key"},"index":0}],"model":null,"object":null}

event: message
data: {"id":null,"choices":[{"delta":{"content":" characteristics"},"index":0}],"model":null,"object":null}

event: message
data: {"id":null,"choices":[{"delta":{"content":" include"},"index":0}],"model":null,"object":null}

event: message
data: {"id":null,"choices":[{"delta":{"content":":\n\n1"},"index":0}],"model":null,"object":null}

event: message
data: {"id":null,"choices":[{"delta":{"content":". Architecture"},"index":0}],"model":null,"object":null}

event: message
data: {"id":null,"choices":[{"delta":{"content":"\n-"},"index":0}],"model":null,"object":null}

event: message
data: {"id":null,"choices":[{"delta":{"content":" Uses"},"index":0}],"model":null,"object":null}

event: message
data: {"id":null,"choices":[{"delta":{"content":" neural"},"index":0}],"model":null,"object":null}

event: message
data: {"id":null,"choices":[{"delta":{"content":" networks with many"},"index":0}],"model":null,"object":null}

event: message
data: {"id":null,"choices":[{"delta":{"content":" hidden"},"index":0}],"model":null,"object":null}

event: message
data: {"id":null,"choices":[{"delta":{"content":" layers"},"index":0}],"model":null,"object":null}

event: message
data: {"id":null,"choices":[{"delta":{"content":"\n-"},"index":0}],"model":null,"object":null}

event: message
data: {"id":null,"choices":[{"delta":{"content":" Inspire"},"index":0}],"model":null,"object":null}

event: message
data: {"id":null,"choices":[{"delta":{"content":"d by the structure"},"index":0}],"model":null,"object":null}

event: message
data: {"id":null,"choices":[{"delta":{"content":" of the human brain"},"index":0}],"model":null,"object":null}

event: message
data: {"id":null,"choices":[{"delta":{"content":"\n- Allows systems"},"index":0}],"model":null,"object":null}

event: message
data: {"id":null,"choices":[{"delta":{"content":" to learn complex patterns"},"index":0}],"model":null,"object":null}

event: message
data: {"id":null,"choices":[{"delta":{"content":" and representations"},"index":0}],"model":null,"object":null}

event: message
data: {"id":null,"choices":[{"delta":{"content":"\n\n2. Key"},"index":0}],"model":null,"object":null}

event: message
data: {"id":null,"choices":[{"delta":{"content":" Features"},"index":0}],"model":null,"object":null}

event: message
data: {"id":null,"choices":[{"delta":{"content":"\n- Automatic"},"index":0}],"model":null,"object":null}

event: message
data: {"id":null,"choices":[{"delta":{"content":" feature extraction"},"index":0}],"model":null,"object":null}

event: message
data: {"id":null,"choices":[{"delta":{"content":"\n- Can"},"index":0}],"model":null,"object":null}

event: message
data: {"id":null,"choices":[{"delta":{"content":" handle"},"index":0}],"model":null,"object":null}

event: message
data: {"id":null,"choices":[{"delta":{"content":" un"},"index":0}],"model":null,"object":null}

event: message
data: {"id":null,"choices":[{"delta":{"content":"structured data (images"},"index":0}],"model":null,"object":null}

event: message
data: {"id":null,"choices":[{"delta":{"content":", text, audio"},"index":0}],"model":null,"object":null}

event: message
data: {"id":null,"choices":[{"delta":{"content":")"},"index":0}],"model":null,"object":null}

event: message
data: {"id":null,"choices":[{"delta":{"content":"\n-"},"index":0}],"model":null,"object":null}

event: message
data: {"id":null,"choices":[{"delta":{},"finish_reason":"max_tokens","index":0}],"model":null,"object":null,"usage":{"completion_tokens":100,"prompt_tokens":0,"total_tokens":100}}

event: message
data: [DONE]

Complex Chat Completion (fixed)

RQ

{
    "messages": [
        {
            "role": "user",
            "content": [
                {
                    "type": "text",
                    "text": "What is the price of scarf?"
                }
            ]
        }
    ],
    "max_completion_tokens": 100,
    "temperature": 0.2,
    "top_p": 0.2,
    "tools": [
        {
            "type": "auto",
            "function": {
                "name": "get_current_price",
                "description": "Get the current price of a item",
                "parameters": {
                    "type": "object",
                    "properties": {
                        "item": {
                            "id": "123"
                        }
                    }
                }
            }
        }
    ],
    "tool_choice": {
        "type": "auto",
        "function": {
            "name": "get_current_price"
        }
    }
}

RS

event: message
data: {"id":"msg_vrtx_0151t4ozgYmPL5BNfjk4KkRy","choices":[{"delta":{"role":"assistant"},"index":0}],"model":"claude-3-5-haiku-20241022","object":null,"usage":{"completion_tokens":1,"prompt_tokens":330,"total_tokens":331}}

event: message
data: {"id":null,"choices":[{"delta":{"content":""},"index":0}],"model":null,"object":null}

event: message
data: {"id":null,"choices":[{"delta":{"content":"I"},"index":0}],"model":null,"object":null}

event: message
data: {"id":null,"choices":[{"delta":{"content":"'ll"},"index":0}],"model":null,"object":null}

event: message
data: {"id":null,"choices":[{"delta":{"content":" help"},"index":0}],"model":null,"object":null}

event: message
data: {"id":null,"choices":[{"delta":{"content":" you find the current price"},"index":0}],"model":null,"object":null}

event: message
data: {"id":null,"choices":[{"delta":{"content":" of a scarf."},"index":0}],"model":null,"object":null}

event: message
data: {"id":null,"choices":[{"delta":{"content":" I"},"index":0}],"model":null,"object":null}

event: message
data: {"id":null,"choices":[{"delta":{"content":"'ll"},"index":0}],"model":null,"object":null}

event: message
data: {"id":null,"choices":[{"delta":{"content":" use"},"index":0}],"model":null,"object":null}

event: message
data: {"id":null,"choices":[{"delta":{"content":" the get_current_"},"index":0}],"model":null,"object":null}

event: message
data: {"id":null,"choices":[{"delta":{"content":"price function"},"index":0}],"model":null,"object":null}

event: message
data: {"id":null,"choices":[{"delta":{"content":" to retrieve"},"index":0}],"model":null,"object":null}

event: message
data: {"id":null,"choices":[{"delta":{"content":" this"},"index":0}],"model":null,"object":null}

event: message
data: {"id":null,"choices":[{"delta":{"content":" information."},"index":0}],"model":null,"object":null}

event: message
data: {"id":null,"choices":[{"delta":{"tool_calls":[{"index":0,"id":"toolu_vrtx_01YPFy2HtYaWAYES6tK4pHk8","function":{"arguments":"{}","name":"get_current_price"},"type":null}]},"index":1}],"model":null,"object":null}

event: message
data: {"id":null,"choices":[{"delta":{"tool_calls":[{"index":0,"function":{"arguments":""},"type":null}]},"index":1}],"model":null,"object":null}

event: message
data: {"id":null,"choices":[{"delta":{"tool_calls":[{"index":0,"function":{"arguments":"{\"item\": "},"type":null}]},"index":1}],"model":null,"object":null}

event: message
data: {"id":null,"choices":[{"delta":{"tool_calls":[{"index":0,"function":{"arguments":"\"scar"},"type":null}]},"index":1}],"model":null,"object":null}

event: message
data: {"id":null,"choices":[{"delta":{"tool_calls":[{"index":0,"function":{"arguments":"f\"}"},"type":null}]},"index":1}],"model":null,"object":null}

event: message
data: {"id":null,"choices":[{"delta":{},"finish_reason":"tool_use","index":0}],"model":null,"object":null,"usage":{"completion_tokens":85,"prompt_tokens":0,"total_tokens":85}}

event: message
data: [DONE]

…eptions

…nthropic-streaming-processor-fix

elasticsearchmachine · 2025-11-12T16:42:52Z

Pinging @elastic/ml-core (Team:ML)

elasticsearchmachine · 2025-11-12T22:22:40Z

💚 Backport successful

Status	Branch	Result
✅	9.2

…Processor (elastic#137926)

…Processor (#137926) (#137991) Co-authored-by: Jan-Kazlouski-elastic <[email protected]>

Refactor model field parsing in AnthropicChatCompletionStreamingProce…

e5ba99a

…ssor

elasticsearchmachine added needs:triage Requires assignment of a team area label v9.3.0 external-contributor Pull request authored by a developer outside the Elasticsearch team labels Nov 11, 2025

Jan-Kazlouski-elastic added 2 commits November 11, 2025 20:25

Add changelog

d48dd39

Enhance error handling and parsing in AnthropicChatCompletionStreamin…

33b06cf

…gProcessor

Add test for parsing alternate field order in the event json

daa8f8d

DonalEvans approved these changes Nov 12, 2025

View reviewed changes

Jan-Kazlouski-elastic added 2 commits November 12, 2025 02:45

Refactor error handling to use String.format for field extraction exc…

4b1a6a2

…eptions

Merge remote-tracking branch 'origin/main' into google-model-garden-a…

4d164ed

…nthropic-streaming-processor-fix

DonalEvans self-assigned this Nov 12, 2025

DonalEvans added >bug auto-backport Automatically create backport pull requests when merged branch:9.2 labels Nov 12, 2025

elasticsearchmachine added v9.2.2 and removed branch:9.2 labels Nov 12, 2025

DonalEvans added :ml Machine learning Team:ML Meta label for the ML team branch:9.2 and removed v9.2.2 labels Nov 12, 2025

elasticsearchmachine added v9.2.2 and removed needs:triage Requires assignment of a team area label branch:9.2 labels Nov 12, 2025

DonalEvans merged commit 366d680 into elastic:main Nov 12, 2025
36 checks passed

DonalEvans mentioned this pull request Nov 12, 2025

[9.2] [ML] Refactor model field parsing in AnthropicChatCompletionStreamingProcessor (#137926) #137991

Merged

DonalEvans pushed a commit to DonalEvans/elasticsearch that referenced this pull request Nov 12, 2025

[ML] Refactor model field parsing in AnthropicChatCompletionStreaming…

cb869eb

…Processor (elastic#137926)

elasticsearchmachine pushed a commit that referenced this pull request Nov 12, 2025

[ML] Refactor model field parsing in AnthropicChatCompletionStreaming…

f426add

…Processor (#137926) (#137991) Co-authored-by: Jan-Kazlouski-elastic <[email protected]>

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Refactor model field parsing in AnthropicChatCompletionStreamingProcessor #137926

Refactor model field parsing in AnthropicChatCompletionStreamingProcessor #137926

Jan-Kazlouski-elastic commented Nov 11, 2025 •

edited

Loading

Uh oh!

Jan-Kazlouski-elastic commented Nov 12, 2025

Uh oh!

elasticsearchmachine commented Nov 12, 2025

Uh oh!

Uh oh!

elasticsearchmachine commented Nov 12, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Refactor model field parsing in AnthropicChatCompletionStreamingProcessor #137926

Refactor model field parsing in AnthropicChatCompletionStreamingProcessor #137926

Conversation

Jan-Kazlouski-elastic commented Nov 11, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Jan-Kazlouski-elastic commented Nov 12, 2025

Uh oh!

elasticsearchmachine commented Nov 12, 2025

Uh oh!

Uh oh!

elasticsearchmachine commented Nov 12, 2025

💚 Backport successful

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Jan-Kazlouski-elastic commented Nov 11, 2025 •

edited

Loading