Skip to content

Remove unnecessary ACP cost estimation fallback#2330

Merged
simonrosenberg merged 1 commit intomainfrom
fix/remove-acp-cost-fallback
Mar 5, 2026
Merged

Remove unnecessary ACP cost estimation fallback#2330
simonrosenberg merged 1 commit intomainfrom
fix/remove-acp-cost-fallback

Conversation

@simonrosenberg
Copy link
Collaborator

@simonrosenberg simonrosenberg commented Mar 5, 2026

Summary

  • Removes _estimate_cost_from_tokens() fallback and _last_estimated_cost tracking from acp_agent.py
  • Removes 4 associated fallback tests from test_acp_agent.py

The fallback was added in #2326 to estimate cost via litellm.model_cost when the ACP server reported $0. Investigation revealed the $0 was caused by using claude-code-acp@0.16.2 which lacked usage tracking entirely. With claude-agent-acp@0.20.2 (already updated in #2326), the Claude Code SDK correctly computes cost via its internal pricing tables and forwards it through UsageUpdate — making the fallback unnecessary.

Test plan

  • All 61 remaining ACP agent tests pass
  • CI run-examples workflow confirms ACP example still reports non-zero cost without the fallback

Agent Server images for this PR

GHCR package: https://github.com/OpenHands/agent-sdk/pkgs/container/agent-server

Variants & Base Images

Variant Architectures Base Image Docs / Tags
java amd64, arm64 eclipse-temurin:17-jdk Link
python amd64, arm64 nikolaik/python-nodejs:python3.12-nodejs22 Link
golang amd64, arm64 golang:1.21-bookworm Link

Pull (multi-arch manifest)

# Each variant is a multi-arch manifest supporting both amd64 and arm64
docker pull ghcr.io/openhands/agent-server:8cb1c5d-python

Run

docker run -it --rm \
  -p 8000:8000 \
  --name agent-server-8cb1c5d-python \
  ghcr.io/openhands/agent-server:8cb1c5d-python

All tags pushed for this build

ghcr.io/openhands/agent-server:8cb1c5d-golang-amd64
ghcr.io/openhands/agent-server:8cb1c5d-golang_tag_1.21-bookworm-amd64
ghcr.io/openhands/agent-server:8cb1c5d-golang-arm64
ghcr.io/openhands/agent-server:8cb1c5d-golang_tag_1.21-bookworm-arm64
ghcr.io/openhands/agent-server:8cb1c5d-java-amd64
ghcr.io/openhands/agent-server:8cb1c5d-eclipse-temurin_tag_17-jdk-amd64
ghcr.io/openhands/agent-server:8cb1c5d-java-arm64
ghcr.io/openhands/agent-server:8cb1c5d-eclipse-temurin_tag_17-jdk-arm64
ghcr.io/openhands/agent-server:8cb1c5d-python-amd64
ghcr.io/openhands/agent-server:8cb1c5d-nikolaik_s_python-nodejs_tag_python3.12-nodejs22-amd64
ghcr.io/openhands/agent-server:8cb1c5d-python-arm64
ghcr.io/openhands/agent-server:8cb1c5d-nikolaik_s_python-nodejs_tag_python3.12-nodejs22-arm64
ghcr.io/openhands/agent-server:8cb1c5d-golang
ghcr.io/openhands/agent-server:8cb1c5d-java
ghcr.io/openhands/agent-server:8cb1c5d-python

About Multi-Architecture Support

  • Each variant tag (e.g., 8cb1c5d-python) is a multi-arch manifest supporting both amd64 and arm64
  • Docker automatically pulls the correct architecture for your platform
  • Individual architecture tags (e.g., 8cb1c5d-python-amd64) are also available if needed

The `_estimate_cost_from_tokens()` fallback and related tracking were
added when the ACP server appeared to report $0 cost. Investigation
revealed the $0 was caused by using an outdated npm package
(`claude-code-acp@0.16.2`) that lacked usage tracking entirely. With
`claude-agent-acp@0.20.2` (already updated in #2326), the Claude Code
SDK correctly computes cost via its internal pricing tables and forwards
it through UsageUpdate — making the litellm-based fallback unnecessary.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@github-actions
Copy link
Contributor

github-actions bot commented Mar 5, 2026

API breakage checks (Griffe)

Result: Failed

Log excerpt (first 1000 characters)

============================================================
Checking openhands-sdk (openhands.sdk)
============================================================
Comparing openhands-sdk 1.11.5 against 1.11.4
::notice title=openhands-sdk API::Ignoring Field metadata-only change (non-breaking): load_public_skills
::warning file=openhands-sdk/openhands/sdk/conversation/conversation.py,line=103,title=Conversation.__new__(delete_on_close)::Parameter default was changed: `False` -> `True`
::notice title=openhands-sdk API::Ignoring Field metadata-only change (non-breaking): temperature
::warning file=openhands-sdk/openhands/sdk/llm/llm.py,line=196,title=LLM.top_p::Attribute value was changed: `Field(default=1.0, ge=0, le=1)` -> `Field(default=None, ge=0, le=1, description='Nucleus sampling parameter. Defaults to None (uses provider default). Set to a value between 0 and 1 to control diversity of outputs.')`
::notice title=openhands-sdk API::Ignoring Field metadata-only change (non-breaking): 

Action log

@github-actions
Copy link
Contributor

github-actions bot commented Mar 5, 2026

Agent server REST API breakage checks (OpenAPI)

Result: Failed

Log excerpt (first 1000 characters)
{"asctime": "2026-03-05 19:04:22,177", "levelname": "WARNING", "name": "openhands.agent_server.config", "filename": "config.py", "lineno": 173, "message": "\u26a0\ufe0f OH_SECRET_KEY was not defined. Secrets will not be persisted between restarts."}
::error title=openhands-agent-server REST API::Breaking REST API change detected without MINOR version bump (1.11.4 -> 1.11.5).

Breaking REST API changes detected compared to previous release:
- added '#/components/schemas/GraySwanAnalyzer-Output, #/components/schemas/LLMSecurityAnalyzer-Output' to the '/items/anyOf[subschema #1: ConversationInfo]/security_analyzer/anyOf[subschema #1: LLMSecurityAnalyzer -> #/components/schemas/SecurityAnalyzerBase-Output]/' response property 'oneOf' list for the response status '200'
- the '/items/anyOf[subschema #1: ConversationInfo]/security_analyzer/anyOf[subschema #1: LLMSecurityAnalyzer -> #/components/schemas/SecurityAnalyzerBase-Output]/' response's property type/format changed from 'object'/'' to 

Action log

Copy link
Collaborator

@all-hands-bot all-hands-bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Linus-Style Engineering Assessment

🟢 Good taste - This is exactly the kind of cleanup that should happen when you fix the root cause of a problem.

Key Insight: The fallback was a band-aid for using an outdated SDK (claude-code-acp@0.16.2). Once the SDK was updated to claude-agent-acp@0.20.2, the band-aid became unnecessary complexity. Removing ~236 lines of code that solve a problem that no longer exists is good engineering.

Verdict: ✅ Worth merging after verification

Important: This touches cost tracking which could affect benchmarks/evals. Before merging, verify the unchecked test plan item - confirm the CI run-examples workflow shows non-zero cost without the fallback. Once verified, this is a clean simplification with no issues.

@github-actions
Copy link
Contributor

github-actions bot commented Mar 5, 2026

Coverage

Coverage Report •
FileStmtsMissCoverMissing
openhands-sdk/openhands/sdk/agent
   acp_agent.py2945581%113–115, 169–172, 174–175, 202, 204, 208, 214, 225–226, 231, 298, 359–360, 397, 407, 412, 423–426, 432–434, 437–439, 441, 443–444, 446, 448, 453, 461–462, 466–467, 471, 478, 481–482, 484, 486–488, 523, 527–528, 652–653
TOTAL19493571070% 

@simonrosenberg simonrosenberg requested a review from neubig March 5, 2026 19:26
@simonrosenberg
Copy link
Collaborator Author

simonrosenberg commented Mar 5, 2026

@neubig you were right to question the necessity of the fallback: bumping the acp version was the real fix

https://github.com/OpenHands/software-agent-sdk/actions/runs/22732193090

Copy link
Collaborator

@all-hands-bot all-hands-bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Clean removal of unnecessary fallback logic. The SDK update (claude-agent-acp@0.20.2) fixed the root cause, making this code obsolete. Removing ~236 lines of workaround code is good engineering.

@simonrosenberg simonrosenberg merged commit 6e99789 into main Mar 5, 2026
35 of 36 checks passed
@simonrosenberg simonrosenberg deleted the fix/remove-acp-cost-fallback branch March 5, 2026 19:35
zparnold pushed a commit to zparnold/software-agent-sdk that referenced this pull request Mar 5, 2026
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants