bug while generating ideas for relevant experimental assays

[gabrielpan147]

Hello,

I was running Robin demo, using ollama with local deployed qwen3:32b. However, when running the Experimental assay generation task ,I got an error:

Step 2: Conducting literature search with FutureHouse platform...
2025-06-03 19:41:50,779 - robin.utils - INFO - Starting literature search for 3 queries using job-futurehouse-paperqa2.
2025-06-03 19:49:24,867 - robin.utils - INFO - Finished processing 3 tasks.
2025-06-03 19:49:24,870 - robin.utils - INFO - Successfully wrote: query_1_think.txt to: robin_output/dry_age-related_macular_degeneration_2025-06-03_19-40-36/experimental_assay_literature_reviews/query_1_think.txt
2025-06-03 19:49:24,871 - robin.utils - INFO - Successfully wrote: query_2_how.txt to: robin_output/dry_age-related_macular_degeneration_2025-06-03_19-40-36/experimental_assay_literature_reviews/query_2_how.txt
2025-06-03 19:49:24,871 - robin.utils - INFO - Successfully wrote: query_3_what.txt to: robin_output/dry_age-related_macular_degeneration_2025-06-03_19-40-36/experimental_assay_literature_reviews/query_3_what.txt
2025-06-03 19:49:24,872 - robin.assays - INFO -
Step 3: Generating ideas for relevant experimental assays...

JSONDecodeError                           Traceback (most recent call last)
Cell In[4], line 1
----> 1 candidate_generation_goal = await experimental_assay(configuration=config)

File ~/Robin/robin/robin/assays.py:114, in experimental_assay(configuration)
    105 assay_proposal_messages = [
    106     Message(role="system", content=assay_proposal_system_message),
    107     Message(role="user", content=assay_proposal_user_message),
    108 ]
    110 experimental_assay_ideas = await configuration.llm_client.call_single(
    111     assay_proposal_messages
    112 )
--> 114 assay_idea_json = json.loads(cast(str, experimental_assay_ideas.text))
    115 assay_idea_list = format_assay_ideas(assay_idea_json)
    117 for assay_idea in assay_idea_list:

File ~/anaconda3/envs/fhouse/lib/python3.12/json/__init__.py:346, in loads(s, cls, object_hook, parse_float, parse_int, parse_constant, object_pairs_hook, **kw)
    341     s = s.decode(detect_encoding(s), 'surrogatepass')
    343 if (cls is None and object_hook is None and
    344         parse_int is None and parse_float is None and
    345         parse_constant is None and object_pairs_hook is None and not kw):
--> 346     return _default_decoder.decode(s)
    347 if cls is None:
    348     cls = JSONDecoder
...
    354 except StopIteration as err:
--> 355     raise JSONDecodeError("Expecting value", s, err.value) from None
    356 return obj, end

JSONDecodeError: Expecting value: line 1 column 1 (char 0)

I'm wondering if you have any ideas about it? Thank you for your help!

Best,
Gabriel

[changbenjamin]

Hi Gabriel,

Thanks for flagging this issue! Are you experiencing this issue also with the default LLMs (o4-mini, Claude 4, etc.)? In my experience, these frontier models have few problems with JSON formatting, so perhaps the local deployed qwen3:32b struggles to output its response in the correct JSON format and therefore fails at the json.loads step.

[gel123gel]

I encountered the same issue when using DeepSeek as well. You may not actually be getting valid JSON. You can try printing out experimental_assay_ideas.text with:
logger.info(experimental_assay_ideas.text)

My output appeared to be in Markdown format; After removing the leading ```json and the trailing ```, it runs correctly..

[gabrielpan147]

Hi Gabriel,

Thanks for flagging this issue! Are you experiencing this issue also with the default LLMs (o4-mini, Claude 4, etc.)? In my experience, these frontier models have few problems with JSON formatting, so perhaps the local deployed qwen3:32b struggles to output its response in the correct JSON format and therefore fails at the json.loads step.

Hello Benjamin,

Thank you for your reply! I did not try with those default LLM with Robin. I may need to check the output as @gel123gel points out, and also try with default LLM.

I will let you know how it works!

Gabriel