Skip to content

Code fence tool#165

Open
KeremTurgutlu wants to merge 1 commit intomainfrom
code-fence-tool
Open

Code fence tool#165
KeremTurgutlu wants to merge 1 commit intomainfrom
code-fence-tool

Conversation

@KeremTurgutlu
Copy link
Copy Markdown
Contributor

@KeremTurgutlu KeremTurgutlu commented May 5, 2026

This PR adds code fence tool feature which let's AI to write executable markdown code (python and bash) to bypass tool schema parsing. The main motivation is to make tool calling as simple as code generation for the LLM.

Initial details can be found in this spec doc, and here are some implementation details which may or may not be different:

  • Only implemented and tested with streaming.
  • It's tested with Chat / AsyncChat across all the test models, and additionally tested with mixed tool calling and code fence tool calling. Meaning that AI can write code fence tools and then call a regular tool in the same tool loop, vice versa.
  • Not all APIs accept a stop tokens argument, so we handle stopping during code fence tool exec via a stop_callables mechanism. During streaming we check the collected text from the chunks so far, and if the stop condition is matched the remaining chunks are yielded as reasoning deltas. All the original chunks are still used (but not streamed) to be able to build the final ModelResponse object without failures and also to get the correct usage metadata. Only the final text in ModelResponse is updated with a trimmed version up until the stop condition.
  • An example system prompt for this feature is included in the notebooks for testing but not exported.
  • _lang2tool is hardcoded and not user configurable. it is used to turn on code fence tool feature by looking for python and bash in the tool schema.
  • _split_msg_on_fences is used in fmt2hist -> mk_msgs in the presence of a code fence tool result. This means stop_callables is not generic and expected to be used with FenceToolStop. Same goes for the _active_fence_langs check which automatically adds FenceToolStop to stop_callables kwargs.
  • python and bash tools in ns need to have the following params to work:
rg = dict(code=code) if lang == 'py' else dict(command=code)

Misc

  • I have made some reorg to be able to use display_stream with Chat examples.

@KeremTurgutlu KeremTurgutlu marked this pull request as ready for review May 5, 2026 16:38
@KeremTurgutlu KeremTurgutlu requested a review from jph00 May 5, 2026 17:13
@KeremTurgutlu
Copy link
Copy Markdown
Contributor Author

KeremTurgutlu commented May 7, 2026

@jph00 In your original implementation, code fence tool is auto activated if lang tools are present in the schema which is also the case here. Also, given that stop_callables logic is tied to code fence tool only I can probably simplify the code a bit more in this PR.

Or we can make the system generic adding callbacks, but not sure if there are other use cases:

class FenceToolStop(Stop):
    def __init__(self, langs): self.langs = langs
    def stop_and_trim(self, text):
        "Stop truthy condition optionally return match to trim by"
        m = _fence_re.search(text)
        if m and m.group(1) in self.langs: return m.group(0)
    def after_msgs(self, msgs):
        "Optionally postprocess messages in Chat._prep_msg before api call"
        return _split_fence_msgs(msgs)
    def after_toolcall(self, msgs):
        "Optionally postprocess messages in Chat._call during tool calling step befor api call"
        m = msgs[-1]
        if m.role == 'assistant':
            if fence := extract_fence_call(m.content or ''):
                lang, code = fence
                out = run_fence_tool(lang, code, self.ns)
                m.content += out 
                if stream: yield mk_stream_chunk(content=out, role='assistant')
        return msgs

This can be a follow up PR if you think it's a good direction, I couldn't decide.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant