Agent loop stops after first tool call with non-Claude models (GLM-5): generateSandboxedUi never invoked

[kogamishinyajerry-ops]

Summary

When using a non-Claude OpenAI-compatible model (tested with GLM-5 via Zhipu coding endpoint https://open.bigmodel.cn/api/coding/paas/v4), the agent correctly calls plan_visualization on the first LLM turn but never proceeds to a second LLM turn to call generateSandboxedUi. The result is that the user sees the assistant's text description of the widget that "would" be built, but no sandboxed iframe is ever rendered (0 iframes in the DOM).

The README warns that "weaker models will produce broken layouts", but this is a different failure mode: GLM-5 successfully generates a complete, valid widget when prompted directly (9.6 KB of JS, all 6 generateSandboxedUi parameters emitted in the correct order). The failure is not model capability — it's that the agent loop doesn't give the model a second turn after the plan_visualization tool returns.

Environment

• Repo commit: 457e60c (current main as of 2026-07-18)
• Model: glm-5 (Zhipu coding endpoint), wired in via langchain_openai.ChatOpenAI with LLM_MODEL=glm-5, LLM_PROVIDER=openai
• Python 3.13, Node 24, pnpm 10, uv
• macOS Darwin 25.5.0
• Same behavior on fresh thread (cleared localStorage + reopened browser)

Repro

1. Replace apps/agent/src/model.py build_model() so that LLM_MODEL=glm-5 + LLM_PROVIDER=openai routes to ChatOpenAI with the Zhipu base_url.
2. make setup + make dev.
3. Open http://localhost:3000, type "Draw binary search visualization".
4. GLM-5 replies with text + calls plan_visualization. UI shows the "✓ Plan: visualization" pill.
5. Nothing else happens. No iframe, no generateSandboxedUi activity event.

Evidence

What the agent loop sees (first LLM turn) — captured by monkey-patching bind_tools on the model:

[MODEL_DEBUG] bind_tools (19): ['write_todos', 'ls', 'read_file', 'write_file', 'edit_file', 'glob', 'grep', 'task', 'query_data', 'plan_visualization', 'manage_todos', 'get_todos', 'generate_form', 'generateSandboxedUi', 'toggleTheme', 'pieChart', 'barChart', 'scheduleTime', 'render_a2ui']

generateSandboxedUi IS in the offered tools. GLM-5 returns:

finish_reason: tool_calls
content: "I'll create an interactive binary search visualization that shows how the algorithm narrows down the search space step by step. Let me plan and build this for you."
tool_calls: [plan_visualization(approach=..., technology=..., key_elements=[...])]

After plan_visualization returns its string, no further LLM call is made — verified by hooking both _generate / _agenerate and the underlying httpx client; zero additional HTTP requests to /chat/completions after the first turn.

Direct replay against GLM-5 API with the same system prompt + same tool list + the plan_visualization tool result already in messages:

messages = [
    {"role":"system","content":"You MUST call generateSandboxedUi now that plan is done."},
    {"role":"user","content":"Draw binary search visualization"},
    {"role":"assistant","content":"I'll build this.","tool_calls":[{"id":"call_1",...,"function":{"name":"plan_visualization","arguments":"..."}}]},
    {"role":"tool","tool_call_id":"call_1","content":"Plan: step-by-step\nTech: SVG\n- array\n- pointers\n- controls"},
]

GLM-5 returns:

finish_reason: tool_calls
content: "" (empty)
tool_calls: [generateSandboxedUi(
    initialHeight=580,
    placeholderMessages=[3 items],
    css="<3201 chars>",
    html="<2086 chars>",
    jsFunctions="<9673 chars>",   # full binary search algorithm + UI
    jsExpressions=["<347 chars>"]
)]
usage: completion_tokens=4839 (incl. 87 reasoning_tokens)

So the model is fully capable of producing the widget in a single shot once the plan_visualization tool result is fed back — but the agent loop never feeds it back.

Hypothesis

When finish_reason=tool_calls and content is non-empty simultaneously (GLM-5's default style — it emits a short sentence alongside the tool call), some layer in deepagents + CopilotKitMiddleware + ag-ui-langgraph appears to treat the message as a final response and does not run the tool-→-LLM-continue cycle. Claude models, per the README's recommended config, apparently emit plan + widget in a single turn or in a way that sidesteps this path; GLM emits them across two turns and gets stuck between.

I tried to nail down the exact layer but the LangChain 1.x callback surface (on_chat_model_start etc.) doesn't fire for the calls that GLM-5 actually answers, so I couldn't trace whether deepagents ran the second turn and it was dropped at the CopilotKit middleware, or whether deepagents itself never re-entered the model.

Suggested fixes / things to try

1. Force a second LLM turn after plan_visualization even when the first turn's message has both content and tool_calls. The current behavior looks like a short-circuit on "has content → treat as final".
2. Or allow plan_visualization to be skipped so a model that prefers to call generateSandboxedUi directly can do so in a single turn.
3. Or document explicitly that non-Claude models must emit plan_visualization and generateSandboxedUi in a single assistant turn, with a system-prompt nudge that makes GLM/etc. do that.
4. Investigate whether ag-ui-langgraph or CopilotKitMiddleware is the layer that drops the continuation — happy to dig further if a maintainer can point me at where to instrument.

Reproducibility script

I have a self-contained Python script that hits GLM-5 directly with the same tool schemas and prompts, reproducing the "2-turn happy path" reliably — can paste it in if useful.

Why I'm reporting this

The project is marketed as "open-source showcase for AI-generated UI" and the README explicitly supports gpt-* fallback via LLM_MODEL. Any user trying it with a non-Claude OpenAI-compatible endpoint (GLM / DeepSeek / Qwen / Kimi / MiniMax / etc.) will hit this silent half-failure and conclude the model is too weak, when in fact it's an agent-loop bug. Worth either fixing or documenting.

[copilotkit-outpost-bot]

🎫 Ticket TKT-4GPJZ7G7 created. Our AI assistant is reviewing your issue...

[copilotkit-outpost-bot]

⚠️ This is an AI-generated response with low confidence. A human agent has been notified and will follow up.

Great Bug Report — Here's What's Likely Happening and How to Work Around It

This is a well-documented issue and your hypothesis is correct. The root cause is almost certainly in how the LangGraph agent loop handles messages where content is non-empty and tool_calls is present simultaneously.

---

Root Cause: The content + tool_calls Short-Circuit

LangGraph's default ToolNode + should_continue edge pattern typically checks something like:

Code example (python)

def should_continue(state):
    last_message = state["messages"][-1]
    if last_message.tool_calls:
        return "tools"
    return END

The problem is upstream of this — when CopilotKit's middleware (or ag-ui-langgraph) processes the streamed response, it may be treating a message with non-empty content as a "final" assistant message and emitting it as a terminal event before the tool call routing logic runs.

Claude typically returns content="" when making tool calls, so this path is never hit with the recommended model.

---

Immediate Workarounds

Option 1: Strip content from tool-call messages (Recommended)

Monkey-patch your model to clear content when tool_calls is present:

</details>python
# apps/agent/src/model.py

from langchain_core.messages import AIMessage
from langchain_openai import ChatOpenAI

class ContentStrippingChatOpenAI(ChatOpenAI):
    """
    GLM-5 (and similar models) emit non-empty content alongside tool_calls.
    Some layers in the CopilotKit/ag-ui-langgraph stack treat non-empty
    content as a terminal message. Strip it to match Claude's behavior.
    """
    
    async def _agenerate(self, messages, stop=None, run_manager=None, **kwargs):
        result = await super()._agenerate(messages, stop=stop, run_manager=run_manager, **kwargs)
        result.generations = [
            self._strip_content_if_tool_calls(gen)
            for gen in result.generations
        ]
        return result

    def _generate(self, messages, stop=None, run_manager=None, **kwargs):
        result = super()._generate(messages, stop=stop, run_manager=run_manager, **kwargs)
        result.generations = [
            self._strip_content_if_tool_calls(gen)
            for gen in result.generations
        ]
        return result

    @staticmethod
    def _strip_content_if_tool_calls(gen):
        msg = gen.message
        if isinstance(msg, AIMessage) and msg.tool_calls and msg.content:
            # Preserve content as a separate message or just drop it
            object.__setattr__(msg, 'content', '')
        return gen


def build_model():
    import os
    if os.getenv("LLM_PROVIDER") == "openai" and os.getenv("LLM_MODEL", "").startswith("glm"):
        return ContentStrippingChatOpenAI(
            model=os.getenv("LLM_MODEL"),
            base_url="https://open.bigmodel.cn/api/coding/paas/v4",
            api_key=os.getenv("LLM_API_KEY"),
        )
    # ... rest of your model routing
<details><summary>Code example</summary>

Option 2: Add a system prompt nudge to suppress content on tool calls

Less reliable, but zero code change:

SYSTEM_PROMPT_SUFFIX = """
IMPORTANT: When calling any tool, output ONLY the tool call with NO preceding text or explanation. 
Do not write any message content when making tool calls. After plan_visualization completes, 
immediately call generateSandboxedUi without any text.
"""

Option 3: Make plan_visualization optional in the graph

If you control the LangGraph graph definition, add a direct edge that allows skipping plan_visualization:

</details>python
# In your graph definition
graph.add_conditional_edges(
    "agent",
    should_continue,
    {
        "tools": "tools",
        "end": END,
    }
)

# Ensure tools node always routes back to agent
graph.add_edge("tools", "agent")  # ← verify this exists for ALL tool returns
<details><summary>Code example</summary>

The key thing to verify is that plan_visualization's tool node routes back to "agent" and not to END.

---

Diagnosing the Exact Layer

To pinpoint whether it's ag-ui-langgraph or CopilotKitMiddleware, add this to your graph:

from langgraph.graph import StateGraph
from langchain_core.runnables import RunnableConfig

# Add a passthrough node after tools to log state
def debug_after_tools(state, config: RunnableConfig):
    messages = state.get("messages", [])
    last = messages[-1] if messages else None
    print(f"[DEBUG] After tools node — last message type: {type(last).__name__}")
    if hasattr(last, 'content'):
        print(f"[DEBUG] content={repr(last.content[:100])}")
    return state  # passthrough

graph.add_node("debug_after_tools", debug_after_tools)
graph.add_edge("tools", "debug_after_tools")
graph.add_edge("debug_after_tools", "agent")

If you see [DEBUG] After tools node in logs but no subsequent model HTTP call, the short-circuit is in the agent node itself (likely the should_continue conditional). If you never see the debug log, the issue is before the tools node runs.

---

Filing This Upstream

This is worth opening as a GitHub issue on the CopilotKit repo. Key points to include:

• The content + tool_calls co-presence behavior
• That it affects any OpenAI-compatible model that isn't Claude (DeepSeek, Qwen, Kimi, MiniMax, GLM all do this)
• Your reproducibility script would be very valuable

👉 File at: https://github.com/CopilotKit/CopilotKit/issues/new

---

Would you like help writing the minimal reproduction script, or do you want me to dig into the specific ag-ui-langgraph event emission code to pinpoint the exact line causing the short-circuit?

---

_{🤖 Generated by CopilotKit AI Support · Documentation}

---

Was this helpful? React with 👍 or 👎