One-Line Summary
A systematic re-examination of tool graph memory (ToolNet, H-EPM) across nine open-weight models and two multi-turn tool-use benchmarks shows that it rarely improves significantly over a memory-free agent, that its occasional gains cost 2.72× the LLM calls and arise from extra tool-calling attempts rather than the memory's (often incorrect) next-tool suggestions, and that storing the same experience as natural-language insight memory performs best in most settings with almost no additional LLM calls (1.03×).
search_holiday, so the agent repeats timestamp_diff with an assumed date until the turn limit (58 LLM calls). Following the insight, the agent retrieves the holiday date first and succeeds in 4 LLM calls.Background & Motivation
LLM agents increasingly solve complex tasks by calling external tools over multiple turns, and memory lets them retain and reuse experience from previous interactions. One prominent design is tool graph memory: successful trajectories are turned into a graph whose nodes are tools and whose weighted, directed edges are tool transitions. ToolNet shows the agent only the successors of the previously invoked tool, using edge weights as preference scores. H-EPM keeps the full tool set and treats the weighted transitions as procedural memory and the edge-attached context summaries as episodic memory; at each step the agent may summarize its state and then receives the two successor tools whose stored contexts best match that summary (or the two highest-weight successors otherwise).
Both methods provide step-level next-tool suggestions. Whether such guidance consistently helps agents across multi-turn interactions, however, had not been established with repeated runs, paired significance tests, or accounting of the extra computation it incurs.
Key questions: (1) Does tool graph memory consistently improve over memory-free baselines across model families, sizes, and benchmarks? (2) How much additional computation does it cost in LLM calls and tokens? (3) Where gains do exist, do they really come from the memory's suggestions? (4) Which failure modes remain even when the suggested tool is always correct, and what form of guidance would resolve them?
The answers turn out to be surprising: the gains are rare, expensive, and largely attributable to the agent simply attempting more tool calls, while even oracle-correct suggestions leave many failures untouched. This motivates rethinking memory as a soft prior about what to do in a situation rather than a prescription of the next tool.
Analysis & Proposed Method
The paper proceeds in four stages, moving from a broad empirical audit of tool graph memory to a lightweight alternative, insight memory, built from the very same experience.
Experimental Results
Task Performance (mean of three runs; + / − = significant gain / loss vs. Base)
| Model | TS Base | TS ToolNet | TS H-EPM | TS Insight | BFCL Base | BFCL ToolNet | BFCL H-EPM | BFCL Insight |
|---|---|---|---|---|---|---|---|---|
| Qwen3.5-2B | 0.420 | 0.428 | 0.436 | 0.488+ | 0.133 | 0.044− | 0.138 | 0.165 |
| Qwen3-8B | 0.500 | 0.507 | 0.476 | 0.556+ | 0.171 | 0.129 | 0.179 | 0.242+ |
| Qwen3.6-35B-A3B | 0.626 | 0.608 | 0.670 | 0.674 | 0.181 | 0.108− | 0.156 | 0.206 |
| gemma-4-E4B-it | 0.588 | 0.547− | 0.554 | 0.620 | 0.221 | 0.094− | 0.252 | 0.285+ |
| gemma-4-26B-A4B-it | 0.690 | 0.655− | 0.705 | 0.705 | 0.566 | 0.369− | 0.519 | 0.531 |
| Ministral-3-8B | 0.631 | 0.588− | 0.672+ | 0.539− | 0.209 | 0.141− | 0.328+ | 0.306+ |
| Mistral-Small-3.2-24B | 0.545 | 0.557 | 0.603+ | 0.620+ | 0.231 | 0.127− | 0.250 | 0.338+ |
| gpt-oss-20b | 0.579 | 0.573 | 0.620 | 0.630+ | 0.325 | 0.152− | 0.369 | 0.427+ |
| gpt-oss-120b | 0.726 | 0.674− | 0.713 | 0.734 | 0.494 | 0.275− | 0.559+ | 0.454 |
TS = ToolSandbox (207 scenarios), BFCL = multi-turn subset of BFCL-v4 (160 tasks). Frontier models show the same pattern on ToolSandbox: GPT-4.1 scores 0.724 / 0.709 / 0.725 / 0.729 and GPT-5.1 scores 0.701 / 0.694 / 0.702 / 0.702 for Base / ToolNet / H-EPM / Insight.
Inference Cost on ToolSandbox (nine-model average per scenario; ratio to Base in parentheses)
| Metric | Base | ToolNet | H-EPM | Insight |
|---|---|---|---|---|
| LLM calls | 5.37 | 5.15 (0.96×) | 14.62 (2.72×) | 5.54 (1.03×) |
| Tokens | 8,681 | 5,937 (0.68×) | 19,225 (2.21×) | 12,404 (1.43×) |
Ablations on ToolSandbox (tool calls relative to Base in flipped scenarios in parentheses)
| Condition | Mistral-Small-3.2-24B | gpt-oss-20b | gemma-4-E4B-it |
|---|---|---|---|
| Base (only LLMs) | 0.545 | 0.579 | 0.588 |
| Tool graph memory (H-EPM) | 0.603+ (2.0) | 0.620 (1.8) | 0.554 (2.0) |
| H-EPM with random tools | 0.571 (2.3) | 0.663+ (2.7) | 0.553 (2.0) |
| H-EPM with the correct tool (oracle) | 0.682+ (1.6) | 0.643+ (2.5) | 0.666+ (2.0) |
| H-EPM w/o suggestions | 0.602+ (1.7) | 0.633+ (2.7) | 0.598 (1.9) |
| H-EPM w/o summarization | 0.595+ (1.8) | 0.633+ (2.6) | 0.604 (1.8) |
| Base + instruction | 0.594 (1.9) | 0.645+ (2.7) | 0.621+ (2.0) |
| Insight memory + H-EPM | 0.600+ (2.5) | 0.585 (2.7) | 0.617 (2.8) |
| Insight memory + instruction | 0.671+ (2.2) | 0.675+ (3.0) | 0.640 (2.7) |
- Tool graph memory rarely helps: ToolNet underperforms Base in most cases, including all nine models on BFCL-v4, with a statistically significant drop in 12 settings. H-EPM is numerically above Base in most settings but its gain is significant in only four.
- Gains are expensive: H-EPM uses on average 2.72× as many LLM calls and 2.21× as many tokens as Base on ToolSandbox, with a similar increase in LLM calls on BFCL-v4. Neither method offers a better performance–cost trade-off than Base.
- Suggestions are mostly wrong: In scenarios that H-EPM fixes on Mistral-Small-3.2-24B, the suggestion is correct in only 30% of decisive turns; in another 50% the agent calls the necessary tool on its own despite a wrong suggestion. In degraded scenarios, 98% of suggestions at decisive turns are incorrect.
- The benefit comes from more tool calls: Replacing suggestions with random tools or removing them entirely performs comparably to H-EPM (and w/o suggestions beats it on two of three models). Scenarios flipped to success make roughly twice as many tool calls as Base, and a single “keep trying tools” instruction matches or exceeds H-EPM with almost no extra LLM calls.
- Correct next tools are not enough: Even the oracle leaves many failures; with tool selection errors reduced to 4%, the rest stem from call necessity (e.g., giving up without a tool call), dependency, and argument errors.
- Insight memory is the best trade-off: Insight is best in most settings of Table 1, significant over Base in far more settings than either tool graph memory, attains the highest pass@3 in almost all settings, keeps LLM calls at 1.03× Base, and recovers most scenarios H-EPM degrades, including every dependency and argument failure. Combining it with H-EPM is worse on seven of nine models and needs 2.68× the LLM calls, while Insight + instruction is the best non-oracle condition on all three ablation models and even surpasses the oracle on gpt-oss-20b.
Why It Matters
Memory for tool-use agents has gravitated toward structured representations that prescribe the next action. This paper provides the first rigorous, significance-tested audit of that idea for tool graph memory and shows that the apparent benefit is largely an artifact of extra computation: the same boost is available from a one-line instruction, and the graph's guidance is more often misleading than helpful.
- Rigor for the memory literature. Repeated runs, paired tests, and cost accounting across nine open-weight and two frontier models expose how easily single-run gains can be mistaken for a real effect, setting a higher evidentiary bar for future agent-memory claims.
- A diagnosis, not just a negative result. The failure taxonomy (call necessity, tool selection, dependency, argument) explains why even perfect next-tool suggestions fall short: tool transitions do not tell an agent whether, when, or with what arguments to call a tool.
- A practical, generalizable alternative. Insight memory distills the same episodic experience into situation-level semantic knowledge, requires no training, adds negligible inference cost, and transfers across scenarios. Its advantage also holds for ExpeL-style insights and for insights built from successful trajectories only, supporting the broader conclusion that memory works better as a soft prior than as a prescription of the next tool.