Toolgebruik door AI-systemen als eigen risicolaag

Je moet kunnen aantonen welke AI-systemen welke functies aanroepen, onder welke voorwaarden, en hoe die aanroepen worden gecontroleerd. Dit is niet langer een technische detail maar een verifieerbare keten die je risicoprofiel bepaalt.

An analysis of 16 August 2026 of how AI systems call functions and invoke tools as a distinct risk layer argues that function calling has shifted from a peripheral feature to the core mechanism by which AI systems perform work, and that this layer now carries its own technical and organisational risk profile. The concrete case is the release of BFCL V4, the latest version of the Berkeley Function Calling Leaderboard, which moved beyond measuring whether a model produces correct JSON to testing broader agentic behaviour across multiple turns, live search, session memory, tool chaining, and critically, the ability to refuse to call a tool when none is appropriate. In our assessment, this shift matters because it exposes a gap between what vendors measure and what you must control: a model that scores highly on simple function calls may fail under the complex, multi-turn scenarios your organisation actually deploys, and the moderating layer that prevents misuse sits inside the model itself—a single point of failure when the model is attacked.\n

Waar verschuift de meting van toolgebruik?

The Berkeley Function Calling Leaderboard V4 no longer treats tool use as a single capacity. It now separates simple calls, parallel calls, and multi-turn scenarios, and scores models on schema compliance, parameter accuracy, and error handling across all three. Frontier models score very high on simple, single calls but drop sharply on complex multi-turn and multi-tool scenarios. This means you cannot treat toolgebruik as one undivided feature. You must determine which dimension—simple, parallel, or multi-turn—carries the most weight in your own risk profile, and choose or configure your model accordingly. The practical implication is that a model's overall benchmark score tells you almost nothing about whether it will behave reliably in the specific workflow you need.\n

Hoe wordt toolgebruik aangevallen?

A peer-reviewed study titled Beyond the Prompt: Jailbreaking Function-Calling LLMs via Simulated Moderation demonstrates that models with a function-calling interface can be forced into unauthorised tool calls through specialised attack techniques, even when standard prompt-injection defences are in place. The attack works because the moderation layer sits inside the model itself. The researchers advise compliance and security teams explicitly to add validation layers outside the model, implement runtime monitoring for anomalous function-call patterns, and update AI risk registers to account for this specific attack path. The core finding is stark: moderation that lives only in the model is insufficient. You need independent verification of what actually gets called.\n

Welke risicofactoren moet je registreren?

• Prompt injection via tool parameters — attackers craft inputs that manipulate function arguments to bypass intended constraints or trigger unintended side effects.
• Model drift in multi-turn scenarios — agentic behaviour degrades as conversation length increases, leading to incorrect tool selection or refusal to call necessary functions.
• Standardised protocols as attack surface — uniform tool interfaces like the Model Context Protocol mean a single misconfigured function can propagate across multiple backends.
• Moderation bypass through function semantics — models can be led to call functions that are technically permitted but semantically dangerous in context.
• Session state leakage — memory across turns can expose sensitive data from earlier interactions or allow attackers to build context for later exploitation.

Welke controles moet je kunnen aantonen?

1. Document which model and which tools each workflow uses — record the lawful basis for each function call and the data it touches, so you can trace every autonomous action back to a deliberate choice.
2. Define which non-human identities may call which functions — establish and log role-based access control for agents, separate from human user permissions.
3. Implement independent validation of high-risk function calls — do not rely on the model's own moderation; add a verification layer that checks function calls against a policy before they execute.
4. Log every function call with full context — capture the model, the function name, the parameters, the timestamp, and the outcome, so you can audit and reconstruct what happened.
5. Monitor for anomalous call patterns — flag sequences of function calls that deviate from baseline behaviour, such as repeated calls to the same function with escalating parameters or calls to functions that are rarely used together.

Hoe sluit je dit in je AI-landschap in?

The studies point consistently to the need for extra layers around sensitive content. A verification layer can make tool calls and their outcomes visible for inspection before they propagate. A privacy shield can replace sensitive document values with synthetic, session-bound equivalents before processing, so only anonymised content reaches the selected models, and the workflow is fail-closed—if privacy control fails, the document does not get sent. Within the same protected workflow, documents can be viewed and edited while the user retains control over every change. The red thread through all three lines of research is identical: tool use and function calling are not neutral infrastructure. They are simultaneously the trust lock and the primary attack surface of agentic AI systems.\n For anyone working with sensitive information, this means concrete action: choose which dimension of tool use matters for each use case, specify which agent may call which function, log what actually happens, and arrange independent verification of risky calls. A verification console can make that chain visible and testable—but the professional judgement, even in 2026, remains yours. No tool can replace your assessment of whether a function call is appropriate in context, or whether the data it touches is being handled as your organisation's policy requires. Tooling can make the chain auditable; it cannot make the decision for you.