Programmatic Tool Calling: Offloading Token-Intensive Operations from the LLM Context
Many AI applications do not reach the model's performance limit, but rather the limit of the available context window.
In classic tool calling, a process proceeds step by step: The language model calls a function, receives its result, evaluates it, and then decides on the next call. This approach works very well for a small number of queries. However, when processing large datasets, extensive document volumes, or multiple SAP data sources, the context grows with each intermediate result. The consequences are higher token consumption, longer runtime, and, in the worst case, a context overflow.
Programmatic Tool Calling is changing this division of labor.
The model generates a program that coordinates the released tools using loops, conditions, or parallel calls. The code runtime can filter, merge, validate, and aggregate the returns. Only the aggregated result is returned to the model context.
The language model thus remains responsible for planning, interpretation, and clear presentation. The repeatable and data-intensive processing, on the other hand, is offloaded to code.
OpenAI: JavaScript Orchestration in the Responses API
OpenAI implements Programmatic Tool Calling using an isolated V8 runtime. The model generates JavaScript, coordinates the approved tools within it, and processes intermediate results within the program. This approach is particularly well-suited for clearly defined workflows in which results can be filtered, merged, aggregated, or validated. If, on the other hand, each individual step requires a new semantic evaluation by the model, direct tool calling remains the better choice. (OpenAI Developers)
Anthropic: Python in the Code Execution Container
Anthropic follows the same basic principle using Python in an isolated code-execution container. Claude can call tools sequentially or in parallel, as well as using loops and conditionals. The intermediate results are processed within the code and are not fully transferred to the context window. Only the final program output is passed back to the model. This reduces the number of model calls and token consumption, particularly in data-intensive multi-tool processes. (Claude Platform)
Milliarum: The Bridge to SAP in Practice
For SAP companies, a technical feature offered by the model provider alone is not enough. What matters most is how it is integrated into existing data access, permissions, and business functions. The Milliarum AI Construction Kit It provides an ABAP-based framework for this purpose, featuring UI5 chat, configurable LLM connectors, function calling, and extensible SAP functions. Various models and APIs—including OpenAI and Anthropic—can be integrated via the connector architecture. Customizing is used to determine which models, functions, and data sources are available to an application. (Milliarum)
By expanding the connector configuration, Milliarum also provides the necessary parameters for Programmatic Tool Calling available. This allows the relevant mechanisms of the OpenAI and Anthropic APIs to be used within the existing architecture of the AI Construction Kit.
SAP functions and defined data access methods remain the controlled tools in this process. Orchestration and the aggregation of extensive return values are performed programmatically.
Example: Consistency Check Using the Customer Master Data
Instead of loading the entire customer base into the prompt, a generated program can:
Retrieve customer data incrementally using approved SAP functions, apply defined consistency rules, identify duplicates, missing required fields, or conflicting attributes, aggregate anomalies, and return only a compact list of results to the language model.
This also allows for extensive consistency checks to be performed, without the context window overflowing with the complete raw data.
For example, the model receives only the affected customers, the violated validation rule, and the relevant documents. Based on this information, it can explain the results, prioritize them, or formulate specific recommendations for action. At the same time, data access, functionality, and model selection remain controllable via the Milliarum configuration.
A clear division of responsibilities
Programmatic Tool Calling distinguishes between two different types of work:
The code handles data selection, filtering, and calculation.
The language model handles planning, categorization, and communication.
It is precisely this separation that makes the approach appealing for enterprise applications. The goal is not to feed more and more data into an ever-larger prompt. Large volumes of data should be processed where it can be done efficiently and in a controlled manner. The model receives only the information for which linguistic or semantic intelligence is actually required.
With the AI Construction Kit, Milliarum brings the new capabilities of OpenAI and Anthropic into a configurable SAP architecture:
The model plans. The code orchestrates. SAP remains the leading source of data and processes.
