AI with SAP On-Premise and a Local LLM: Data Sovereignty Through a Controllable Inference Architecture

Generative AI can enhance SAP applications in areas such as document analysis, summarizing business information, semantic search, and providing users with dialog-based support.

For SAP on-premises systems, this raises a fundamental architectural question: Should the large language model be accessed via an external cloud service or run within the organization’s own infrastructure?

Cloud-based models from OpenAI and Anthropic

OpenAI and Anthropic provide powerful language models via managed APIs. Companies do not need their own GPU infrastructure and do not have to worry about deploying or scaling the models.

The SAP application or an integration layer sends requests via HTTPS to the cloud endpoint and receives the generated response. This approach allows for a quick start, but results in external data flows and a dependency on the network connection, pricing model, availability, and technical specifications of the respective provider.

Particularly when dealing with sensitive SAP data, data protection, data classification, logging, retention periods, and regulatory requirements must be taken into account. For certain use cases, a local inference architecture may therefore be more appropriate.

Local AI with the Milliarum AI Construction Kit

With the local approach, the SAP system, integration logic, and model inference remain within the company's controlled infrastructure or a private cloud environment.

One possible architecture consists of the following components:

SAP On-Premise → Milliarum AI Construction Kit → vLLM Connector → Gemma 31B → NVIDIA GPU

The Milliarum AI Construction Kit serves as the integration layer between the SAP system, the business use cases, and the locally operated language model.

Milliarum provides a connector for this purpose that allows vLLM to communicate via a OpenAI-compatible API connects to. This means that SAP-related applications do not need to be developed directly for a proprietary inference interface. They communicate with the local model service via an established API format.

This technical abstraction makes it easier to swap out models, quantization formats, and inference engines without having to reimplement the entire SAP integration.

Gemma 31B with 4-bit quantization

For local use, Gemma 31B offers an interesting balance between model size, performance, and hardware requirements.

A model of this scale has sufficient capacity for more demanding tasks such as text analysis, structured response generation, summarization, subject-matter assistance, or the processing of extensive contexts. At the same time, the memory requirements for smaller systems are comparatively high when full numerical precision is maintained.

Through a 4-bit quantization The model weights are stored with reduced precision. This significantly reduces the amount of GPU memory required. Depending on the quantization method used, this may result in a limited loss of quality, but it enables much more compact operation.

Gemma 31B with 4-bit quantization is therefore a suitable compromise between:

• Model quality and number of parameters,
• required GPU memory,
• achievable inference speed,
• Energy and infrastructure needs,
• technical feasibility on compact systems.

One possible target configuration is a small NVIDIA-based inference computer with a RTX PRO 6000 Blackwell and 96 GB of GPU memory. In addition to the quantized model weights, the available memory provides sufficient capacity for the KV cache, longer contexts, parallel queries, and other runtime components.

However, the actual performance achievable does not depend solely on the model size. Key parameters include, in particular, context length, batch size, number of concurrent users, quantization format, and desired response latency. The model’s suitability should therefore be tested using real-world SAP use cases and representative prompts.

vLLM and Possible Alternatives

vLLM provides the model as a high-performance inference service. Key benefits include efficient management of the KV cache, continuous batching, streaming outputs, and support for OpenAI-compatible API endpoints. This allows for efficient use of the GPU even when handling multiple concurrent requests.

However, vLLM is not a mandatory requirement. With SGLang Alternative inference engines are available that also offer OpenAI-compatible interfaces, optimized cache mechanisms, and efficient processing of complex LLM workloads.

The selection should be based on the model used, the quantization, the target hardware, and specific measurement values for throughput, latency, and stability.

Integration into SAP Processes

The real added value does not come solely from running a local language model, but from its controlled integration with the relevant SAP processes and data.

Possible use cases include:

• Summaries of project, assignment, or report information,
• Explanation of complex SAP data,
• Analysis and classification of documents,
• Generation of context-based text suggestions,
• Use of internal documents and knowledge resources,
• AI-powered assistance features in SAP-related interfaces.

The AI Construction Kit allows you to control which information is passed to the model. Existing SAP authorizations, business objects, and process logic can be integrated into the solution.

Even in a fully on-premises environment, a comprehensive security strategy is required. This includes authentication and authorization, network segmentation, encrypted communication, controlled logging, model versioning, and protection against prompt injection and unauthorized data access.

Conclusion

Cloud APIs from OpenAI and Anthropic make it easy to get started quickly and reduce the need for in-house infrastructure. A local architecture, on the other hand, offers more control over data flows, models, operating parameters, and dependencies.

The combination of SAP On-Premise, the Milliarum AI Construction Kit, a OpenAI-Compatible vLLM Connector, Gemma 31B with 4-bit quantization and a compact NVIDIA inference platform provide a technically feasible foundation for on-premises AI applications.

A limited proof of concept using real SAP processes should clarify what model quality, response latency, and parallelism can be achieved under the intended operating conditions.