Generative AI can enhance SAP applications in areas such as document analysis, summarizing business information, semantic search, and providing users with dialog-based support.<\/p>\n
For SAP on-premises systems, this raises a fundamental architectural question: Should the large language model be accessed via an external cloud service or run within the organization\u2019s own infrastructure?<\/p>\n
Cloud-based models from OpenAI and Anthropic<\/strong><\/p>\n OpenAI and Anthropic provide powerful language models via managed APIs. Companies do not need their own GPU infrastructure and do not have to worry about deploying or scaling the models.<\/p>\n The SAP application or an integration layer sends requests via HTTPS to the cloud endpoint and receives the generated response. This approach allows for a quick start, but results in external data flows and a dependency on the network connection, pricing model, availability, and technical specifications of the respective provider.<\/p>\n Particularly when dealing with sensitive SAP data, data protection, data classification, logging, retention periods, and regulatory requirements must be taken into account. For certain use cases, a local inference architecture may therefore be more appropriate.<\/p>\n Local AI with the Milliarum AI Construction Kit<\/strong><\/p>\n With the local approach, the SAP system, integration logic, and model inference remain within the company's controlled infrastructure or a private cloud environment.<\/p>\n One possible architecture consists of the following components:<\/p>\n SAP On-Premise \u2192 Milliarum AI Construction Kit \u2192 vLLM Connector \u2192 Gemma 31B \u2192 NVIDIA GPU<\/strong><\/p>\n The Milliarum AI Construction Kit serves as the integration layer between the SAP system, the business use cases, and the locally operated language model.<\/p>\n Milliarum provides a connector for this purpose that allows vLLM to communicate via a\u00a0OpenAI-compatible API<\/strong>\u00a0connects to. This means that SAP-related applications do not need to be developed directly for a proprietary inference interface. They communicate with the local model service via an established API format.<\/p>\n This technical abstraction makes it easier to swap out models, quantization formats, and inference engines without having to reimplement the entire SAP integration.<\/p>\n Gemma 31B with 4-bit quantization<\/strong><\/p>\n For local use, Gemma 31B offers an interesting balance between model size, performance, and hardware requirements.<\/p>\n A model of this scale has sufficient capacity for more demanding tasks such as text analysis, structured response generation, summarization, subject-matter assistance, or the processing of extensive contexts. At the same time, the memory requirements for smaller systems are comparatively high when full numerical precision is maintained.<\/p>\n Through a\u00a04-bit quantization<\/strong>\u00a0The model weights are stored with reduced precision. This significantly reduces the amount of GPU memory required. Depending on the quantization method used, this may result in a limited loss of quality, but it enables much more compact operation.<\/p>\n Gemma 31B with 4-bit quantization is therefore a suitable compromise between:<\/p>\n One possible target configuration is a small NVIDIA-based inference computer with a\u00a0RTX PRO 6000 Blackwell and 96 GB of GPU memory<\/strong>. In addition to the quantized model weights, the available memory provides sufficient capacity for the KV cache, longer contexts, parallel queries, and other runtime components.<\/p>\n However, the actual performance achievable does not depend solely on the model size. Key parameters include, in particular, context length, batch size, number of concurrent users, quantization format, and desired response latency. The model\u2019s suitability should therefore be tested using real-world SAP use cases and representative prompts.<\/p>\n vLLM and Possible Alternatives<\/strong><\/p>\n vLLM provides the model as a high-performance inference service. Key benefits include efficient management of the KV cache, continuous batching, streaming outputs, and support for OpenAI-compatible API endpoints. This allows for efficient use of the GPU even when handling multiple concurrent requests.<\/p>\n However, vLLM is not a mandatory requirement. With\u00a0SGLang<\/strong>\u00a0Alternative inference engines are available that also offer OpenAI-compatible interfaces, optimized cache mechanisms, and efficient processing of complex LLM workloads.<\/p>\n The selection should be based on the model used, the quantization, the target hardware, and specific measurement values for throughput, latency, and stability.<\/p>\n Integration into SAP Processes<\/strong><\/p>\n The real added value does not come solely from running a local language model, but from its controlled integration with the relevant SAP processes and data.<\/p>\n Possible use cases include:<\/p>\n The AI Construction Kit allows you to control which information is passed to the model. Existing SAP authorizations, business objects, and process logic can be integrated into the solution.<\/p>\n Even in a fully on-premises environment, a comprehensive security strategy is required. This includes authentication and authorization, network segmentation, encrypted communication, controlled logging, model versioning, and protection against prompt injection and unauthorized data access.<\/p>\n\n
\n