Generative AI in SAP On-Premise: Working with SAP Data in the Local Chatbot

Where does generative AI provide real added value when it comes to business data from SAP? The first areas of application are emerging in the form of chat-based interactions with SAP data. For example, this allows users to query business data such as product master data, documentation, bills of materials, or project structures. A second scenario involves simplifying the handling of complex SAP processes: Using natural language, users can access, modify, or even create sales orders, projects, or products. First, an internal decision must be made as to whether the chatbot is permitted to access company-specific data and, consequently, extract and process knowledge from it. At this point, at the latest, the question arises: cloud or on-premises?

On-Premises or Cloud: Security Versus Innovation?

Many companies use SAP on-premises because the security-related benefits outweigh the drawbacks: Despite higher resource requirements, these systems offer greater control over sensitive data. With the appropriate investment, on-premises systems can be tailored to a company’s individual needs—another argument against adopting cloud systems. The same applies to the use of AI: The further away the data is in the cloud, the less control the company retains over data security. A hotly debated topic in SAP circles is that AI developments from Walldorf focus almost exclusively on cloud tools. Given this outlook, companies using SAP on-premises face a dilemma regarding AI: they must either switch to a cloud-based solution, implement a different ERP system, or remain without access to AI capabilities.

Server farms are usually unnecessary

With these companies in mind, the Milliarum AI Construction Kit was recently launched. It allows users to expand their SAP systems with AI-based features. With preconfigured functions for OpenAI, Azure OpenAI, and locally operated open-source models, chatbots can be quickly rolled out across organizations. The first application-specific chatbots for SAP On-Premise are set to follow shortly. These will allow companies to select their own large language model and connect it locally to their own AI system. After all, to interact with your own SAP data, you don’t need a massive chatbot that speaks over 100 languages and has been trained with several hundred billion parameters if, in the end, only German is to be used for limited use cases. A server farm is therefore unlikely to be necessary for most use cases in the SAP environment.

Open-Source LLM Available

Once the goals and use cases for AI in the SAP environment have been defined, the next step is to select a hosting provider that meets the company’s security requirements. One option is to choose one of the major players in the market: OpenAI is a U.S.-based cloud-hosting provider, while Microsoft Azure offers comparable features and complies with European regulations. An alternative is local open-source large language models (LLMs) running on the company’s own hardware. These can be found for various use cases on platforms such as Hugging Face. The Fraunhofer Institute recently recommended German-language-optimized models based on Mistral 7b. It already runs on hardware with performance requirements similar to those of a gaming PC and is available starting at around 10,000 euros. The open-source model Mixtral 8x22b is currently considered the most powerful European LLM, fluent in five languages and trained with over 140 billion parameters. It requires at least 300 GB of graphics memory for productive inference operations, which does not necessarily have to be provided by an NVIDIA data center. Even older generations of NVIDIA graphics cards are fully sufficient in terms of performance and can be used cost-effectively.

The ecological footprint is relatively small

In addition to the financial aspect, choosing an open-source model on your own hardware saves a significant amount of electricity compared to the massive data centers that typically underpin cloud-based applications. They do not require company-owned hardware and are usually based on a usage fee depending on the number of words entered. Depending on the application scenario, this can quickly become a significant cost factor for companies. With proper preparation, cloud-based systems can be up and running in as little as one day. Local applications, on the other hand, require the purchase of hardware. This hardware must then be integrated into the company’s network, the AI model must be deployed on it, and—depending on the intended use—it must be trained using the company’s own parameters. Although the implementation is complex, operation meets the company’s own security requirements, and the ongoing costs are significantly lower than those of a cloud-based AI system, where each API call is billed.

Moving Toward Autonomous SAP Processes

The better the quality of the SAP data, the more powerful the generative AI built on top of it will be. Preparing the input data is a crucial step in the process of implementing AI. However, SAP data provides a relatively simple foundation for such systems. After all, the business data is already available in the SAP software. However, it must first be extracted from the system in order to be processed and then fed back into the software. Experts are currently also grappling with the question of to what extent the use of AI ultimately yields any productivity benefits in the SAP environment. In particular, there appears to be significant potential for rapid optimization when it comes to bottleneck resources. While AI is currently used primarily as a chat function and in programming environments to generate reports directly from SAP data and to create and modify SAP objects, it should be possible in the future to operate entire processes in SAP using natural language. In this process, data is stored in such a way that the AI can access it through training. It is already foreseeable that processes—including machines—will increasingly be automated and controlled in the background by AI, with the chat function serving as the sole user interface. Nevertheless, the use of generative AI in the SAP environment is still in its infancy: its actual benefits—whether cloud-based or on-premises—have yet to be proven.