An insurance company wants to build a course where its claims handlers practise difficult customer conversations. Participants write freely how they would respond to an upset customer, and an AI replies as the customer and gives feedback afterwards. The course is almost finished when IT security asks the question: where do the participants’ answers go?
The answer is often a cloud service run by an American company. That is where employees’ wording ends up, along with examples from real cases and parts of the internal material the course is built on. For many organisations that is fine with the right agreements. For others, such as healthcare, banking, defence or the public sector, it is a full stop.
The alternative is to run the language model yourself. It is often called local AI or self-hosted AI, and in a couple of years it has gone from an experiment and test stage to something you can build real training on. This post covers what it means, what works well in a training context, what works less well, and what to ask for when you order.
What local actually means
Local does not necessarily mean the model runs on a computer under a desk. There are three levels, and what unites them is that you decide where the model runs and who sees the data.
The first level is your own computer. With a program such as Ollama, for example, you download a model and run it on an ordinary laptop. That is the way to test, and it goes a long way for a course developer who wants to try how a model answers the course’s questions before anything is built.
The second level is your own server, in your own server room or with a hosting provider you already have an agreement with. Here the model runs for all participants, and this is the level most people mean by self-hosted.
The third level is a cloud environment you control yourself within the EU, for example a rented server with computing power at a European provider. The model is still yours and the data does not leave the environment you chose, but someone else looks after the hardware.
Common to all three is that the model is open, meaning whoever trained it released it so that anyone can download and run it. The well-known families are Llama from Meta, Mistral from the French company of the same name and Gemma from Google, but the list grows every month. Open here means the model’s weights are free to use, not always that the training data is known or that the licence allows everything. Read the licence before you build on it.
Why the question matters now
Two years ago an open model could not match the best commercial models, and the choice was easy. It no longer is, at least not for bounded tasks. An open model that is given access to the entire course content and a clear instruction now gives feedback the participant can use. At the same time the hardware has caught up, so a server with one or two graphics cards handles a course with many simultaneous participants.
The second driver is legal. GDPR has always required that you know where personal data is processed and by whom, and transfers to countries outside the EU are something lawyers watch closely. The EU AI Act adds a requirement that you understand the AI systems you use, which we wrote about in the post on AI literacy.
The third is cost. A cloud model charges per call, and a course that gives feedback after every answer makes many calls. At small volumes the cloud is cheapest. At large volumes, or when the same model is used across several courses and systems, your own server can become cheaper over time. Do the sums before you decide, and include operations.
Generative AI that runs with you
Perhaps the most common need in training and AI is what we started with: an AI the course participant can talk to. It can be an AI coach that asks how the course participant intends to handle a situation, a counterpart in a role-play, or a tutor that answers questions about the course content. All of that can run with an open model on your own server.
What decides the quality is not primarily the model but what it is told. An AI coach that is given the course content, the course goals, the company’s procedures and a description of what a good solution contains becomes good even with a mid-sized open model. The same coach without that information becomes generic and a little dull, however large the model.
Feedback on the participant’s answer is the use case where self-hosted models work best of all. The task is bounded: here is the question, here is the participant’s answer, here is what a good answer contains, write three sentences on what was good and what was missing. An open model does that well, and it is exactly the kind of answer where you do not want employees’ wording to leave the building.
In the post One AI explains, another decides we described how the language model should handle the conversation while a smaller decision model decides the next step in the course. That split suits self-hosting particularly well. The decision model is small and fast and can run on almost anything, and the conversation model gets whatever hardware there is.
We like Ollama for development. That is how we try new open models against real course questions before we propose them to a client, and how we know what a given model can and cannot do in Swedish.
Other AI that can also run locally
Language models get most of the attention, but several other tasks in a training production can be self-hosted, and some of them are easier to start with.
Speech to text. Open models such as Whisper transcribe what is said in a training film and do it well in Swedish and English. That gives subtitles, searchable films and material for summaries, without an internal film of the CEO presenting next year’s strategy being uploaded anywhere. Whisper is also what we use ourselves in our production when we transcribe training film, on our own machine.
Translation. An open language model translates course text between Swedish and English well enough for a first draft that a person then reviews. It is often cheaper than and as good as a translation service, and the material stays with you.
Search in your own material. With a model that turns text into numbers, a so-called embedding model, a course or a learning portal can answer questions based on your own procedures and handbooks instead of whatever the model happens to have learned. That is what makes the tutor in the course answer with your procedure and not a generic one.
Image generation. Open image models exist and run locally, but here the gap to the cloud services is still wider, and the need to keep the data at home is usually smaller. In this area we almost exclusively use APIs against the latest models, and therefore not locally.
What works less well, or takes more than you think
Smaller languages are uneven. The largest open models write good Swedish, but the smaller ones that fit on cheap hardware easily slip into English word order or Swedish that sounds translated. Always test the model in your language with your own examples before you decide, and test it on the answers that are hard, not only the easy ones.
Long, open conversations are harder than short, bounded ones. An AI coach that has to hold a twenty-turn conversation with nuance and memory of what was said earlier demands more than one that gives feedback on a single answer. Here the best cloud models are usually still better, and this is where you weigh quality against control.
Response time depends on the hardware. A model that answers quickly on a powerful server feels sluggish on a weak one, and a sluggish AI coach is worse than none. Measure response times with the number of simultaneous participants you actually expect, not with one tester.
Operations is work. Someone has to update the model when a better one arrives, monitor the server, handle it going down on a Sunday evening and make sure access is restricted. A self-hosted model that anyone on the network can reach, with logs nobody looks at, is worse from a data protection point of view than a cloud service with a proper data processing agreement. Local is not the same as secure.
Three questions that decide
When we reason with clients about where the model should run, it usually comes down to three questions.
- 1. What data is given to the AI? General course text, for example about fire safety, can be sent almost anywhere. Employees’ own answers, customer cases and internal procedures can argue for considering running the model locally.
- 2. How many calls will be made, and what do they cost? A few hundred a month is almost always cheaper in the cloud. Feedback after every answer, month after month, can argue for your own hosting if the total running costs are lower than the cost of the model in the cloud.
- 3. How good does the answer have to be? Feedback against a rubric, subtitles and translation drafts are within reach of an open model. A long, free coaching conversation in nuanced language can still justify the best cloud model.
The answers do not have to be the same for the whole course.
You can run the feedback on participants’ answers on your own server while a general summary of a public chapter goes to the cloud. What matters is that someone has decided, and that it can be explained to IT security.
What it means when you order
Ask the supplier where the model runs and what data leaves your organisation. The answer should fit in one sentence. Also ask whether the model can be swapped. A course built against one supplier’s service is hard to move. A course where the model sits behind an interface of your own can start in the cloud and move home when volume or requirements change, or the other way round.
Ask for a test with Swedish speech and with your own examples and sources. Also test response times under heavier load. And ask for a description of who runs it and how successive switches to new models are handled.
Self-hosted models do not make digital learning better in themselves. But they can be what decides whether, and how much, AI can be used in a training programme at all. For some organisations, self-hosted models are a prerequisite for letting course participants have a dialogue with AI in the first place.
Further reading
- Ollama. The program that makes it easy to download and run open language models on your own computer or server.
- Hugging Face. The catalogue where most open models are published, with licences and comparisons.
- Whisper. The open speech-to-text model, which handles Swedish and English.