How to Create a Chatbot Like ChatGPT by Running LLM Locally in Windows via Ollama and Open WebUI

In this post we will see that how to create a chatbot like ChatGPT by running Large Language Model (LLM) locally in Windows via Ollama and Open WebUI
Creating a chatbot that runs a large language model (LLM) on your local machine can be done in several ways. Here are some methods you might consider:
- Using API and Programming: This approach is often difficult and time-consuming.
- Chatbot UI: This method requires API access from OpenAI. It’s very good if you want to use it remotely, but as of my knowledge, it may not support local model access.
- Ollama and Open WebUI: These can be used with APIs from OpenAI and locally stored, trained LLM models.
In this post, I will discuss how we can use Ollama and Open WebUI.
Install Ollama
Ollama is a framework for building and running language models on a local machine. It can be used for not only chatbots but also AI assistants inside your editor like VSCode. For more information, you can explore Ollama’s blog and Continue Code Assistant.
Ollama supports Nvidia and Apple’s M-series GPUs. Nvidia GPUs with at least 4 GB of memory will work, and for M-series Macs, at least 16 GB of memory is recommended. I suggests using at least a GeForce RTX 3060 Ti GPU with 8 GB GDDR6 memory. While Ollama can run in CPU-only mode, the responses may be slow or poor quality. For instance, one user reported that a smaller model, tinyllama, was able to stream responses at a normal rate but the quality was poor.
- Download Ollama from here and install it.
- Available Ollama models can be found here.
- To run a particular model, open the terminal and run the Ollama command. For example, to run
llama3:8B, use:1ollama run llama3:8B - Once running, you can visit http://localhost:11434/ to confirm if Ollama is running.
For additional documentation, refer to their documentation.
Install Open WebUI
We will be using Docker Desktop.
- Download and install Docker Desktop from here.
- To confirm the Docker installation, run the following command in the terminal:
docker --versionand it should display the Docker version. - If you have an Nvidia GPU and want GPU support with the OpenAI API, use the following command:
1docker run -d -p 3000:8080 --gpus all --add-host=host.docker.internal:host-gateway -e OPENAI_API_KEY=openai_secret_key -v open-webui:/app/backend/data --name open-webui --restart always ghcr.io/open-webui/open-webui:cuda
Remember to replace openai_secret_key with your secret key, which can be found here.
If you want to install without GPU or without OpenAI API, follow the instructions provided here.
Once installed, you can visit http://localhost:3000/, sign up, and start using it.
Installing New Models
To install a new model, you can use ollama run <model name> in the terminal. To find new models visit Ollama model repository. For example: ton install qwen2:0.5b which is is a new series of large language models from the Alibaba group. Now you will have the option to choose the model from the Open WebUI portal. Moreover, you can also download new models in the settings of the Open WebUI portal.
How to start using Open WebUI later on
- Start Ollama from the Start menu.
- Open Docker Desktop and run the following command in the terminal:
docker start open-webui
Troubleshooting
Sometimes, after restarting your PC, you may encounter some port errors with Docker when starting open-webui. In such cases, delete open-webui from Docker and restart your computer. Then run:
|
|
Now, you have a powerful local setup for running a chatbot using large language models. Enjoy!