Skip to content
Velloreth
Artificial Intelligence 5 min read

Run an LLM Locally on Your Own Computer (Windows, Mac, Linux)

Run a language model offline on your own PC with Ollama. Install steps, hardware basics, your first chat, and how to call it from code.

By

How to Run an LLM Locally on Your Own Computer (Windows, Mac, Linux)

Running a large language model (LLM) on your own computer is no longer a research project. With tools like Ollama, you can download a model, run it on your laptop, and chat with it without sending data to a remote server.

This guide explains what running a model locally means, what hardware you actually need, how to pick a model, and what trade-offs to expect. It is written for developers and technically curious readers who want to experiment locally without becoming infrastructure engineers.


Laptop running a local AI model
Running an LLM locally means inference happens on your hardware, not in the cloud.

What “Local” Actually Means

When you run a model locally, the inference happens on your computer’s CPU or GPU. The model weights live on your disk, and the computation happens on your hardware. Nothing is sent to a remote server unless you explicitly configure it to do so.

This matters for privacy, but it also means you are responsible for hardware costs. There is no free tier when you are the one providing the compute.


Hardware Reality Check

The most important constraint for local inference is RAM, not CPU speed.

A model’s weights must live in memory to generate text. A 7-billion parameter model at 4-bit quantization needs roughly 4–5 GB of RAM just for the weights, plus overhead for the runtime and context window. On an 8 GB machine, that leaves little room for anything else.

Here is a rough guide for what fits:

Model SizeApprox. Size (4-bit)Minimum RAMComfortable RAM
3B parameters~2 GB4 GB8 GB
7B parameters~4–5 GB8 GB16 GB
13B parameters~7–8 GB16 GB24 GB
70B parameters~35–40 GB48 GB64 GB+

If your laptop has 8 GB of RAM total, a 7B model is likely the practical ceiling. Anything larger will swap to disk and become painfully slow.

GPU VRAM works the same way: a model must fit in GPU memory to run at GPU speed. If it does not fit, the system falls back to system RAM, which is much slower.


Quantization: Smaller Models, Slightly Less Accuracy

Quantization reduces the precision of a model’s weights. A full-precision model uses 16-bit or 32-bit numbers for each weight. A quantized model might use 4-bit integers.

The result is a much smaller model that runs on consumer hardware, at the cost of some accuracy. For many practical tasks, 4-bit quantized models work well enough. For tasks requiring precise reasoning, the difference can be noticeable.

Ollama and similar tools distribute quantized models by default, which is why you can run a 7B model on a laptop rather than needing a data center GPU.


Getting Started with Ollama

Ollama is one of the most popular tools for running models locally. It handles model downloads, quantization, and serving through a simple command-line interface.

The typical workflow is:

  1. Download and install Ollama from ollama.com for your operating system.
  2. Run ollama pull <model> to download a model.
  3. Run ollama run <model> to start chatting.

Common small models include:

  • llama3.2:3b — small, fast, good for simple tasks
  • llama3.2:7b — balanced, good general-purpose model
  • llama3.1:8b — slightly larger, more capable
  • gemma2:2b — very small, good for limited hardware
  • gemma2:9b — mid-sized, capable

You can list available models with ollama list and remove ones you no longer need with ollama rm.


What to Expect: Performance and Limits

Local inference is slower than hosted APIs, especially on CPU. A small model might generate 1–3 tokens per second on a CPU, while a GPU can do much better. Do not expect ChatGPT-speed responses from a laptop.

Local inference is also not free after the hardware cost: you paid for the hardware upfront and you pay for the electricity to run it. The trade-off is privacy and no per-token fees, not capability.

Finally, local models are not magically smarter than hosted models. A small local model will generally be less capable than a large hosted model. The trade-off is privacy, offline use, and no API fees — not capability.


Questions and Answers

Q: Do I need a GPU to run a model locally?

A: Not strictly. CPUs can run small models, but they are slow. A GPU with sufficient VRAM makes local inference practical. For a 7B model, 8 GB of VRAM is a comfortable starting point.

Q: Can I run any model on 8 GB of RAM?

A: No. A 7B parameter model at 4-bit quantization needs roughly 4-5 GB just for weights, plus overhead. Larger models will not fit comfortably on an 8 GB machine.

Q: Is running locally free?

A: No. You pay upfront for hardware and ongoing electricity. There are no per-token API fees, but hardware costs are real.

Q: Is local inference completely private?

A: Yes, if the model runs entirely on your hardware and never phones home. However, you should verify that the specific tool you use does not phone home.

Q: Which model should I start with?

A: Start with llama3.2:3b if you have 8 GB of RAM, or llama3.2:7b if you have 16 GB. Both are capable, general-purpose models that run well on consumer hardware.


Conclusion

Running a model locally means inference happens on your hardware. It is private, works offline, and has no per-token fees — but it requires hardware, and small models are less capable than large hosted ones.

Start small with a 3B or 7B model, see what your hardware can handle, and expand from there. Local inference is not about replacing cloud models for everything; it is about having a private, always-available option for tasks where privacy and offline access matter.


This article is conceptual and does not include installation commands or specific version numbers that may change. Check ollama.com for current installation instructions and supported models before installing.

Frequently asked questions

Do I need a GPU to run a model locally?
Not strictly. CPUs can run small models, but they are slow. A GPU with sufficient VRAM makes local inference practical.
Can I run any model on 8 GB of RAM?
No. A 7B parameter model at 4-bit quantization needs roughly 4-5 GB just for weights, plus overhead. Larger models will not fit comfortably on an 8 GB machine.
Is running locally free?
No. You pay upfront for hardware and ongoing electricity. There are no per-token API fees, but hardware costs are real.
Is local inference completely private?
Yes, if the model runs entirely on your hardware and never phones home. However, you should verify that the specific tool you use does not phone home.
Which model should I start with?
Start with llama3.2:3b if you have 8 GB of RAM, or llama3.2:7b if you have 16 GB. Both are capable, general-purpose models that run well on consumer hardware.