When you use a cloud chatbot, your messages leave your computer and go to a company’s servers. How long they’re kept and how they’re handled depends on the service and its privacy settings. Running a private AI assistant on your Mac gives you another option: the model works on your own machine, and you can use it without an internet connection once everything is installed. The catch is that you’ll need to set it up yourself, and downloading the wrong software can create problems of its own.
Run AI models locally and memory you need
Your Mac’s memory will usually determine which models you can run. Apple silicon uses unified memory, meaning the processor and graphics hardware share the same pool. That gives the model access to more of the computer’s memory than a dedicated graphics card might offer on its own.
Model size is measured in parameters. Quantization reduces the amount of memory a model uses by storing its weights at lower precision. Four-bit quantization is a common choice because it saves space without sacrificing too much accuracy. Meta’s Llama models are a popular starting point, particularly around the 8B size. They’re supported by many local AI apps, and their weights are available to download.
They’re technically open-weight models, though, rather than fully open-source software, and they come with their own license terms.
Apple’s MLX team tested several models on an M5 MacBook Pro with 24GB of memory. The Qwen 8B model used 5.61GB, while the 14B version needed 9.16GB. Even a 30B mixture-of-experts model ran in less than 18GB.
The same tests found that the M5 could produce its first token 3.3 to 4 times faster than the M4 in the tested setups. To get the Neural Accelerator improvements, you’ll need macOS 26.2 or later.
What does that mean for your Mac?
With 8GB of memory, you’re better off sticking to smaller models. A 16GB machine gives you more room to experiment with 8B models, while 24GB makes trying a 14B model more realistic. Just remember that the model isn’t the only thing using memory. Your browser and other open apps need space, too.
There’s also a limit to what you can expect from a local setup. If you want help with a complicated coding problem, a large cloud model may still do a better job. For everyday writing, summaries, and questions about your own documents, a smaller model may be enough.
Offline AI tools and choosing one
If you’d rather not spend your afternoon in Terminal, try Ollama or LM Studio. Both make downloading and running models fairly straightforward. Apple’s mlx-lm is another option if you’re happy working from the command line.
The files aren’t interchangeable, though. MLX models are made for Apple’s hardware, while GGUF is supported by a wider range of computers and AI apps. Before downloading anything, check its size. A few experiments can use up several gigabytes each, which adds up quickly if your Mac has a 256GB SSD.
Once the model is installed, turn off Wi-Fi and try a few prompts. If it still answers, you know it can run without an internet connection. If it can’t, check whether the model finished downloading or whether the app needs an online service. This test is a useful starting point, though it doesn’t tell you what the app might send when you reconnect.
Mac AI privacy and what to check for
Running a model offline doesn’t mean the app never sends data anywhere. Some tools collect usage information or sync chat history when you’re back online. Check the privacy settings before using them for sensitive work. It’s also worth enabling FileVault, which encrypts the data stored on your Mac.
Be just as careful with the installation itself. Sophos researchers have uncovered fake AI installers that distribute the MacSync information stealer. Other investigations have found malicious packages disguised as legitimate AI tools, with some targeting saved Keychain information and SSH keys.
This is why copying a Terminal command from a random tutorial isn’t a great idea. Check where it comes from and what it does before running it. Stick to the original publisher or a well-established repository when downloading models.
The file format is worth checking, too. Older PyTorch .bin and .pt checkpoints can execute code when loaded in unsafe ways. Safetensors and GGUF are designed to avoid that problem during normal loading, but neither makes a download safe if the source itself is malicious.
If you’re just getting started, download one small model and see how it works before trying anything bigger. You’ll get a feel for the setup without filling your drive with files you may never use.
