Why You Should Enable Speculative Decoding in Your Local AI Engine

When tuning a local AI setup, most people focus on things like offloading and quantization. But these optimization techniques aren’t lossless, so speed gains always come at the cost of some output quality.

Speculative decoding is a completely lossless optimization hack that hastens token generation by anywhere between two to three times the speed. It pairs your primary AI model with a smaller model that consumes fewer resources for each token generated.

When preparing a response, the smaller model guesses ahead of your primary model to save time on generating tokens. But the response isn’t printed until your primary model validates the guesses, so there’s no loss in actual output quality as with model routing.

Speculative decoding can be enabled in a few steps on LM Studio and Llama.cpp, making it a quick alternative to lossy performance enhancements like quantization.

In this guide, I’ll cover how speculative decoding works, explain how to enable it on Llama.cpp and LM Studio, offer advice on picking compatible draft models, and suggest workloads and use cases that might benefit most from the performance gain.

How speculative decoding works

If you’re not sure about how speculative decoding works, think of it this way: a senior programmer charges higher billable hours, which makes them too expensive for a project where everything needs to be built from scratch. So they hire a team of junior consultants to handle large chunks of routine code, which the senior can then jump in to verify.

Nothing actually makes it to production unless the senior verifies it, so the project still benefits from the senior’s experience and skill. But a team of junior developers can write code much faster, so everything also gets finished quicker.

Speculative decoding works on a similar principle. When your AI model is preparing a response to a query, a smaller model works ahead by guessing what the response is going to be. Each time the smaller model guesses correctly, you save time and resources on token generation. Whenever a guess is incorrect, your primary model simply takes over and continues generating the rest of the response as it normally would.

It’s also worth stressing that the two models act independently of each other during token generation, so the guesses from the smaller model don’t influence the response from the larger one. Each token is only printed into a response if the two models are in agreement, or the system defaults to the primary model for the correct token and the smaller model picks up again afterwards.

Enabling speculative decoding on your system

Not all platforms and libraries that run local AI models support speculative decoding. Ollama for example, despite several community requests, still doesn’t offer it as a standard option.

High-performance engines like vLLM and SGLang do support it, and they offer many options to configure and expand upon it, but they’re better suited for resource-heavy production workloads. So if you want an accessible model runner that includes draft-model pairing as a configuration choice, your best options are Llama.cpp and LM Studio. Depending on which runner you’re using, you can access the feature by following these steps:

LM Studio

1. Make sure you’re on LM Studio version 0.4.14 or higher.
2. LM Studio hides speculative decoding options in the default view. To reveal them, navigate to Settings > Developer and enable Developer mode.
3. Go to the My Models tab, denoted by a folder icon, from the dashboard interface in LM Studio. Download a primary model, followed by a smaller version of the same model that features a lower parameter count. Both the primary model and the draft model need to be from the same family (e.g., Llama, Qwen, DeepSeek R1, etc.).
4. Not all AI models offer smaller versions suitable for speculative decoding. If you’re not sure which LLM to pick, just click on the cog icon next to the model name and look at the side panel that opens up. Navigate to the Inference tab in the side panel and click on the dropdown under Draft Model in the Speculative Decoding section. This will automatically reveal compatible draft models that will work with your chosen LLM.
5. Once you pick a draft model from the dropdown, speculative decoding is toggled on automatically. You can just start chatting and take advantage of the faster token generation speeds.

Screenshot by Dan Ackerman

Llama.cpp

1. Download both the primary and draft models in GGUF format. Remember that like in LM Studio, their tokenizer and vocabulary have to match or the pair will just fail to load.

2. From your desktop Terminal, execute the following command. Remember to replace ‘main-model’ and ‘draft-model’ with the file names of your downloaded models.

   Llama-server -m main-model.gguf -md draft-model.gguf

3. Once both models are loaded, you can simply power up your Llama.cpp server and start prompting as you usually do. Speculative decoding will now be enabled.
4. If you’re curious about the exact performance gains, Llama.cpp prints the draft model’s acceptance rate in the server logs, which tells you how many tokens were guessed accurately per response.

How to pick a good draft model

Always remember to pick a draft model from the same family as your main model. This is a hard requirement rather than a suggestion, since speculative decoding will not work unless both models share the same vocabulary and tokenizer.

If you’re looking for options, here are some model pairings recommended in the official documentation from LM Studio:

You may have noticed that there’s a pattern to the difference in parameter count between the main and draft models. Theoretically the bigger the size gap in parameters, the more noticeable the improvement in token generation speed with speculative decoding.

But if the draft model is too weak to guess the main model’s tokens correctly, you’ll have fewer correct guesses and therefore fewer opportunities to generate tokens faster. You’ll still be burning resources on the draft model’s compute without getting as much in return.

As a rule of thumb, a 7B model pairs best with a 1B draft model or smaller. Similarly, a 14B model can pair with a 3B model and a 32B main model can have a draft model of up to 7B parameters before the overhead starts eating into the efficiency gains. Ultimately, you’ll want to mix and match different primary and draft pairs to see how much you’re gaining in token generation speed and compare it to the number of correct guesses per response. You’ll want to see high values in both for the best results.

When is speculative decoding worth turning on?

Of course, the technology behind speculative decoding is such that only certain tasks benefit most from the added efficiency despite the overhead of running two models concurrently.

Generative tasks that rely on structured data formats or have predictable output patterns, like coding or data science, tend to see the most gains. That’s because these tasks rely on structured response formats (syntax, formulas, JSON, etc.) which are easier for the draft model to guess ahead on.

Tasks that require a high degree of creativity perform much worse, since responses there are more difficult to predict. In fact, VRAM-constrained setups may perform worse on such tasks with speculative decoding turned on, since the compute overhead from the draft model consumes additional resources.

With that in mind, let’s end with a simple table to visualize the efficiency gains across different use cases.