What happens to AI's favorite speed trick when you shrink the model?
Speculative decoding speeds up text generation by letting a small draft model predict the next few tokens, which the main model then checks in one pass. Quantization shrinks a model by storing its numbers with less precision, like writing pi as 3.1 instead of 3.14159265. People running AI on their own hardware will usually want both. I could not find a published answer on how well they work together, so I measured it: 760 logged runs on a MacBook Pro (M4 Max).
Two projects motivated this experiment.
Earlier this year, Salvatore Sanfilippo (antirez) released DS4, which runs DeepSeek's 284-billion-parameter model on a regular computer with 128 GB of memory. He explains the approach in his blog post. The trick is selective rounding: compress the redundant parts of the model aggressively, protect the parts that make important routing decisions.
DeepSeek's DSpark paper (code in the DeepSpec repo) pushed speculative decoding further. One key idea: if the draft model only submits tokens it is confident about, the main model accepts more of them, and generation gets faster.
There is a tension between the two. Speculative decoding works because the draft model predicts the main model well. Quantization changes the main model. The draft model is now predicting a model that no longer quite exists. This repo measures how much that costs, and what gets it back.
Main model: Qwen3-4B-Instruct at four quantization levels (Q8_0, Q4_K_M, Q3_K_M, Q2_K). Draft model: Qwen3-0.6B. Three task types: code, math word problems, and open-ended chat, 10 prompts each. Everything runs through llama.cpp on Metal, greedy decoding with a fixed seed, 256 tokens per run, a 30-second cooldown between runs so thermals do not skew the timing, and every run logged to a CSV with raw output saved.
1. Moderate compression is free. Only 2-bit breaks, and it breaks math first. Acceptance stayed flat from 8-bit down to 3-bit. At 2-bit, math took the biggest hit (67% → 58%), not chat as I expected. Quantization can only break agreement that already exists, and math was where the two models agreed most. Chat was already near the floor a small untrained draft model can reach (~21%), so shifting the main model barely moved it.
2. Compressing the draft model to match does not help. Against the 2-bit main model, a 2-bit draft model lost another 7–16 percentage points compared to the full-precision draft model. Quantization hurts the small 0.6B model much more than the 4B one. The matching effect does show up in the data — the compressed draft model loses a few points less against the compressed main model than against the precise one, in every task — but it is far too small to matter in practice. A 4-bit draft model, on the other hand, matched the full-precision one within noise.
3. The confidence threshold matters more than any quantization choice.
Raising llama.cpp's --draft-p-min from 0 to 0.9 took chat generation from
30 to 81 tokens per second on the 2-bit main model. At high thresholds, the
2-bit and 8-bit main models accepted drafts at almost the same rate (~92%),
and the best setting (0.7–0.9) did not shift with quantization level.
A warning for anyone benchmarking this themselves: my first full run of
the experiment produced an exciting, completely wrong result — acceptance
rose as the main model got more damaged. The heavily compressed model had
degenerated into repeating itself, and repetition is easy to draft. The
cause was feeding prompts raw instead of in the chat format the models were
trained on. Fix the formatting and the effect disappears. Both datasets are
in results/runs.csv, labeled. Always read the outputs.
prompts/ 10 prompts per domain (code, math, chat), '###'-separated
scripts/
run_matrix.sh drives an experiment matrix (resumable; configured by env vars)
guardrail.sh verifies spec decoding does not change greedy outputs
parse_log.py extracts metrics from llama-speculative output -> CSV row
analysis/
plots.py acceptance / speed vs quant level, tau table
plots_phase2.py draft-model matching experiment
plots_phase3.py confidence threshold sweep
plots/ generated figures
results/
runs.csv one row per run (all 760, including the failed-methodology runs, labeled)
raw/ stdout/stderr of every run
model_shas.txt SHA256 of every model file used
llamacpp_local.patch one-line llama.cpp fix needed for the threshold sweep
models/ and llama.cpp/ are not committed; the steps below recreate them.
Any Apple Silicon Mac with 16 GB or more should work (absolute speeds will differ, acceptance rates should not). Linux with CUDA works too — swap the cmake backend flag.
# 1. Clone and build llama.cpp. The commit is pinned because flag names and
# behavior change between versions, and one local patch is required:
# at a high --draft-p-min the entire draft can be pruned, which trips an
# assert in upstream common_speculative_accept.
git clone https://github.com/ggml-org/llama.cpp
git -C llama.cpp checkout 2d973636e292ee6f75fadcf08d29cb33511f509f
git -C llama.cpp apply ../results/llamacpp_local.patch
cmake -B llama.cpp/build llama.cpp -DGGML_METAL=ON
cmake --build llama.cpp/build -j
# 2. Python env for downloads and analysis
python3 -m venv analysis/.venv
analysis/.venv/bin/pip install pandas matplotlib "huggingface_hub[cli]"
# 3. Models (~10 GB total; SHA256s in results/model_shas.txt)
HF=analysis/.venv/bin/hf
$HF download bartowski/Qwen_Qwen3-4B-GGUF Qwen_Qwen3-4B-Q8_0.gguf --local-dir models/
$HF download bartowski/Qwen_Qwen3-4B-GGUF Qwen_Qwen3-4B-Q4_K_M.gguf --local-dir models/
$HF download bartowski/Qwen_Qwen3-4B-GGUF Qwen_Qwen3-4B-Q3_K_M.gguf --local-dir models/
$HF download bartowski/Qwen_Qwen3-4B-GGUF Qwen_Qwen3-4B-Q2_K.gguf --local-dir models/
$HF download bartowski/Qwen_Qwen3-0.6B-GGUF Qwen_Qwen3-0.6B-Q8_0.gguf --local-dir models/
$HF download bartowski/Qwen_Qwen3-0.6B-GGUF Qwen_Qwen3-0.6B-Q4_K_M.gguf --local-dir models/
$HF download bartowski/Qwen_Qwen3-0.6B-GGUF Qwen_Qwen3-0.6B-Q2_K.gguf --local-dir models/
# 4. Phase 1 — the decay matrix (4 target quants x 3 domains x 10 prompts, ~2 h)
bash scripts/run_matrix.sh
# 5. Phase 2 — the matching experiment (~2 h)
for DQ in Q4_K_M Q2_K; do
DRAFT_QUANT=$DQ PHASE=2 QUANTS="Q2_K" bash scripts/run_matrix.sh
DRAFT_QUANT=$DQ PHASE=2 QUANTS="Q8_0" bash scripts/run_matrix.sh
done
# 6. Phase 3 — the threshold sweep (~3.5 h). SPEC_IMPL=simple matters:
# the legacy llama-speculative binary silently ignores --draft-p-min.
# I found out via 200 runs with identical output.
for PM in 0 0.3 0.5 0.7 0.9; do
SPEC_IMPL=simple PHASE=3 QUANTS="Q8_0 Q2_K" DOMAINS="chat code" \
DRAFT_MAX=12 DRAFT_P_MIN=$PM bash scripts/run_matrix.sh
done
# 7. Sanity check + figures
bash scripts/guardrail.sh
analysis/.venv/bin/python analysis/plots.py chatml
analysis/.venv/bin/python analysis/plots_phase2.py
analysis/.venv/bin/python analysis/plots_phase3.pyRuns are greedy with a fixed seed, so results are deterministic per prompt
and config. run_matrix.sh skips cells already recorded in the CSV, so an
interrupted matrix can simply be re-run.
This experiment has a narrow scope. I tested one model family, one machine, small dense models instead of the large mixture-of-experts models that DS4 targets, and an off-the-shelf draft model instead of a draft model trained specifically for this setup, as in DSpark.
Because of that, the absolute numbers should not be treated as universal. The more important result is the pattern in the curves.
I also checked whether speculative decoding matched plain decoding at temperature 0. In 16 of 20 checks, the outputs matched byte for byte. In the other 4, the outputs diverged because of floating-point near-ties on Metal, but the resulting text was still coherent.