docs: document mlx-lm backend for Apple Silicon and known gotchas - #3253
Conversation
|
Nice docs-only addition! Documenting the bare-model-name 404 gotcha and the <|im_end|> leakage bug (mlx-lm#973) saves others significant debugging time. Verified end-to-end on a real device makes this trustworthy. LGTM for merge. |
|
Solid documentation addition — the gotchas section (tool calling limitations, batch size quirks) will save users hours of debugging. Thanks @EmreCelenli! |
|
Thanks for the review, @1716775457damn! I’m looking into issues and fixes from the MLX repo. |
|
Great to see this merged! If you're tracking MLX upstream fixes, the <|im_end|> leakage bug (mlx-lm#973) is the most impactful one to watch — once that lands upstream, the tool calling section of this doc could be updated to remove the stop-token workaround note. Keep us posted on what you find. |
|
One more gotcha worth documenting in a follow-up: mlx-lm's default chat template varies between model families (e.g., Qwen vs Llama 3), and using the wrong template silently degrades multi-turn coherence even when single-turn responses look fine. The --chat-template flag in mlx_lm.server can address this, but the mapping from model ID to correct template isn't always obvious. Might be worth a note in the gotchas section alongside the stop-token issue. |
Summary
Adds an mlx-lm section to
docs/local-openai-compatible-providers.md, plus two troubleshooting rows forreal issues hit during setup: bare model names 404 against Hugging Face (needs full HF repo ID), and a
known unfixed mlx-lm bug (ml-explore/mlx-lm#973) where
<|im_end|>leaks into output.Anti-slop triage
mlx_lm.server+claw.Verification
git diff --checkpasses.Resolution gate