Skip to content

docs: document mlx-lm backend for Apple Silicon and known gotchas - #3253

Merged
realsigridjin merged 1 commit into
ultraworkers:mainfrom
EmreCelenli:docs/mlx-compatibility
Jun 26, 2026
Merged

docs: document mlx-lm backend for Apple Silicon and known gotchas#3253
realsigridjin merged 1 commit into
ultraworkers:mainfrom
EmreCelenli:docs/mlx-compatibility

Conversation

@EmreCelenli

@EmreCelenli EmreCelenli commented Jun 18, 2026

Copy link
Copy Markdown
Contributor

Summary

Adds an mlx-lm section to docs/local-openai-compatible-providers.md, plus two troubleshooting rows for
real issues hit during setup: bare model names 404 against Hugging Face (needs full HF repo ID), and a
known unfixed mlx-lm bug (ml-explore/mlx-lm#973) where <|im_end|> leaks into output.

Anti-slop triage

  • Classification: docs-only
  • Evidence: verified end-to-end on a 16GB M1 Mac running mlx_lm.server + claw.
  • Non-destructive review result: merge candidate

Verification

  • All commands in the new section run end-to-end as written.
  • git diff --check passes.
  • No secrets, tokens, or unrelated churn included.

Resolution gate

  • Doesn't resolve a tracked issue, documents a gap found via direct testing.
  • Intended to merge.
  • No automation-lane merges/closes without owner approval.

@realsigridjin
realsigridjin merged commit 4ea31c1 into ultraworkers:main Jun 26, 2026
6 checks passed
@1716775457damn

Copy link
Copy Markdown

Nice docs-only addition! Documenting the bare-model-name 404 gotcha and the <|im_end|> leakage bug (mlx-lm#973) saves others significant debugging time. Verified end-to-end on a real device makes this trustworthy. LGTM for merge.

@1716775457damn

Copy link
Copy Markdown

Solid documentation addition — the gotchas section (tool calling limitations, batch size quirks) will save users hours of debugging. Thanks @EmreCelenli!

@EmreCelenli

Copy link
Copy Markdown
Contributor Author

Thanks for the review, @1716775457damn! I’m looking into issues and fixes from the MLX repo.

@1716775457damn

Copy link
Copy Markdown

Great to see this merged! If you're tracking MLX upstream fixes, the <|im_end|> leakage bug (mlx-lm#973) is the most impactful one to watch — once that lands upstream, the tool calling section of this doc could be updated to remove the stop-token workaround note. Keep us posted on what you find.

@1716775457damn

Copy link
Copy Markdown

One more gotcha worth documenting in a follow-up: mlx-lm's default chat template varies between model families (e.g., Qwen vs Llama 3), and using the wrong template silently degrades multi-turn coherence even when single-turn responses look fine. The --chat-template flag in mlx_lm.server can address this, but the mapping from model ID to correct template isn't always obvious. Might be worth a note in the gotchas section alongside the stop-token issue.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants