Llm Inference

The Rapid-MLX DSH Provider is a native npm plugin for the DeepSeek Harness (dsh) that lets the agent read its model configuration straight from a running Rapid-MLX server’s /v1/models endpoint instead of hand-written settings.yaml. Instead of copying context windows, reasoning efforts, and max_model_len values by hand — and re-copying them every time you switch models — the provider serves those facts directly from Rapid-MLX’s JSON schema, keeps reasoning control truthful, and compacts the context at the real memory limit of your Apple Silicon machine. ...

LLM inference engineering is the discipline of making large language models run fast, cheap, and reliably in production. It centers on two phases — prefill and decode — and on the KV cache, the memory structure that stores attention keys and values. The single biggest win in the field is PagedAttention, which manages the KV cache like an operating system manages virtual memory, cutting memory waste from up to 60–80% down to near zero and boosting serving throughput by up to 24x over naive implementations. ...