AI & Local Model Deployment
Articles on deploying and running AI models locally, including open-source model ecosystems (LM Studio, Ollama, etc.), on-device inference, hardware requirements, and the shift from cloud-hosted to self-hosted AI development tools.
AI Model Local Deployment: Inkling-2-Mini 4-Bit Apple Silicon Port
An engineering report on the claimed first public 4-bit Apple Silicon port of Inkling-2-Mini, its MLX-Serve implementation, DSpark drafting path and reported bottleneck.
AI Model Local Deployment: Stock Qwen’s Wider Agentic Reach
A small November 2026 repository-agent test found stock Qwen3.8-27B solved 60% of task-runs, versus 27% for its coding LoRA. The result suggests a specialization trade-off, not a universal verdict.
Harness adapters on llama.cpp: what a 0.32 GB LoRA actually buys you at Q4
A measurement of harness-hub/Qwen3.8-27B-pi-adapted — a rank-32 LoRA for Qwen3.8-27B — served through **llama.cpp** on a single 32 GB consumer/workstation GPU, at **Q4_K_M**, with speculative decoding enabled.
MiniCPM vs Qwen3.8 Flash Next: Four-Turn Growing-Context Measurements
Four-turn growing-prefix measurements compare MiniCPM and Qwen3.8 Flash Next on one M4 Max, including cached follow-up latency. The configured 131K context is not a 131K prompt; neither model is an overall winner.
AI Model Local Deployment: Evidence on 8-Bit MoE Router Gates
A controlled MiniCPM comparison found BF16 gates added 18.803 MiB; KL was 0.0179 versus 0.0199 for 8-bit, with 100% top-1 agreement for both. The conclusion applies only to this pack and comparison.
AI Model Local Deployment: A First Llama 3.1 8B Run
A practical Transformers walkthrough for Llama 3.1 8B Instruct, with careful notes on chat formatting, serving choices, and one reported NPU failure.
MiniCPM-V 4.7 1B: A Text-Trunk Scorecard for AI Model Local Deployment
An audited standard-v2 scorecard for the public MiniCPM-V 4.7 1B 8-bit MLX text-trunk pack, including task results, throughput, memory, and scope limits.
AI Model Local Deployment: From Fine-Tuning to a Reliable Runtime
A practical path from task definition and adapter tuning to quantization, local serving and evaluation.
Choosing a Local AI Model by Context, Speed and Quality
A practical reading of a local model comparison: what context, throughput, latency and quality scores do—and don’t—tell you when deploying an AI model.
AI Model Local Deployment: Grug 27B’s Context Limits on M4 Max
Speed-only measurements of Grug 27B show why its long-context latency makes it impractical for local inference on an M4 Max.
AI Model Local Deployment: What the Qwen2.5-VL vLLM Thread Shows
A careful reading of one Hugging Face discussion about running a quantized Qwen2.5-VL checkpoint with vLLM—and what it does not establish.
AI Model Local Deployment on Jetson: A Practical llama.cpp Path
A hands-on guide to deploying quantized vision-language models locally on Jetson with llama.cpp, GGUF, CUDA, projector files and an OpenAI-compatible server.