How I Run Local LLMs for Coding (and Why the Quant Matters)

AI;DR — This post was written with AI assistance from my own local model, then edited by me. The setup, tests, and numbers are all real.

TLDR; I run a coding assistant entirely on my own hardware — a single RTX 3090 Ti with 24GB of VRAM, serving a 27B model to OpenCode over the LAN. No API bills, no data leaving the house. This post is about the two things that actually bit me: the quantization type silently corrupting output, and a single missing sampler flag causing repetition loops. Both had non-obvious fixes.

I have written before about keeping things local — the Rust proxy, the self-hosted everything. Local LLMs are the same instinct. If the model runs on my desk, I control it, and I can leave it running.

How I Got Here

I did not land on the current setup on day one. It took a couple of months of swapping models and tuning knobs, and most of the interesting lessons came from the dead ends.

I started with a Claude-distilled reasoning model — a Qwen3.6-35B-A3B distilled from Claude 4.7 Opus. It was fine, but it refused things a coding assistant should not refuse, so I moved to an uncensored variant, and then to a cleaner Wasserstein-abliterated build, which is still my fast fallback at ~174 tok/s.

The turning point was understanding MoE versus dense. Those 35B-A3B models are Mixture-of-Experts: 35B total parameters but only ~3B active per token, so they are fast. I tested a true dense 27B alongside them. The dense model was measurably smarter on benchmarks — but about 3.5x slower because it activates roughly nine times more parameters per token. For an interactive coding agent, speed is a feature, so the fast model usually wins. The exception is when accuracy really matters, and that is where I landed on a dense 27B coder fine-tune (Jackrong’s Qwopus, and later a Qwen3.8 build) as the daily driver, keeping the fast MoE around for when I want throughput.

Two rabbit holes were worth the time:

• Should I switch off llama.cpp to vLLM? vLLM has better speculative decoding and concurrency. But on my RTX 3090 Ti — an Ampere card — the FP8 weight path that makes long context cheap is not supported (it needs Ada or Hopper). Switching would have crushed my context window from 200K down to ~32-64K. Not worth it. Long context matters more to me than raw latency on this hardware, so I stayed on llama.cpp.
• Speculative decoding (MTP). This is the trick that gets me ~70 tok/s on a dense 27B. The model drafts several tokens ahead and the main pass verifies them in one shot. It was fiddly to get working — most Qwen GGUFs ship their MTP heads for vLLM only, not llama.cpp — so finding a build with a working mtp-*.gguf sibling was half the battle.

I also spent an afternoon overclocking the VRAM (+1500 MHz via LACT, the Wayland-friendly NVIDIA tool) and measured a whole ~2% speedup. That tiny number was actually useful data: it proved this workload is not purely memory-bandwidth bound, so there was no point chasing more.

If there is one meta-lesson from all of that: measure on your own box. Nearly every “best model” or “best setting” claim I read turned out to be conditional on hardware I do not have.

The Setup

The rig is a Pop!_OS box with an RTX 3090 Ti (24GB, Ampere). It runs llama.cpp as a systemd service, exposing an OpenAI-compatible API on port 8080. My editor (OpenCode) points a provider at http://<box>:8080/v1 and never knows the difference.

The model is bytkim’s Qwen3.8-27B-pi, an agentic-coding fine-tune of the dense 27B, quantized to Q4_K_M (about 16.5GB) with its MTP head in a separate ~2GB draft file. With a q4_0 KV cache on both the main model and the draft, it fits a 200K token context window and leaves about 1GB of the card free. Generation runs about 95 tokens/second on a 6.7K-token prompt and drops to about 52 at 107K. On a coding agent that fires a lot of tool calls, that is plenty.

Here is the actual command, pulled from the systemd unit that runs it today (it changed when I swapped to the pi model; see the follow-ups below):

llama-server \
  -m ~/models/qwen3.8-27b-pi/Qwen3.8-27B-pi-Q4_K_M.gguf \
  -md ~/models/qwen3.8-27b-pi/mtp/mtp-Qwen3.8-27B-pi-Q4_0.gguf \
  -c 200000 \
  --parallel 1 \
  -ngl 99 \
  --ubatch-size 1024 \
  --spec-draft-n-ubatch 512 \
  --cache-type-k q4_0 \
  --cache-type-v q4_0 \
  --cache-type-k-draft q4_0 \
  --cache-type-v-draft q4_0 \
  --flash-attn on \
  --no-context-shift \
  --cache-ram 32768 \
  --ctx-checkpoints 8 \
  --jinja \
  --chat-template-file ~/llm-units/qwen38-chat-template.jinja \
  --spec-type draft-mtp \
  --spec-draft-n-max 4 \
  --reasoning-effort low \
  --reasoning-budget 16384 \
  --temperature 0.7 \
  --top-p 0.8 \
  --top-k 20 \
  --min-p 0 \
  --presence-penalty 1.0 \
  --alias Qwen3.8-27B-pi \
  --host 0.0.0.0 \
  --port 8080

Every one of those flags is doing something specific:

• -m / -md: the main model GGUF and the separate MTP draft GGUF. The Jackrong build I ran before had its MTP heads inside the main file. The pi model ships them as their own file, so the draft needs its own slice of VRAM.
• -c 200000 — the context window, in tokens. This is the ceiling for prompt + tool output + reply combined, and it is the number everything else below is tuned around, to make it fit in 24GB of VRAM.
• --parallel 1 — one request slot, not the default of several. I’m the only client (OpenCode, over the LAN), so reserving KV cache for concurrent requests I’ll never make would just waste VRAM I need for context.
• -ngl 99 — the one that matters most. It offloads every layer to the GPU (“number of GPU layers”). Get this wrong and llama.cpp falls back to CPU for the rest — the difference between ~170 tok/s and ~2 tok/s.
• --ubatch-size 1024: the physical batch size llama.cpp uses internally for prompt processing. I ran 2048 on the old model. I dropped it to 1024 to make room for the draft model at 200K, and prefill barely noticed (1,366 tok/s at 50K, 1,165 at 95K).
• --spec-draft-n-ubatch 512 / --cache-type-k-draft q4_0 / --cache-type-v-draft q4_0: the same batch and KV tricks, applied to the draft model. Its KV cache defaults to full f16. Without all three, the server OOMs at 200K before the first token.
• --cache-type-k q4_0 / --cache-type-v q4_0 — quantizes the KV cache itself, not just the model weights, down to 4-bit. This is the flag that actually makes a 200K context window possible at all; at full-precision KV it would run out of VRAM long before 200K.
• --flash-attn on — the fused flash-attention kernel instead of the naive implementation. Faster, lower memory, no downside on Ampere.
• --no-context-shift — turns off llama.cpp’s default behavior of silently sliding old tokens out of the context window once you hit the limit. For a coding agent that depends on everything it has read and decided so far, I want a hard error at 200K, not a session that quietly forgets its first half.
• --cache-ram 32768 / --ctx-checkpoints 8 — these two work together. llama.cpp can checkpoint prompt-cache state to host RAM so a long agentic session doesn’t have to re-process the entire prompt on every turn. --cache-ram caps that at 32GB of RAM; --ctx-checkpoints 8 is how many checkpoint slots it keeps around. I’ve watched this cache absorb a 97K-token prompt down to 294 tokens of actual reprocessing — worth every megabyte of host RAM.
• --jinja / --chat-template-file — use a real Jinja2 chat template instead of a hardcoded one, loaded from a file I control rather than whatever’s baked into the GGUF. Needed for tool-calling to format correctly, and easier to patch when a model’s shipped template has a bug.
• --spec-type draft-mtp / --spec-draft-n-max 4: turns on speculative decoding with the MTP draft model, drafting up to 4 tokens ahead per step. Acceptance on the pi model sits between 61% and 73% depending on prompt size, and that is what gets a dense 27B well past what plain autoregressive decoding would manage.
• --reasoning-effort low / --reasoning-budget 16384: on the old model I ran --reasoning off, because Qwen3.8 defaults to an aggressive “xhigh” thinking mode that burns tokens overthinking simple requests. With the pi model I turned thinking back on, at low effort and capped at 16K thinking tokens per reply, because the output was better with it.
• --temperature 0.7 --top-p 0.8 --top-k 20 --min-p 0: the pi model card’s settings for non-thinking mode. I kept them after turning thinking back on. The card’s thinking profile runs hotter, at temperature 1.0.
• --presence-penalty 1.0: the repetition fix from the section below. I ran DRY (--dry-multiplier 0.8) alongside it for a while and dropped it, because the output was better without it. The test that settled it was asking the model to repeat a long file path five times. With DRY at 0.8 the path came back mangled (dropped /, stray spaces, a capital I in .venv). At 0.2 it was milder but still broken. With DRY off it came back clean 5 out of 5 times, and MTP accepted 68 of 68 drafted tokens. DRY penalizes repeated n-grams, and repeating an exact path is exactly what an agent has to do.
• --alias Qwen3.8-27B-pi: the model name the OpenAI-compatible /v1/models endpoint reports, so OpenCode’s model picker shows something readable instead of the raw GGUF filename.
• --host 0.0.0.0 --port 8080 — bind every interface, not just localhost, so other machines on the LAN (my editor) can reach it.

The Quant Type Quietly Broke Things

When Unsloth shipped their Dynamic 3.0 quants for Qwen3.8, the pitch was great: more accuracy at the same file size. So I downloaded their IQ4_XS build and tested it.

It produced garbage. Not random garbage — subtle garbage. My username came out as spencerkittlesson (extra ‘s’). Paths came back with a Cyrillic е swapped in for the Latin e. MindTouch became MMindTouch. The kind of corruption you would miss on a quick glance and then spend an hour debugging in a generated script.

The isolation test told the story: run it CPU-only (-ngl 0) and it was perfect. Run any part of it on the GPU and it corrupted. That pointed straight at a CUDA kernel bug for that model’s linear-attention layers on Ampere cards — not something I could fix.

Here is the part that took a second round to figure out: it was the quant type, not the build. IQ4_XS is an “i-quant.” When I tested a Q4_K_M “k-quant” of the same model, on the same GPU, it ran completely clean. Same 27B model, same 4-bit target, wildly different result — because the i-quant path hit the broken kernel and the k-quant path did not.

The lesson: on an Ampere GPU, stick to k-quants (Q4_K_M, Q5_K_M, and friends). The i-quants (IQ*) squeeze into less VRAM, which is tempting, but they lean on kernels that are not always solid. I verify every new model with a dumb little test now — ask it to echo an exact path ten times and check the characters are right before I trust it with real work.

One Flag Fixed My Looping Problem

The other issue was repetition. In long agentic sessions the model would occasionally get stuck repeating itself. Short prompts were fine; long ones would eventually self-reinforce a pattern with nothing to break it.

When I checked the live server config, the cause was plain: I had no repetition control turned on at all. No presence penalty, no DRY sampler. With zero guard, short generations rarely hit a repeating loop, but a long session would find one and never climb out.

Unsloth’s model card recommends a presence_penalty of 1.5 for this model in non-thinking mode. Before changing my daily driver, I tested it — five loop-prone prompt types (long enumerations, repetitive code, the classic “continue this list” trap) at 1200 tokens each, across presence_penalty values of 0.6, 1.0, and 1.5.

The results were clear enough to act on:

• 0.6 — 4 of 5 clean. Only the nastiest “keep going without restarting” prompt degraded.
• 1.0 — 5 of 5 clean, balanced, no side effects.
• 1.5 — also 5 of 5, but it started stopping early and wandering off-format on structured tasks.

Speed was identical across all three, so there was no performance cost either way. I landed on 1.0, not the recommended 1.5 — it fixed the failure without the format-drift. One flag:

--presence-penalty 1.0

Unsloth’s number was a good starting point, but the right value for my workload was milder. Test it against the thing you actually do, not the thing the docs assume you do.

Follow-up: unlocking the 3090 Ti, and a faster llama.cpp

A while after publishing, I swapped the daily driver to a newer dense Qwen3.8 build — bytkim’s Qwen3.8-27B-pi, an agentic-coding fine-tune. Unlike the Jackrong model, its MTP head ships as a separate draft GGUF you load with -md, so the draft model wants its own slice of VRAM. At -c 200000 it would not even start — cudaMalloc failed: out of memory before the first token. The fix was three flags that shrink the draft’s footprint without touching the main model’s context:

  --spec-draft-n-ubatch 512 \
  --cache-type-k-draft q4_0 \
  --cache-type-v-draft q4_0 \

The draft’s KV cache was defaulting to full f16 while the main model ran at q4_0. Quantizing the draft cache and halving its batch (plus dropping the main --ubatch-size to 1024) brought a 200K window back, with about 1GB of headroom left on the card. The model reads a ~188K-token prompt without falling over, which is the whole reason I run this thing instead of renting one.

The thing I actually got nerd-sniped by: unlocking the card

GA102 Ampere cards — the 3090, 3090 Ti, 3080 — ship with their FP16/BF16 tensor-core issue rate halved in firmware relative to the pro-grade A-series silicon. Someone reverse-engineered a kernel module that writes the pro speed selectors before the NVIDIA driver binds the card. I installed it. A raw 8192³ GEMM confirmed it took: BF16 with FP32 accumulate went from the stock ~87 TFLOPS to 142, TF32 from ~43 to 75. That part worked exactly as advertised.

Here is the catch, and it is a good lesson in not trusting a headline number. It barely helped llama.cpp. For quantized weights, llama.cpp’s matmul path either uses its own INT8 kernels or dequantizes to FP16 and accumulates in FP16 — and FP16-accumulate math was never throttled on consumer cards, so the unlock has nothing to speed up. There is an env var (GGML_CUDA_CUBLAS_COMPUTE_TYPE=bf16) that switches to the BF16-plus-FP32-accumulate path the unlock does touch, and at a large batch size it won about 11% on prefill. But at the batch size my 200K context actually needs (--ubatch-size 1024), that gain evaporated and small prompts got 30% slower. For an agent that sends a lot of small follow-up turns, that is a net loss, so I left it off.

What did move the needle was boring: I rebuilt llama.cpp from a months-old checkout to current master. That alone bought 14-19% faster prefill across every prompt size, from the upstream kernel work, not from anything I tuned. The unlock is a neat trick and the card is genuinely faster at FP32-accumulate math now — it just is not the math this workload leans on. Measure the thing you run, not the benchmark on the box.

Follow-up, November: two weeks on the pi model

I found out how hard the box was working by accident. OpenCode doesn’t talk to llama.cpp directly. It goes through OmniRoute, a router on my laptop that also fronts my paid models. Its dashboard showed a constant stream of requests to qwen, so I went looking for whatever was hammering it.

Nothing was broken. It was 1,080 calls in about 23 hours, steady at 100 to 200 an hour, each carrying 60K to 150K tokens of context. Almost all of that input was prompt-cache reads. This is what an agent looks like from the server side: one request per tool step, and every request re-sends the whole conversation. It is also why --cache-ram is worth the host RAM.

Since the swap on October 25, 40 of my OpenCode sessions have run on the pi model. That includes the scheduled morning check of my GitHub PRs, a set of subagents that worked through review threads on four PRs, and a lot of everyday research questions.

I benchmarked the router before trusting it with all that. Going through OmniRoute instead of hitting llama.cpp directly added 0.1 to 0.9 seconds to time-to-first-token and under half a second of wall time. Generation speed moved both ways between runs, which is MTP draft acceptance being random, not the router.

Two things in that path did bite me. The first is that OmniRoute estimates tokens at roughly four characters per token instead of counting them. Against the pi model’s real tokenizer, code comes in about 13% heavier than the estimate (272K real tokens estimated at 237K). So a code-heavy prompt can pass the router as “under 200K” and still get rejected by llama.cpp. What actually keeps sessions under the limit is OpenCode’s compaction, so I dropped the model’s output limit in OpenCode from 32K to 16K. That makes it compact around 184K instead of 167K.

The second was a confusing error the first time an image ended up in a session:

[500]: image input is not supported - hint: if this is unexpected, you may need to provide the mmproj

The pi model is text-only on my box, since I don’t load a vision projector (--mmproj). But my OpenCode model entry didn’t say what inputs it accepts, and when that’s missing OpenCode assumes text and images. One line in opencode.json fixed it:

"gpupc/Qwen3.8-27B-pi": {
  "modalities": { "input": ["text"], "output": ["text"] }
}

Speed history: six months on one card

I went back through my OpenCode sessions and pulled every benchmark I ran on the 3090 Ti. These aren’t apples to apples. The prompts, lengths, and quants changed along the way. But the shape of it holds up: MoE models were fast and I kept leaving them, and every dense model was slow until speculative decoding showed up.

Two things in that table stand out. Context size cost nothing until I hit the VRAM wall. The Qwopus Coder ran 60.9 tok/s at 150K, 60.7 at 200K, and 60.6 at 262K, because decode cost scales with the context you actually use, not what you allocate. The other one is how much “real work” numbers trail benchmark numbers. Qwopus v2 did 108 tok/s on a clean prompt and 68 on a 12K-token refactor.

Where I’m at now

Same service file as above, same llama.cpp build (2280f0fd, from October 25), measured November 9 straight against the server with prompt caching off and 256 generated tokens:

Prompt reading is within a few percent of October. Generation is lower, and the reason is draft acceptance, not the box. Today’s prompts were synthetic code, and MTP accepted about 50% of its drafts. The October runs landed between 61% and 73%. MTP speed depends heavily on how predictable the text is, so I treat generation as a range: roughly 50 to 95 tok/s, depending on how deep into the context the session is and what the model is writing.

The card sits at 23.6 of 24.5GB with the model loaded, so about 1GB of headroom.

Would I Do It Again?

Yes. Once it is dialed in, a local coding model is genuinely useful and costs nothing per token. But it is not plug-and-play. The two things that cost me the most time were not “which model is best” — they were the quant type interacting badly with my specific GPU, and a sampler default I never set.

Both are the kind of thing you only find by testing on your own hardware with your own prompts. Which, honestly, is half the fun.