Skip to content

Bloglocal-llm

Evaluating local models on one 12 GB GPU

Five local models tested on one RTX 4070 (12 GB) with LM Studio and llama.cpp: Qwen3-VL-8B, Gemma 4 12B QAT, Gemma 4 26B A4B, Qwen3.8-27B and Qwen3.6-35B-A3B.

3hship9 min read

Share

Contents

A new local model shows up every week, and the forum numbers rarely match what it does on your card with your work. This is a repeatable way to find out: one real job, a small hand-labelled set, a few speed readings and some quick capability checks.

The worked example is one round on a single RTX 4070 with 12 GB of VRAM, run through LM Studio on llama.cpp. It covered five models:

Model Type Ran on
Qwen3-VL-8B (Q4_K_M) Dense GPU only
Gemma 4 12B QAT Dense GPU only
Gemma 4 26B A4B Mixture of experts GPU and RAM
Qwen3.8-27B Dense GPU and RAM
Qwen3.6-35B-A3B Mixture of experts GPU and RAM

Each was tested for:

  • Accuracy against 40 hand-labelled documents, scored field by field, plus valid JSON
  • Speed: time to the first token and tokens per second
  • Load placement: how far tuning the expert offload moves a mixture-of-experts model (here, 34.1 to 56.1 tok/s on an idle card)
  • Prompt size: first-token time and recall on prompts of a few thousand tokens
  • Capabilities: text extraction, vision, thinking on and off, and tool calling
  • Memory: peak VRAM against the card’s baseline, and whether a bigger model fits at all

The method at a glance

  1. Pick one real job and hand-label a small set to score against.
  2. Load each model the way you would actually run it, and write the settings down.
  3. Score accuracy and speed together.
  4. Before calling a mixture-of-experts model slow, move its experts.
  5. Test the prompt sizes you will really send.
  6. Run a few small capability checks.
  7. Read memory numbers against a baseline, and do the fit arithmetic before a big download.

1. One job, a small labelled set

Start from work you already do, not a public benchmark. The job here is structured extraction from short documents: for each one the model returns a three-way label with a supporting quote, a yes/no flag, a number and a category. The score is the average agreement with hand labels on those four fields, plus whether the output was valid JSON.

Forty hand-labelled documents was enough to separate the models. Keep the counts in mind when you read the scores: on 40 items, 0.975 is 39 right and 0.90 is 36, so a gap of 0.025 is one document.

Label a second, separate set before you start tuning. The prompt in this round was tuned on the same 40 documents that score it, so the numbers are in-sample, and that is the first thing to change in the next round.

2. Load each model the way you would run it

The round’s setup:

  • GPU: an RTX 4070 with 12 GB of VRAM
  • Runner: LM Studio 0.4.25 on llama.cpp (CUDA)
  • Context window: 8K tokens, 4K for Qwen3.8-27B
  • Layers on the GPU: all of them for the two smaller models, 30 to 60% for the three bigger ones
  • Test set: 40 hand-labelled documents (8 for Qwen3.8-27B)
  1. Gemma 4 12B QAT57.1 s to load
    Share on the GPU
    All of it
    Context window
    8,192 tokens
  2. Qwen3-VL-8B91.6 s to load
    Share on the GPU
    All of it
    Context window
    8,192 tokens
  3. Gemma 4 26B A4B481.2 s to load
    Share on the GPU
    40%
    Context window
    8,192 tokens
  4. Qwen3.8-27B200.3 s to load
    Share on the GPU
    60%
    Context window
    4,096 tokens
  5. Qwen3.6-35B-A3B98.8 s to load
    Share on the GPU
    30%
    Context window
    8,192 tokens

    Qwen3-VL-8B was still loaded in the GPU.

Three things worth knowing before you start:

  • Loading is slow. It took from about a minute to eight minutes per model. Budget for it.
  • Unload everything else first. During the Qwen3.6-35B-A3B run another model was still on the card, and its numbers paid for it.
  • A failed load is not a failed model. In an earlier round Qwen3.8-27B wouldn’t load on an older llama.cpp runtime. On a newer one it loaded and ran. Update the runtime and retry before you drop a model.

3. Score accuracy and speed together

Every model gave zero wrong “yes” answers. Switch the metric to see how the order changes; the dashed tag marks the model that ran on 8 documents instead of 40.

Extraction suite, one row per model. Pick a metric; bars start at zero.

Mean of four agreements with the hand labels. The Qwen3.8-27B value is from 8 documents and cannot be compared with the other four.

Show every metric as a table
Extraction suite and the 2 to 4k-token prompt. Qwen3.8-27B ran on 8 documents; the others on 40. Every model scored zero wrong "yes" answers.
ModelnMean scoreMain labelCategoryValid JSONTokens per second (tok/s)First token (s)First token, long prompt (s)
Qwen3-VL-8B400.9221.0000.8161.0075.50.4918.96
Gemma 4 12B QAT400.9350.9000.8951.0042.31.573.47
Gemma 4 26B A4B400.9470.9750.8681.0024.92.8119.18
Qwen3.6-35B-A3B400.9090.9000.8950.955.236.75156.81
Qwen3.8-27B80.9691.0000.8751.008.424.7329.41

Qwen3-VL-8B (Q4_K_M) won. It matched the main label on all 40, always returned valid JSON, and was the fastest accurate model at half a second to the first token and 75.5 tokens per second. It did not win on the average: its 0.922 is second lowest of the four models that ran all 40 (0.909 to 0.947), and its category agreement, 0.816, was the lowest of the five.

That is the point of scoring fields separately. Decide which field the job can’t get wrong, then pick on that and on speed. A one-document lead on the average shouldn’t decide anything.

Qwen3.8-27B had the highest average, 0.969, but on only 8 documents and at almost 25 seconds to the first token and 8.4 tokens per second. A small sample like that is a reason to rerun it on the full set, not a result.

4. Before calling a model slow, move its experts

Two mixture-of-experts models, Qwen3.6-35B-A3B and Gemma 4 26B A4B, came in far below what forum posts reported. One poster saw 40 to 50 tok/s for Gemma 4 26B A4B on a laptop with 8 GB of VRAM, up from about 20 before tuning. Another saw 52 to 59 tok/s for Qwen3.6-35B-A3B on a 12 GB laptop GPU with short prompts, and a third thread reported 14 to 35 tok/s on 12 GB cards. On this card they first ran at 24.9 and 5.2 tok/s on the extraction suite, with a fixed share of layers on the GPU.

The difference was where the weights lived. A mixture-of-experts model only uses a few experts per token, so the expert weights can sit in system RAM while every other layer stays on the GPU. In llama.cpp that is --n-cpu-moe N, which keeps the experts of the first N layers on the CPU. A plain GPU layer ratio, which is what LM Studio’s default load uses, moves whole layers instead.

Terminal window
# a commenter's 12 GB RTX 3060 setup, excerpt (llama.cpp)
llama-server -m model.gguf --n-gpu-layers 99 --n-cpu-moe 31 --ctx-size 80128 --flash-attn on --no-mmap
# the 12 GB laptop thread's command shape (llama.cpp)
llama-cli -c 4096 -fa on -ctk q8_0 -ctv q8_0 --fit on -fitt 768 --temp 0 --single-turn
# the first load in this round (LM Studio): a whole-layer ratio, no expert offload
lms load qwen3.6-35b-a3b -y --gpu 0.3 -c 8192

Here is the same Qwen3.6-35B-A3B on the same card at the same 8K context, with only the placement changing:

tok/s Placement
1.1 LM Studio’s default: 16 of 41 layers on the GPU, everything else on the CPU, read while the card was shared
4.5 Every layer on the GPU, all experts in RAM, while another job shared the card
34.1 The same load on an idle card, using 7.6 GiB of VRAM
56.1 17 expert layers on the GPU and 23 in RAM, on an idle card, using 10.7 GiB of VRAM

On an idle card 34.1 to 56.1 is about 1.6 times faster, in line with the forum’s 52 to 59. The 1.1 and 4.5 readings came from a shared card as well as from the default placement. An earlier reading of the same setting was 52.8.

Gemma 4 26B A4B is the counterexample. Its first default reading, 19 tok/s, was taken while another job held the card. Alone, LM Studio’s default placement already gave 60.8 tok/s. The best setting found (about 30% of the expert layers on the CPU, flash attention and an 8-bit key/value cache) gave 63.3, about 4% more. The other three models gained little or nothing. Qwen3.8-27B doesn’t fit the card: its best was 8.2 tok/s with 61% of the layers on the GPU, against 6.4 by default, and more layers on the GPU collapsed prompt speed, so it suits batch work, not interactive use. Gemma 4 12B QAT and Qwen3-VL-8B fit whole, and LM Studio’s default placement was already their best.

Measure on an idle card. Contention alone made Gemma 4 26B A4B look three times slower.

To find the setting on your own card:

  1. Start with every expert in RAM (--n-cpu-moe set to the layer count) on an idle GPU.
  2. Lower the number a few layers at a time, so more experts move onto the GPU.
  3. Stop when the speed stops rising or the readings start to swing. Here 23 held at 50 to 56 tok/s, while 22 read 48 once and 29 twice, and lower than that got slower or failed to load.
  4. Take 3 to 5 readings per setting, and look at the spread as well as the best one.

In LM Studio the same control is the per-model load setting for expert layers on the CPU. It saves with the model and becomes --n-cpu-moe when the model loads. One more catch: a thinking model spends tokens before even a short answer, so also time a finished answer, not just tokens per second.

5. Test the prompt sizes you will send

The best model changed with prompt size. For prompts under 2K tokens Qwen3-VL-8B won. For prompts of a few thousand tokens every model found all three hidden facts in the long-context probe, but the wait for the first token was very different:

Model First token
Gemma 4 12B QAT 3.5 s
Qwen3-VL-8B 19.0 s
Gemma 4 26B A4B 19.2 s
Qwen3.8-27B 29.4 s
Qwen3.6-35B-A3B 156.8 s

Those prompts ran from about 2K to 4K tokens. Load the context window your longest real prompt needs: with an 8K window here, the 16K-and-up class couldn’t be tested at all.

6. Run a few small capability checks

Each model also ran short checks of 2 to 10 items. They tell you whether a feature works, not which model is best at it.

Capability checks and the extraction-suite mean, per model. Each cell is a score from 0 to 1, shaded by rank so close scores stay apart; a blank cell was not claimed, not zero. Hover, focus or tap a cell for its item count.
ModelText extractionVisionVideoTool callingLong contextThinking onThinking offSuite mean
Qwen3-VL-8B1.000.751.001.001.00not claimednot claimed0.922
Gemma 4 12B QAT1.001.001.001.001.000.751.000.935
Gemma 4 26B A4B0.801.001.001.001.001.001.000.947
Qwen3.6-35B-A3B1.000.751.001.001.001.000.750.909
Qwen3.8-27B1.001.001.001.001.001.001.000.969
CheckQwen3-VL-8BGemma 4 12B QATGemma 4 26B A4BQwen3.6-35B-A3BQwen3.8-27B
Text extraction1.001.000.801.001.00
Vision0.751.001.000.751.00
Video1.001.001.001.001.00
Tool calling1.001.001.001.001.00
Long context1.001.001.001.001.00
Thinking onnot claimed0.751.001.001.00
Thinking offnot claimed1.001.000.751.00
Suite mean0.9220.9350.9470.9090.969

The checks are small, with 2 to 10 items each. On a four-item check, one item moves a cell from 1.00 to 0.75.

  • Text extraction. Four models got 10 of 10 fields. Gemma 4 26B A4B got 8 of 10.
  • Vision. Qwen3-VL-8B and Qwen3.6-35B-A3B scored 0.75, each missing the code-reading item (Qwen3-VL-8B read 4821 as 4211). The other three scored 1.00.
  • Thinking. Test it on and off. Gemma 4 12B QAT answered 3 of 4 with thinking on and 4 of 4 with it off, while writing about 2,500 tokens per answer against about 240. Qwen3.6-35B-A3B went the other way: 4 of 4 on, 3 of 4 off.
  • Tool calling. All five scored 1.00.

7. Read memory against a baseline

Peak VRAM for every model read 11.4 to 11.7 GiB, which looks like each one filled the card. It didn’t: before four of the five loads the card was already at 11.3 to 11.6 GiB. Read the card before you load a model and subtract, or measure the model on its own. Measured alone, Qwen3-VL-8B used about 9.4 GB.

What the GPU memory reading shows

Peak VRAM per model against a 12 GiB card (12,288 MiB). The dark part is the baseline read before the load, so the bar is the whole GPU, not the model.

  1. Gemma 4 12B QAT11,615 → 11,916 MiB
  2. Qwen3-VL-8B11,812 → 11,832 MiB
  3. Gemma 4 26B A4B11,831 → 11,983 MiB
  4. Qwen3.8-27B11,866 → 11,939 MiB
  5. Qwen3.6-35B-A3B3,027 → 11,917 MiB

Would Qwen3.8-Flash-Next fit?

The smallest version of the model is 67.56 GiB. Choose where it has to go.

RAM + VRAM: 67.56 GiB of weights against 75.2 GiB leaves 7.64 GiB for the operating system and the model's working memory. Unsloth gives 75 GB of RAM plus VRAM as the minimum for this version. So it was "not attempted", not "does not fit".

Show the numbers as a table
Qwen3.8-Flash-Next, Unsloth's smallest 1-bit version: 67.56 GiB. Differences are derived.
WhereAvailableWeights minus available
The GPU alone12.0 GiB of VRAM55.56 GiB short
RAM + VRAM75.2 GiB (63.2 GiB RAM + 12 GiB VRAM)-7.64 GiB spare

Do the fit arithmetic before a big download, and check the free disk space too. The smallest Qwen3.8-Flash-Next download, Unsloth’s 1-bit quant, is 67.6 GiB. Against 63 GiB of RAM plus 12 GiB of VRAM that leaves about 7.6 GiB for everything else, under Unsloth’s stated 75 GB minimum, so it wasn’t attempted. The explorer above runs the same arithmetic.

What came out of it

  • Short documents: Qwen3-VL-8B with the tuned prompt. At about 81 tok/s, 61,000 documents take roughly 18 hours.
  • Prompts of a few thousand tokens: Gemma 4 12B QAT.
  • Next round: a held-out set, a real long-prompt run, Qwen3.8-27B on all 40, and the extraction suite rerun with the tuned settings. Expert placement is tuned for all five on an idle card, but the scores and suite speeds above predate it.

Verdict

On one 12 GB card, Qwen3-VL-8B suits short documents at about 81 tok/s, Gemma 4 12B QAT prompts of a few thousand tokens. Subtract the card’s memory before each load, or every model looks full.

Test your own card: download LM Studio at lmstudio.ai and run these models against your own prompts.

Sources:r/LocalLLM: llama.cpp settings for 12 GB, r/LocalLLM: Gemma 4 26B A4B on 8 GB, r/LocalLLaMA: Qwen3.8 27B and Qwen3.6 35B-A3B on 12 GB.

Responses

Sign in to write a response

Keep reading