The Same Model Twice

A model’s label hides three variables that decide its output quality: precision, task type, and finish. A comparison between two models that has not fixed all three tells you nothing. We learned this by spending ten evaluation rounds looking for “a stronger model”. The stronger model was the same model, quantized to a quarter of its precision, doing a harder job.

Core takeaways

• Two checkpoints with the same parameter count can differ in precision, task type, and finish. Each of the three changes output quality on its own.
• Quantization compresses the model’s weights. Text models lose little at 4-bit. Voice models get audibly rougher, and the damage lands in texture and stability.
• A model producing its own trained-in voices does a solved task. The same model cloning a stranger’s voice from a short clip works from far less information, and at equal size the preset voice wins almost by construction.
• “Base” in a model name means the un-tuned checkpoint meant to be built on. The variant suffix tells you which task the checkpoint was finished for.
• Before comparing two local models, record three lines for each: exact checkpoint ID, quantization level, and variant. Ten rounds of evaluation gave us less information than one cat config.json.
• Full precision costs about double the disk and memory of 4-bit. For final-quality audio, buy precision. For bulk text, buy parameters.

What happened

We generate character voices locally for an audio project. The original voices came from a TTS model we called “the old model”. A newer download, which we called “the stronger model”, kept producing voices that were less clear, less stable, and similar to one another. We ran ten rounds of bake-offs on it: new seeds, blends, reference engineering, style instructions. The new model never delivered the quality we wanted. In every blind A/B against the old model’s output, the old model won or tied.

Then we read the two checkpoints off the disk instead of relying on what we assumed about them:

• “Old model”: Qwen3-TTS-12Hz-1.7B-VoiceDesign-bf16. 1.7B parameters, full bf16 precision, 4.2 GB.
• “New model”: Qwen3-TTS-12Hz-1.7B-Base-4bit. The same 1.7B parameters, 4-bit quantized, 2.2 GB.

Same architecture, same size, same family. The “upgrade” was a drop in precision under a different variant name. The variant name mattered as well. VoiceDesign generates voices it was trained to finish. Base clones arbitrary voices from about 60 seconds of reference audio, which is a harder task done with much less information. We had stacked two downgrades, called the stack “stronger”, and then spent two days puzzled that it underperformed.

Precision

Quantization is a compression of the model’s weights. Text models lose little at 4-bit. Voice models get audibly rougher, and the damage lands hardest in texture and stability, which are the qualities a voice is judged on. The sign is one block in the checkpoint’s config.json: a quantization entry, bits: 4 in ours. Checking it takes ten seconds. Nobody checked it for two days.

Task type

A model generating its own trained-in voices is doing a solved task. Those voices are finished products, polished end to end. The same model cloning a stranger’s voice from a short clip is working from far less information. At equal size, a preset voice beats a cloned voice almost by construction. “The cloning model sounds worse” is the expected result. We read it as a defect.

Finish

“Base” in a model name means the un-tuned checkpoint meant to be built on. The variant suffix carries real information. The family we were using ships as Base (cloning), VoiceDesign (designed voices), and CustomVoice. CustomVoice is a third variant, plausibly purpose-built for the exact original-voice task we were hand-rolling on Base. We never noticed it for two days because we never listed the family.

How to pin the three variables

Before comparing any two local models, record three lines per model: exact checkpoint ID, quantization level, variant/task type. On disk that is ls ~/.cache/huggingface/hub plus the quantization block in each config.json. If a comparison is already running and one side disappoints, check its quantization before abandoning it. The bf16 or 8-bit sibling of a 4-bit checkpoint usually exists, and it usually explains the gap. Read the whole variant family before building workarounds. The tool you are improvising may already ship as a variant.

What full precision costs

Full precision costs roughly double the disk and memory of 4-bit (4.2 GB against 2.2 GB here) and generates more slowly. For text LLMs at long context, that trade often favors quantization: more parameters at less precision can win. The rule depends on the domain. For final-quality audio, buy precision. For bulk text, buy parameters.

When pinning is not worth it

Most of the time the defaults are fine, and ceremony around every download is its own waste. The discipline pays at the moment you are about to compare. When two models’ outputs will be judged against each other, their labels must be fully expanded, because that is when a hidden variable turns directly into wasted rounds.

What the case shows

The model that “underperformed” was never weaker. It was the same model with two handicaps we had added ourselves: lower precision and a harder task. Ten rounds of evaluation produced less insight than one cat config.json. Quality differences between models are explained by precision, task, and finish before they are explained by capability, and the label states none of the three. Expand the label before you compare.

How to practice this

1. Before comparing two local models, write three lines for each: exact checkpoint ID, quantization level, and variant. Run ls ~/.cache/huggingface/hub to get the checkpoint IDs. Notice whether the two models share the same parameter count.
2. Open each checkpoint’s config.json and look for a quantization block. Notice whether one side reads bits: 4 while the other is full precision.
3. If one side of a running comparison disappoints, check its quantization before you abandon it. Look for the bf16 or 8-bit sibling of the 4-bit checkpoint. Notice whether the sibling closes the quality gap.
4. List the whole variant family before you build a workaround. Notice whether a variant already ships for the task you are hand-rolling.
5. Weigh the cost by domain before choosing precision. Notice that full precision takes about double the disk and memory, 4.2 GB against 2.2 GB here. For final-quality audio, take precision. For bulk text, take parameters.