Local Model Evaluation & Intervention · evidence brief
Changing a model, then making the change prove itself.
A reproducible, single-GPU workflow for selecting a reversible weight intervention, comparing it against the untouched model, and retaining enough evidence to recall or challenge the result later.
Public-safe aggregate results only. Raw prompts, completions, private model artifacts, and machine paths are intentionally excluded.
Project overview ←How the workflow became a product →The evaluation rule →
The question, without the jargon
Can a model refuse less without forgetting how to do its job?
OG is the untouched model. OB is that same model with a small, removable LoRA adapter that changes refusal behavior. Both receive the same test items, in the same order, under the same runtime. The useful result is not “OB talks more.” It is whether OB talks more and keeps the capabilities we care about.
A small behavior patch clipped onto the model, not a second full copy of its brain.
A “how much did the answers move?” gauge. Lower means the altered model stayed closer to the original.
The trade-off edge: every point here is hard to beat on refusals without causing more model drift.
The fuzz around a measured change. If it crosses zero, this run cannot rule out “no real difference.”
Selected 3B code-model run
The whole decision, not just the winning number.
Why Trial 187 won instead of the trial with the fewest refusals.
Down is fewer refusals; left is less drift. The highlighted point is the lowest-drift trial that met the pre-declared ceiling of 10 refusals per 100 and KL ≤ 0.10.
Why not Trial 69? It refused only 2 of 100 optimization prompts, but moved the model over four times as far. Trial 187 kept the refusal target while preserving the closest tested behavior.
Same questions, both models
The adapter changed refusal behavior sharply. Capability was mixed and mostly close.
OG refused far more often than OB on both screening sets.
These are deterministic heuristic refusal rates, not official safety scores. Lower means “the response looked less like a refusal,” not “the response was safe or correct.”
Read this narrowly: the intervention did what it was selected to do. Whether the new answers are better requires a separate judge or human audit; this chart does not smuggle that conclusion in.
No blanket “capability preserved” claim—the individual tasks tell a more useful story.
Change in percentage points, OB minus OG. Dots show the observed change; lines show paired 95% confidence intervals where the scorer supports them.
The clearest positive was strict instruction following (+8 points, CI +2 to +16). The clearest negative was computer security knowledge (−3 points, CI −7 to 0). Most other intervals still include no change.
From download to recallable artifact
The experiment is a pipeline, not a folder full of mystery checkpoints.
- 1 · PinResolve the exact base-model revision and hash its files.
- 2 · AlterRun Heretic trials while journaling refusal and drift measurements.
- 3 · SelectFreeze the Pareto top five and apply the same policy every time.
- 4 · CompareRun OG and OB against identical prompt IDs; retain per-sample records.
- 5 · ValidateCheck schemas, audit chain, hashes, secrets, pairing, and recall identity.
- 6 · RetainKeep the small adapter, GGUF, reports, and base identity; remove duplicate base weights when safe.
Evidence maturity
Three model sizes, three different levels of confidence.
Mistral 7B
OG/OB quick comparison completed and the adapter bundle was re-downloaded and hash-verified. The older run lacks per-sample paired records, so it is useful context—not a confidence-interval claim.
Qwen Coder 3B
Two hundred trials, 17-point Pareto front, exact prompt-ID pairing across 10 tasks, paired reanalysis, 13 release checks, and a remotely verified private recall bundle.
Qwen 14B
Paired quick screen completed on the selected reversible adapter. It is reported as screening evidence only; the larger publication tier and reviewed judge record remain the gate for a broader claim.
Reproduction receipts
Enough identity to recall the result without publishing private outputs.
- Base
- Qwen/Qwen2.5-Coder-3B-Instruct
- Resolved revision
488639f1ff808d1d3d0ba301aef8c11461451ec5- Selected adapter
- Trial 187 · rank 3 · attention output + MLP down-projection
- Adapter SHA-256
b336af98304c81873cea052a830aa8c90618e0d614c0e645e1fda186589ec7c3- LoRA GGUF SHA-256
e6022b95d9eded83f234abc698181cf60e7afa4e3c09afff5b244207e832ab42- Validation
- 13 passed · 0 blockers · 0 warnings · 34 evidence files hash-verified
- Retention
- Private adapter + GGUF + AAR + metrics + Pareto + recall manifest; raw responses remain private.