Local Model Evaluation & Intervention

Who it is for — and why

Teams changing local-model behavior who need to show which adapter was selected, what it improved, what it may have cost, and whether the exact result can be reconstructed later.

What was verified

• The Qwen2.5-Coder-3B campaign searched 200 Heretic trials, retained a 17-point Pareto front, and selected Trial 187 by a frozen rule: 10 refusals per 100 at KL divergence 0.0078
• The selected 3B adapter and untouched base ran through the same prompt ids across 10 capability and refusal tasks; the 75-minute run completed with no failed stages or retries
• Paired screening moved heuristic refusal rates from 85.0% to 6.7% on StrongREJECT-small and from 71.7% to 13.3% on XSTest aggregate, while code-line description accuracy was unchanged and instruction-following improved on this sample
• Thirteen release checks passed with zero blockers or warnings; the private adapter, GGUF, AAR, metrics, Pareto data, recall identity, and 34-file hash manifest were uploaded and re-verified

Limits

• These are quick-tier screening results from one seed on one RTX 3060 with 12 GB VRAM; they are not yet a publication-tier model claim
• StrongREJECT-small and XSTest figures use deterministic refusal heuristics, not an official StrongREJECT judge or a human safety audit
• The 7B comparison is a retained legacy run without per-sample paired records; it is context rather than confidence-interval evidence
• HumanEval+ and MBPP+ are excluded until a validated execution sandbox exists; raw prompts, completions, weights, and private Hub artifacts remain private
• The run predates run-start evaluator Git metadata, so package and report-time provenance exist but the exact evaluator source commit at launch does not

Access and evidence