Turning model modification into a gated product decision

Context

The first version of this work behaved like a research project: pick a model, run a long evaluation, inspect the result, and decide what to do next. That works once. It does not support iterative weight changes, because every new adapter reopens the same questions—was the base identical, did both arms see the same prompts, which checkpoint was selected, did the run fail halfway through, what can be deleted, and what evidence is strong enough to publish. On one 12 GiB GPU, a thirty-six-hour test is not just slow; it removes the feedback loop the product needs.

Decision

Treat evaluation as a product operating system rather than a final QA step. Define the decision and resource envelope first; make model selection, alteration, candidate selection, paired evaluation, evidence maturity, retention, and publication separate gates; and require every gate to leave a checkable artifact behind.

Approach

• Started with an evidence contract: the decision the run must support, the behavior to change, the capability floor to protect, the GPU/disk/time envelope, and the conditions that stop the experiment instead of extending it indefinitely.
• Turned model selection into a feasibility funnel. Task fit, license, architecture and tokenizer support, quantization, adapter compatibility, VRAM, disk footprint, and overnight runtime all have veto power before an expensive run begins.
• Pinned the untouched base by model revision and file hashes, then made a removable LoRA adapter the intervention unit. That keeps every experiment reversible and makes retaining a result cheaper than retaining another full model copy.
• Journaled the full Heretic search and preserved the Pareto front rather than only the winner. A frozen policy selects the lowest-drift candidate that meets the refusal threshold, so the rule does not move after an attractive trial appears.
• Separated the quick gate from the publication tier. A quick or code-focused suite must finish in an iteration-friendly window and decides whether more compute is warranted; it is never silently promoted into a publication claim.
• Required OG and OB to run the same prompt ids under the same runtime, with per-sample records retained. Refusal signals and capability measures remain separate, and confidence intervals are added only where the scorer and records actually support them.
• Made checkpoints and failure state first-class: live progress, resumable queues, a single GPU lease, time budgets, top-five Pareto retention, and explicit gates for selection, public promotion, deletion, and cleanup.
• Packaged the adapter, optional GGUF, base identity, hashes, metrics, Pareto data, audit chain, AAR, limitations, and recall instructions together. Once that bundle re-verifies, duplicate base weights can be removed without erasing the experiment's identity.

Outcomes

• The first complete code-model cycle searched 200 interventions, retained a 17-point Pareto front, selected Trial 187 under the frozen policy, and completed its 10-task paired quick suite in 75 minutes—short enough to support a nightly iteration rather than a multi-day wait.
• The selected 3B adapter passed 13 of 13 release checks with zero blockers or warnings; its private recall bundle was uploaded, re-downloaded, and hash-verified before the local result was treated as safely retained.
• A completed Mistral 7B run was imported without pretending it contained evidence it never recorded: its adapter and aggregate comparison remain useful, while absent per-sample pairing is carried forward as a named limitation.
• A Qwen 14B selected adapter completed the paired quick screen on the same consumer GPU. Its larger publication claim remains deliberately withheld behind the publication-tier and reviewed-judge gates.
• The workflow now supports an operator from SSH through a menu or scriptable CLI: create or resume a campaign, inspect the queue and GPU, freeze the Pareto shortlist, evaluate, validate, package, and export machine-readable metrics for plotting.
• The output is a decision record rather than a model trophy: promote, revise, or stop are equally valid conclusions, and the evidence says which one is currently justified.

Control boundary

This is a public-safe operating-method case study built from work on my own hardware and models. Raw prompts, completions, private model artifacts, local paths, and access details stay private. The 3B and 14B numbers described here are quick-tier screening evidence from one seed, not universal safety or capability claims; deterministic refusal heuristics are not an official StrongREJECT judge, and code-execution benchmarks remain excluded until a validated sandbox exists. The workflow records those absences rather than filling them with proxies.

What this taught me

The technical product-management move was not choosing a better benchmark. It was deciding what each level of evidence is allowed to authorize. The model shortlist is a portfolio decision under hardware and licensing constraints. Pareto selection is a prioritization decision between competing outcomes. The quick suite is a spend gate. The publication tier is a claim gate. The recall bundle is lifecycle management. Once those decisions are named, the engineering becomes easier to operate and much harder to overstate—and a negative result still leaves the product in a better state because the next iteration starts from a trusted baseline instead of a pile of checkpoints.