Useful Offline, Harmful Online

After causal self-rollout failed, the next attempt removed online tree search. A permutation-invariant neural controller would read the bounded Gaussian-mixture belief and directly predict the full-information oracle’s six action values.

Offline, the idea worked well enough to be interesting. On fresh repeated control, the frozen policy was harmful. One capped dataset-aggregation round improved the states the policy actually visited and produced a promising known-schedule specialist—but the same corrected controller failed under random hazards.

The central lesson is not simply “distribution shift is bad.” It is more specific:

A state representation can contain substantial information about oracle value while a policy trained on that state still fails through sequential feedback.

The source artifacts, grouped uncertainty, and claim ledger are in dwrtz/ml-examples at commit ddc9215.

What the controller could see#

The controller received the complete bounded mixture as a set of component features, together with current causal context and compute information. A Deep-Sets-style encoder made the representation invariant to component order. There was no recurrent memory and no exact-grid input.

Training targets came from the full-information one-step oracle. For each state, the dataset stored all six action values rather than only the winning label. The replicated corpus contained 1,408 states, 8,448 complete action-value rows, and 94 independent physical groups. The primary split used 946 training, 253 validation, and 209 untouched in-distribution test states.

We measured recovered oracle gain rather than top-1 accuracy alone. If the oracle improves over a fixed policy by \(G^*\), and the learned action improves by \(G_\theta\), recovery is

This credits near-optimal actions and preserves the scale of the decision.

The offline signal was real—and heterogeneous#

Two target constructions were tested. Raw targets directly regressed the relative oracle action values. Conditional targets averaged repeated oracle estimates within each physical group to reduce target noise.

The raw target won validation selection, so 64.4% is the formal primary test result. The conditional model’s 76.8% test recovery is a legitimate held-out measurement, but the already opened test set cannot be used to retroactively replace the selected target.

Five-fold rotation across physical groups then evaluated the frozen conditional architecture for three seeds. Recovery was 68.9%, 74.1%, and 67.7%, or 70.2% with a 3.4-point standard deviation across seeds. Every grouped confidence interval was wide, and all three crossed the declared 80% promotion line.

That line was a project decision rule, not a law of nature. Results of 76.8%, 78.4%, and 70.2% do not show that the state is “insufficient.” They show substantial, reproducible, and heterogeneous predictive structure that did not meet a chosen promotion convention.

The same conditional model recovered positive oracle value in three offline shifts: 58.7% under periodic-variance shift, 71.1% under delayed-probe-variance shift, and 59.3% under changed dynamics. An adversarial equal-moment suite was negative at -3.3%. Seven of twelve constructed pairs with identical moments had different oracle actions, while the full component set distinguished every pair. Shape information mattered; moments alone could not represent it.

Repeated control changed the problem#

The original frozen policy was then evaluated on fresh decision states and paired rollouts. At one step it was weakly useful only in the known-schedule setting. Over four steps it became harmful:

Controller latency was not the culprit: 2.94 ms mean and 3.14 ms at the 95th percentile. The selected actions changed the next approximate belief, which changed the next input, which changed the next action. Supervised errors entered a feedback loop.

On 512 known-schedule states per step, top-1 agreement fell from 25.0% at the root to 18.6% after one learned action. Mean action-value regret rose from 0.3413 to 0.6157. Recovery of the fixed-policy gap collapsed from 27.6% to 5.0% before partially rebounding later.

Mode drops rose only from 3.1% to about 5%. Basin-mass error stayed near 0.10–0.12. The diagnosis was therefore not a single dramatic loss of modes. It was fresh-state mismatch plus sequential covariate shift across the complete bounded belief.

One correction, declared in advance#

The study allowed one capped DAgger-style correction. The architecture, target, width, and seed remained fixed. Training combined 946 original states with a deterministic cap of 946 policy-visited states labeled by the oracle. Opened test and online results did not select a new model or aggregation weight.

On the selected policy-visited slice, recovered gain rose from 13.5% to 30.2% and mean action-value regret fell from 0.4593 to 0.3703. On all visited states, descriptive recovery rose from 9.3% to 26.8%. Original validation recovery held steady, 73.1% to 73.6%.

This is the expected shape of a useful dataset-aggregation update: it improved the distribution induced by the policy without buying the result through a new architecture search.

Fresh final seeds then gave a mixed answer:

The known-schedule four-step result is a promising specialist. It improved mean cost by 27.8%, beat every declared simple comparator in that setting, and cut worst-tail cost roughly in half. Its grouped interval still crosses zero, and the delayed families that motivated the option-value study remained negative. It is evidence worth preserving, not a promoted policy.

Random hazards supplied the stopping signal. One-step harm was resolved, the four-step mean stayed negative, and a simple local-nonlinearity rule was far better. The declared rule said to stop after the single correction if the candidate still lost to fixed and the best simple heuristic. It did.

What did not happen next#

We did not respond by adding another DAgger round, recurrent memory, a wider network, a Set Transformer, more random seeds for model selection, or policy gradient tuning on the opened evaluation. Those are new research branches, not free repairs to the same claim.

A context gate that routes known schedules to the corrected neural specialist and random hazards to a simple causal rule is an appealing post-hoc hypothesis. The current results cannot validate it: they are exactly the results that suggested the gate. It needs a separately declared experiment and new seeds.

The first cycle therefore ends with two truths intact. The bounded mixture state contains useful oracle-value signal. The tested learned controller is not a general online solution. Under random hazards, the most useful result came from a far simpler policy.