A fast decision model for the enemy pilots

What was tried

On October 2, we set out to trial a small, fast trained model for the enemy AI, and a way to train it. It belongs to a new class of model that its developers call “System 1” models, after the fast, intuitive System 1 in Daniel Kahneman’s account of how people think: a small model that answers a structured question in one quick pass, made for decisions in real time, where a reasoning model works through a problem step by step. It might not be the right answer for enemy AI, which is why it needed a trial.

In the trial, the scripted pilot kept flying the ship. About three times a second the model was given a few sentences describing the fight (range, bearing, how hurt each ship is) and picked one of four goals: pursue, attack, break off or reposition. An answer took about 8 ms on the M5 Mac the game is built on.

Training it on outcomes

The labels score outcomes. At a moment in a scripted duel, a labeler replays the duel once for each goal with the pilot holding that goal, scores what followed (damage dealt less damage taken, a kill or a loss), and writes the moment in words with a probability for each goal. The model was fine-tuned on those labels and flown against the scripted pilot in headless duels among city blocks, with sides swapped.

Generations 1 to 4 each flew 60 duels. Generation 1 trained for about 4 minutes and won 43 to 17 by pressing the attack. A pilot that held attack for the whole duel scored 35–25 on the same duels, so generation 1 had learned something beyond that. Generation 2, with six times the data, learned only to attack and scored exactly what holding attack scores. Generations 3 and 4 used different label designs and each leaned on one goal, a different one each time.

The diagnosis

Three label designs in a row had produced a one-goal model, so training stopped and an analysis of the labels and checkpoints looked for the cause. It found three things.

• Each fine-tuned model answered with about the average label of its training set in every situation. The pilot switches goals only when another goal beats the current one by 0.1, so it held whichever goal it started on or first landed on.
• A goal the model chose never ended in play, while the labels had scored each goal for only 2 or 4 s.
• A bag-of-words regression, which only counts the words in the description, predicted the labels about as well as the fine-tuned model. It predicted the scripted pilot’s own goal 88 to 91% of the time, so the words describe the situation well and the weak part is the outcome labels.

We decided there would be no more training until we had recorded flights of our own. Checks on a later set of labels found that half the moments have a clearly best goal and that the description does not say which. With shields and missile lock-on in the game, a pilot that only attacked beat the scripted pilot 52 to 3 in 120 s duels, with 5 draws. A first model trained on our flights scored 4 wins, 7 losses and 49 draws against the scripted pilot.

Instinct plus reasoning

On October 9, we settled on a blend. Each enemy carries a personality matrix, a set of numbers that is its core behavior, and the model adds to it. Every enemy pursues two goals in order, survive first and win second. We built the matrix and a survival-first arbiter, added the model as an advisor with a weight, and let an experiment set that weight.

The experiment was a round robin of four pilots: the scripted pilot, the model as flown, a pilot that only attacks, and a blend of the model with the scripted survival rules. Each seed was flown twice with sides swapped, and all six pairings took 11 minutes.

The gap is an Elo rating difference, as in chess. A positive number means the first pilot wins more, and +100 is about a 64% score. The model’s goal choices rated 80 points under holding attack, and the survival rule made no difference to that. The model added nothing to goal choice, so its weight in the enemy profiles is zero for now. The arbiter now picks the dogfight enemies’ goals and rates +101 over the scripted pilot it replaced.

Where it stands

The trial covers one model, one kind of label and one kind of opponent. The labels came from scripted duels between two equal ships. A human pilot flies differently, and fights with two or three enemies may reward survival more. Further training waits for labels that do not come from scripted duels, such as our flights.

We are still investigating fast models of this kind. This one stays in the build until we have seen how the scripted pilots perform and have tried tuning it under the blended design. A setting in the game still lets it fly the enemies.