Finding 1.4-second stalls by replaying play

What we found in play

On October 6, play in the large cities hit many slowdowns, once down to a complete crawl. The game’s telemetry put them in the simulation, 0.7 to 1.4 s after charges that broke 140 to 500 pieces of structure. The stalls reported in play ran from 0.2 to 1.4 s.

The marker key

That morning we wanted a way to point at exact moments, to get better information about performance problems. The game now keeps the last 4,096 frames of live play in memory, about 34 s at 120 frames a second, with the time each part of the frame and each phase of the simulation took. Pressing X on the keyboard or Y on a gamepad shows a “Marked” count and saves that moment with the camera position, the 20 s before it and the 5 s after. Any frame over 40 ms is saved with its breakdown even when nothing was marked. Recording a frame took the main thread under 5 µs at the median, and the game ended in the same state with recording on and off.

Replaying the session

The game logs each play session. A replay tool sets off the logged blasts and cannon rounds again at their original game times, without a window, in the city that session last loaded. A replay of our mixed city up to one of the stalls reached 836 ms in a single physics step. A physics step is one slice of simulated time, 1/60 s, that the engine computes at once. That made the stall repeatable, so each cause could be measured, fixed and measured again. Each fix below was checked by replaying a logged session and comparing a state digest, a fingerprint of the whole world at the end of a run. In each case the digest matched, so the fix changed timing and nothing else.

Causes, one at a time

On a replay of our October 7 session, a second round of work on the compaction took it from 125 and 134 ms to 60 and 65 ms. The bookkeeping after rubble comes down also got cheaper. In a replay of our large-city session at the low tier, steps over 33 ms fell from 40 to 79 a run to 4 to 48.

Suspects that saved nothing

In the heaviest window of a replay of our first city, run in Low Power Mode (the throttled macOS setting used as the baseline), a rocket barrage left up to 22,000 bodies awake, 99% of them fracture debris. About half were nearly still, held awake by one group of 7,000 to 9,800 touching bodies that the physics engine can only put to sleep together. Steps there reached 84 to 114 ms. Five changes were tried, and each saved nothing: fewer job threads, fewer solver iterations, a looser contact cache, a looser sleep threshold and a blast floor. Freezing rubble in place after 1 s instead of 2 did help, cutting the slowest step to 62 to 70 ms.

What we decided

There were two ways to cut the load, freezing rubble sooner or making fewer fragments. We kept the fracturing as it is and chose the freeze, at 1.4 s rather than the 1 s the test used. The change at the high tier, from 2 s to 1.4 s, waits on a second replay to confirm it.

The next morning

On the morning of October 7, ten minutes of play in the large downtown city became the new replay baseline. The one-second freezes were almost gone. A huge missile dropped in the middle of the city, and several other large drops, caused no freeze at all. We marked one moment with floating debris.