How Cartmark tests whether a game works

1. A machine with a screen (a games console box under the TV, or a Windows PC) is handed one game.
2. It starts the game in an emulator, the same way you would, and watches.
3. Every fifteen seconds it takes a picture of the screen and sends it, with anything the emulator has written to its log, back to the server.
4. The server reads the words on each picture and the log, and answers: carry on, press a button, give it more time, or stop, the answer is already clear.
5. At the end, the last picture decides the result, unless an earlier picture already showed the game itself. All the pictures are joined into a short video you can watch on the game's page.

Nothing is decided from a file's name or a database entry. Every "works" on the site is backed by a picture of the game running, and now by a recording of how it got there.

"Booted" is a deliberately modest claim. It means the game started. It does not mean every level plays perfectly; that is a separate, longer test.

Each picture is read by optical character recognition, the same technology that turns a scanned letter into text. The words are then matched against what games and emulators say at particular moments. A few real examples from that week:

• "Installing required game data. Please do not turn off the PlayStation 3 system": the game is running and installing. That counts as booted, and the test is given more time.
• "Precompiling shader interpreter variants. Please wait. Building base variant 5916 of 6650": the emulator is preparing itself, not the game. Also given more time.
• "The PS3 application has likely crashed, you can close it.": the test stops at once as failed to load; there is no point watching a crashed game for another four minutes.
• "Install Error. An error has occurred while installing.": the game's own install failed. Failed to load.
• "PRESS START BUTTON": the game is at its title screen and waiting. The machine presses Start, as a player would, up to three times, and then ends the test: the game has clearly started.

The words are kept with the result, so every verdict says why it was given.

A test has a time window, a few minutes depending on the console. Two things change it:

• Ending early. When the answer cannot change (a crash, or a game that has reached its title screen and is waiting for a button), the test stops there. That frees the machine for the next game.
• Running longer. When the screen shows the game installing, or the emulator compiling, the window is pushed back so it always stays two minutes ahead of the run. It is never pushed back by more than twenty minutes in total, so a game stuck on an install screen forever cannot hold a machine forever.

Pressing buttons is done with the emulator's own keyboard layout: Enter for Start, and the key the emulator maps to Cross, or to A on Nintendo systems. Where a machine's emulator layout is not known, nothing is pressed rather than guessing.

Some PlayStation 3 games in the library had been copied by a tool that "helpfully" renamed files by guessing their type. A video called SMLOGO.M2V became SMLOGO.M2V.mpg; a sound bank called common.snd became common.snd.au. The game then asks for its file by the original name, does not find it, and stops.

The emulator writes every file it could not find to its log. This is a real line from a God of War III run:

·E 0:00:05.464330 sys_fs: 'sys_fs_stat' failed with 0x80010006 : CELL_ENOENT, "/dev_bdvd/PS3_GAME/USRDIR/CINEMATICS/SMLOGO.M2V"

(CELL_ENOENT is the console's "no such file".) The machine sends these lines to the server, which looks in the game's folder for exactly one file that must have been the missing one: the same name with one extra ending, or the same name one folder higher. If there is exactly one, it is given back its original name. If there are two candidates, nothing is done, because guessing would be worse than leaving it.

The repair is careful by design:

• Nothing is renamed, moved, overwritten or deleted. The original file gets a second name (a "hard link"), which takes no extra space and can be removed to undo the repair exactly.
• Nothing outside that game's own folder is ever touched.
• Once one file shows how a folder was damaged, every other file damaged the same way is repaired too, instead of one per test.
• The game is then tested again straight away, ahead of everything else on that machine, so its new result reflects the repaired folder.

What it looked like that morning

To prove this was the repair and not luck, the repairs were then undone by hand and the games queued again. God of War III's shows the result: the run at 06:03:10 found the file missing again, the server restored it, and the retest six minutes later booted, without anyone touching it.

The same morning a PC tested Silent Hill HD Collection. For twenty minutes the pictures showed the game installing its data. Then the PC's power settings turned its screen off, and the last picture was black. The test was recorded as "console screen only", which was wrong: the game had plainly started.

Two changes came from that run:

• While a test is running, the PC now asks Windows to keep its display on, the way a video player does, and withdraws the request when the test ends. The owner's power settings are not changed.
• A test now remembers the first time it saw the game itself (installing, or at its own prompt). If the last picture shows less than that, the result is still "booted", with a note saying what was seen and when. The first test after the change, at 08:05:59 on the console box, was recorded exactly this way: "the game was seen installing its data at 13 s; the final frame showed less (boot_logo)".

On any game's page, the boot result opens a recording of the test: the frames from launch to the end, with the final picture held for a few seconds. Underneath are the machine, the emulator and its version, and the words that decided the result. If a result looks wrong to you, the recording shows exactly what the machine saw.

• It does not play the game. A game that boots and then breaks on level three is "booted" here.
• It cannot read words in a picture that has none. A silent, wordless title screen is judged by how the picture looks, which is less certain, and such results are tested a second time with a longer window.
• It does not repair a file it cannot identify with certainty. Those games stay failed until someone looks.

• The emulator's own log is read for a cause on every failed test: a copy the emulator refuses is replaced and searched for again; missing firmware or a core that will not load is put down to the set-up, not the game. A log line changes a verdict only when its phrase has been confirmed and the log is provably that test's own.
• On the TV box, a key is pressed only while the test's own game is on screen and nobody is at the controls; a game started by mistake is closed within seconds.
• A game you are seen playing gets its boot test at the front of the queue, started the way you played it, and a test never leaves an emulator muted afterwards.
• A "wrong shelf" verdict offers the file a move to the system it is really for, and arcade sets are proven file by file against MAME's own list before a test is spent.
• A machine that starts the same games the others start is trusted on that system; one that fails what the others run has its results held, with a note to check its emulator setup.