It doesn't check the file. It plays the game.
A checksum tells you a file is the right file. It can't tell you the disc image is missing its second track, that the emulator needs firmware you don't have, or that a copy tool renamed half the folder. Only switching the game on can. So Cartmark switches every game on.
Five steps, no hands
A real machine is handed one game
The box under the TV, a Windows PC that is sitting idle, or for cartridge-era systems the server's own test bench running the same emulator core. A game somebody asks for is tested on the machine they play on, and a game you are seen playing gets its test at the front of the queue, started the way you played it.
It launches the game the way you would
Same emulator, same settings, same controller layout. A Wii or Wii U game gets the pad the game takes.
Every fifteen seconds it takes a picture
The picture, and anything the emulator wrote to its log, go back to the server.
The server reads the screen
It reads the words in each picture and answers: carry on, press a button, give it more time, or stop, the answer is already clear.
A verdict, with a recording
The pictures are joined into a short video that stays on the game's page, beside the machine, the emulator version and the words that decided it.
Nothing is decided from a file's name or a database entry. Every "works" is backed by a picture of the game running.
What a result means
| Result shown | What the machine saw |
|---|---|
| Booted | The game's own screen: a title, a menu, its own logo, its install screen |
| Console screen only | The console's or emulator's own screen, never the game's |
| Black / no picture | Nothing drawn at all |
| Failed to load | The emulator or the game said it had stopped: a crash, an install error, a missing disc |
| Too slow | The emulator was still preparing itself when time ran out |
| Wrong shelf | The file is really a game for another system, and is offered a move there |
"Booted" is a deliberately modest claim. It means the game started. Whether level three plays perfectly is a different, longer test, and Cartmark says so rather than pretending.
It reads the screen like a person would
The words on each picture are matched against what games and emulators say at particular moments. Real examples from one week:
- "Installing required game data. Please do not turn off the system": the game is running and installing. Counted as booted, and given more time.
- "Precompiling shader variants. Please wait.": the emulator is getting ready, not the game. More time.
- "The application has likely crashed": stopped at once. No point watching a crashed game for four more minutes.
- "PRESS START BUTTON": the machine presses Start, as a player would, and ends the test. The game has clearly started.
A crash frees the machine in seconds. A long install is never cut off early, and never allowed to hold a machine for more than twenty extra minutes. The words are kept with the result, so every verdict says why it was given.
It reads the emulator's own words too. With each test the machine sends the end of the emulator's log. A copy the emulator itself refuses is replaced and searched for again, while missing firmware or a core that will not load is put down to the set-up, never the game. A log line changes a verdict only when its phrase has been confirmed and the log is that test's own.
A test never leaves an emulator silent. Windows remembered the mute a test set, so some games played without sound. Each emulator a test muted gets its sound back the first time it plays on each speaker or headset, and a mute you set yourself stays.
Real tests, exactly as the machines saw them
Everything pictured here is homebrew: games written by independent developers for old hardware, many for community game jams. Three recordings first, then the final frame of ten more tests.
Ninety-one broken files, repaired in one pass
Some PlayStation 3 games had been copied by a tool that "helpfully" renamed files by guessing their type. A video called SMLOGO.M2V became SMLOGO.M2V.mpg. The game asks for its file by the original name, doesn't find it, and stops.
The emulator's log names every file it couldn't find. Cartmark reads those lines, looks in the game's own folder for the one file that must have been the missing one, and gives it its name back. Then it tests the game again, straight away.
| Time | Game | What happened |
|---|---|---|
| 06:03 | God of War III | "Console screen only". One renamed video found and restored; retest queued |
| 06:09 | God of War III | Retest on the same PC: booted |
| 07:19 to 07:26 | Silent Hill HD Collection | Four tests in seven minutes, each restoring one more sound file |
| 08:06 | Silent Hill HD Collection | With repair-by-pattern in place: 91 files restored in one pass, and the game filmed starting |
The repair is careful
- Nothing is renamed, moved or deleted. The file gets a second name that takes no extra space and can be removed to undo the repair exactly.
- Nothing outside that game's folder is touched.
And it knows when to stop
- Two candidates? It does nothing. Guessing would be worse than leaving it.
- Can't identify the file with certainty? The game stays failed until a person looks.
To prove it was the repair and not luck, the repairs were undone by hand and the games queued again. The files were found missing, restored, and the games booted, without anyone touching them. Read the full account.
When two machines disagree, Cartmark asks why
A game that fails on one PC and starts on another is not a broken game. It is a question about the PC. Cartmark keeps score: a machine that starts the same games the others start is trusted on that system, and results can carry to it without repeating every test. A machine that fails what the others run has its results held, with a note to check its emulator setup.
Every sentence in the list below was written by the system at the moment it became true.
Four more checks every file gets
- Is it the right file? Checksums against the No-Intro and Redump preservation lists say which exact release it is. Each system's own list is on its page, and a game the lists know but the shelf lacks can be monitored from it in one press.
- Is it really that game? Serial numbers are read from inside disc images and checked against the name. That caught 57 Xbox games stored as one. PC Engine CD, PC-FX, 3DO, CD-i and Amiga CD32 discs are known by the label each disc carries.
- Is it in your language? A copy in none of your preferred languages is set aside and a better one searched for, before a test is spent on it.
- Will achievements work? Each file is hashed the way RetroAchievements does it: green for the current set, red for a retired dump, grey for never supported. Where a game is held without the dump RetroAchievements accepts, the system's page lists the upgrade.
- Is it safe to run? Every Windows game and installer is virus-scanned before anything can run it, and a copy found to hold a threat is set aside out of reach. How scanning works.
Every file, tested again from nothing
The database was begun again with only its settings kept, and the whole library was imported and checked as if new: no earlier result was carried over. In the first 23 hours 21,043 tests were recorded on 20,851 files.
- 19,098 started (nine in ten), each with a picture of the game's own screen.
- 806 showed a black screen and 120 showed nothing at all. These are left for a second, longer look; a black screen alone never sets a copy aside.
- 690 need real hardware: systems the server's bench cannot judge, queued for the TV box or a PC.
- 227 reached only the console's own screen, 48 turned out to be on the wrong shelf, and 33 were refused by the emulator itself.
About nine tests in ten ran on the server's own test bench, which starts cartridge-era games with the same emulator cores the machines use. The rest ran on the TV box and two Windows PCs, and a game somebody asks for is tested on the machine they play on.
Since 0.30 every import goes through eight steps before it reaches a shelf: the file, its platform, its language and a real start. A machine's "it started" counts only with a picture from that test, and only a failure the emulator itself reports sends a copy back to be replaced. Since 0.39 an import that is blocked mostly settles itself: what today's readers prove is filed and checked without you, and the few that need a person have one button each.
What this test does not do
- It does not play the game through. A game that boots and then breaks on level three is "booted" here.
- A silent, wordless title screen is judged by how the picture looks, which is less certain. Those are tested a second time with a longer window.
- It only starts a test on a machine nobody is using. Your game night comes first, and a game left open behind another window still counts as you playing.