Painfully Obvious.

You might have noticed that your list of installed games looks a bit bland in EmulationStation without artwork, game descriptions, and the like. You could use the metadata scraper that comes with RetroPie, but for MAME games I think it’s better to leverage the pedantry of the community and fill your game lists with more reliable metadata.

Game lists

It is with great ambivalence that I share this fact with you: EmulationStation stores its game metadata in a plaintext, human‐readable format known as XML. I’ll explain what XML is for those of you who don’t know. For those of you who do know, this will be like listening to someone describe the plot of Rocky V if you were unfortunate enough to have seen Rocky V yourself.

XML was handed down from the gods back in the early twenty‐aughts back when people thought that the problem with existing serialization formats was that they weren’t verbose enough. To completely neutralize the main upside of verbosity — the ability to resolve ambiguity from context — they also decreed that an XML parser must fail irrecoverably at the first sign of non‐well‐formedness.

For its faults, XML is at least easy to read and write in a text editor. I’ll take XML any day over a more opaque format. The fact that it can be read and written rather easily by both machines and humans is what made possible the scripts I’ll explain below.

Editing the XML

EmulationStation’s gamelists exist in /home/pi/.emulationstation/gamelists/. Within that directory are subdirectories for each system, and in each such subdirectory is a file named gamelist.xml. Hence the arcade game list can be found at /home/pi/.emulationstation/gamelists/arcade/gamelist.xml.

First, a caveat. The gamelist XML can always safely be read. But it can’t safely be written to unless EmulationStation isn’t running. If you edit any gamelist files while ES is open, your changes will be reverted when ES exits or when the system reboots.

In the last post we talked about a script called quit-emulationstation that I wrote for this very reason. I’ll put it into this post’s accompanying gist for convenience.

Anyway, memorize these two rules for editing gamelist XML:

1. Make a backup of any XML file before you edit it. It takes five seconds and has saved my ass at least a half‐dozen times.
2. Make sure EmulationStation isn’t running so that you don’t spend an hour on bulk edits only to have your changes lost.

Game artwork

Your RetroPie theme will likely have a way to show artwork for a particular game. It’s up to the individual what kind of artwork they want to use — game marquee? game logo? a picture of the cabinet? — but I chose to use a representative screen capture from each game. Other kinds of artwork vary wildly in aspect ratio and are thus hard to harmonize inside an EmulationStation theme. (Screen captures also vary, just not quite as wildly.)

The RetroPie docs on screenshots cover two scenarios: taking your own screenshots and using a utility to scrape screenshots off the web for the games in your list. I went a third way.

I got a full set of MAME screenshots from progettosnaps.net. Like all other MAME-related enthusiast sites, the design has not been updated since the late 90s, and that’s how you know this site is legit. Anyway, once it was downloaded, I was staring at a folder full of PNGs, each one with a filename corresponding to its ROM name.

Apparently there’s no problem I can’t solve by creating a new folder in my home directory. So I created /home/pi/screens, then created subfolders arcade and daphne for the two systems I was emulating. (I’ve got only three Daphne games; I’ll find their screenshots manually.) The PNGs corresponding to the games I needed went into /home/pi/screens/arcade.

At this point, for an arcade game whose ROM is named foo, you can confidently state that its artwork exists at /home/pi/screens/arcade/foo.png.

So all that’s left to do is to update the EmulationStation metadata for each game. You can use this script to do it in bulk.

The script

#!/usr/bin/env ruby

require 'pathname'

begin
  require 'nokogiri'
rescue LoadError => e
  puts "This script requires nokogiri:"
  puts "  $ gem install nokogiri"
  exit 1
end

SYSTEM = 'arcade'

def screenshot_path_for_game(system, game)
  Pathname.new("/home/pi/screens/#{system}/#{game}.png")
end

GAMELIST_DIR = Pathname.new("/home/pi/.emulationstation/gamelists/#{SYSTEM}")

GAMELIST_BACKUP_PATH  = GAMELIST_DIR.join("gamelist.xml.#{Time.now.to_i}.bak")
GAMELIST_CURRENT_PATH = GAMELIST_DIR.join('gamelist.xml')

# Backup current gamelist.xml.
GAMELIST_BACKUP_PATH.open('w') do |f|
  f.write(GAMELIST_CURRENT_PATH.read)
end

GAMELIST_XML = Nokogiri::XML(GAMELIST_CURRENT_PATH.read)

# Traverse the gamelist.
GAMELIST_XML.css('gameList > game').each do |g|
  path_node = g.at_css('path')
  basename = Pathname.new(path_node).basename('.zip').to_s

  art_node = g.at_css('image')
  current_art = art_node.content rescue nil
  new_art = screenshot_path_for_game(SYSTEM, basename)

  puts "GAME: #{basename}"
  if new_art == current_art
    puts "  no change needed"
    next
  end

  # Does the screenshot exist?
  if new_art.file?
    # Update the artwork's modified time so we can more easily tell which
    # artwork files we aren't using later on.
    path.touch
  else
    puts "  could not find screenshot at #{new_art}"
    next
  end

  # Create the `image` element if needed.
  if !art_node
    art_node = Nokogiri::XML::Node.new('image', GAMELIST_XML)
    g.add_child(art_node)
  end

  art_node.content = new_art.to_s
  puts "  changed art path to #{new_art}"
end

# Write out the XML.
GAMELIST_CURRENT_PATH.open('w') do |f|
  GAMELIST_XML.write_xml_to(f)
end

puts "...done."

This script loads the gamelist for a certain system, enumerates all the games in the list, and updates (or creates) pointers to the artwork file for that game.

A couple things:

• This treats the gamelist as the source of truth, not your ROM directory or artwork directory. If you add some games to your ROMs folder, they won’t show up in your gamelist.xml until you (re)launch EmulationStation, at which point ES will notice the new files and create barebones metadata entries for them.
• This script automatically backs up your gamelist.xml to a unique filename each time it’s run. I trust you to clear out the clutter manually once you’ve verified that your bulk edits didn’t screw anything up.

Game category

We can use a similar approach to categorize our arcade games. RetroPie’s built‐in scraper tool is a good abstract strategy for getting metadata about a game, but MAME’s advantage is its community of pedants. For any meaningful piece of metadata you can think of, someone’s already maintaining a file containing that metadata for every game MAME emulates, even the really obscure ones.

The aforementioned progettosnaps.net also hosts a file called catver.ini. It’s a pretty damned impressive taxonomy for a dataset as weird and varied as arcade games. Ever think about what genre Gauntlet is? Now you don’t have to: it’s “Maze / Shooter Large.” And Windjammers, a sports game about a sport that doesn’t exist, is properly labeled as “Sports / Misc.”

Using this list, it’s easy to bulk‐edit your game categories to match what’s defined in catver.ini.

The script

#!/usr/bin/env ruby

require 'pathname'

begin
  require 'inifile'
  require 'nokogiri'
rescue LoadError => e
  puts "This script requires nokogiri and inifile:"
  puts "  $ gem install nokogiri inifile"
  exit 1
end

CATVER_PATH = Pathname.new(ARGV[0] || '/home/pi/catver.ini')
CATVER = IniFile::load(CATVER_PATH)
CATEGORIES = CATVER['Category']

GAMELIST_DIR = Pathname.new('/home/pi/.emulationstation/gamelists/arcade')

GAMELIST_BACKUP_PATH  = GAMELIST_DIR.join("gamelist.xml.#{Time.now.to_i}.bak")
GAMELIST_CURRENT_PATH = GAMELIST_DIR.join('gamelist.xml')

# Backup current gamelist.xml.
GAMELIST_BACKUP_PATH.open('w') do |f|
  f.write(GAMELIST_CURRENT_PATH.read)
end

GAMELIST_XML = Nokogiri::XML(GAMELIST_CURRENT_PATH.read)

# Traverse the gamelist.
GAMELIST_XML.css('gameList > game').each do |g|
  path_node = g.at_css('path')
  basename = Pathname.new(path_node).basename('.zip').to_s

  genre_node = g.at_css('genre')
  current_genre = g.at_css('genre').content rescue "(none)"
  new_genre = CATEGORIES[basename]

  puts "GAME: #{basename}"

  # Do we need to do anything for this game?
  if new_genre == current_genre
    puts "  genre is up to date"
    next
  end

  # Do we have a genre to give it?
  unless new_genre && !new_genre.empty?
    puts "  no genre found in INI"
    next
  end

  # This game has a genre. Write it to the XML.
  # Create the node if it didn't exist before.
  if !genre_node
    genre_node = Nokogiri::XML::Node.new('genre', GAMELIST_XML)
    g.add_child(genre_node)
  end

  genre_node.content = new_genre

  puts "  Current genre: #{current_genre}"
  puts "  New genre:     #{new_genre}"
end

# Write out the XML.
GAMELIST_CURRENT_PATH.open('w') do |f|
  GAMELIST_XML.write_xml_to(f)
end

puts "...done."

Some notes:

• The script assumes that the category file lives at /home/pi/catver.ini. If it’s elsewhere, change the script or else specify the right path in the first argument: assign-categories ~/custom/path/catver.ini.
• You can re‐run this script whenever you add new games.
• The goal here is to keep each game’s genre harmonious with what catver.ini says it is. If you think, for instance, that Windjammers should belong to a category of your invention (like “Sports / Pong‐like”) and change the XML accordingly, then that change will get overwritten the next time you run this script. Beware.

Building your own game list

Finally: let’s liberate all your game metadata from its XML prison.

80% of the point of an arcade cabinet is that it gets used at parties. Don’t throw parties? Now you have a reason to throw parties. I wanted a way to say “here are the games I’ve got installed; let me know if you want me to add any.” So I wrote a script to build a simple web page from the metadata in my gamelist.xml files.

My own version has a few more bells and whistles, but I’ve made a simpler version that spits out a single HTML file that you can drop on any server you own. It uses CDN‐hosted jQuery and Bootstrap and fonts loaded from Google Fonts so that you don’t have to manage any local dependencies. You can click on any game name to get a pop‐over with its description.

The script

#!/usr/bin/env ruby

require 'nokogiri'
require 'optparse'
require 'pathname'

SYSTEMS = ARGV

NAME_MAP = {
  'arcade' => 'Arcade Games',
  'daphne' => 'Laserdisc Games'
}

GAMELIST_ROOT = Pathname.new('/home/pi/.emulationstation/gamelists/')

output = []

$options = {
  require: ['name', 'genre', 'developer']
}

opts = OptionParser.new do |o|
  o.banner = "Usage: make-game-list [options] [systems]"
  o.separator ""

  o.on('-r', '--require=FOO,BAR', "Skip games that lack any of these fields (default: name, genre, developer)") do |value|
    $options[:require] = value.split(',')
  end
end

begin
  opts.parse!
rescue OptionParser::InvalidArgument => e
  STDERR.puts("#{e.message}\n\n")
  STDERR.puts(opts)
  exit 1
end

def fails_requirements?(meta)
  $options[:require].any? { |k| meta[k.to_sym].nil? }
end

def html_for_game(game)
  id = File.basename( game.at_css('path').content, '.zip' )
  name = game.at_css('name').content
  date = game.at_css('releasedate').content rescue nil
  year = date.nil? ? nil : date[0..3]
  genre = game.at_css('genre').content rescue nil
  developer = game.at_css('developer').content rescue nil
  description = game.at_css('desc').content rescue nil

  return '' if fails_requirements?({
    name:      name,
    year:      year,
    genre:     genre,
    developer: developer
  })

%Q[
  <tr>
    <td data-game="#{id}" data-value="#{name}">
      <a href="#" class="game-link" data-toggle="popover" data-title="#{name}">#{name}</a>
      <div class="game-description">#{description}</div>
    </td>
    <td>#{year || '?'}</td>
    <td>#{genre || '?'}</td>
    <td>#{developer || '?'}</td>
  </tr>
]

end

def html_for_system(path)
  xml = Nokogiri::XML( path.open )
  system = path.dirname.basename.to_s

  games = xml.css('gameList > game')

  rows = games.map { |game| html_for_game(game) }

%Q(
<h2 class="system-title">#{NAME_MAP[system] || system}</h2>
<table class="table table-bordered table-collapsed table-striped sortable">
  <thead>
    <tr>
      <th>Game</th>
      <th>Year</th>
      <th>Genre</th>
      <th>Manufacturer</th>
    </tr>
  </thead>
  <tbody>
    #{rows}
  </tbody>
</table>
)
end

SYSTEMS.each do |system|
  path = GAMELIST_ROOT.join(system, 'gamelist.xml')
  output << html_for_system(path)
end

output = output.join("\n")

content = <<-HTML
<!DOCTYPE html>
<html>
  <head>
    <meta charset="utf-8">
    <title>Nostalgia-Tron Games List</title>

    <link rel="stylesheet" href="https://fonts.googleapis.com/css?family=News+Cycle:700|Oxygen">
    <link rel="stylesheet" href="https://maxcdn.bootstrapcdn.com/bootstrap/3.3.7/css/bootstrap.min.css">

    <script src="https://ajax.googleapis.com/ajax/libs/jquery/2.2.4/jquery.min.js"></script>
    <script src="https://maxcdn.bootstrapcdn.com/bootstrap/3.3.7/js/bootstrap.min.js"></script>

    <style type="text/css" media="screen">
      h1, h2, h3:not(.popover-title) {
        font-family: 'News Cycle', sans-serif;
      }

      h3.popover-title {
        font-weight: bold;
      }

      h2.system-title {
        margin: 2.5rem 0 2rem;
      }

      body {
        font-family: 'Oxygen', sans-serif;
        padding-top: 3rem;
        padding-bottom: 3rem;
      }

      .popover {
        font-family: 'Oxygen', sans-serif;
      }

      .game-description {
        display: none;
      }
    </style>
  </head>

  <body>

    <div class="container">
      #{output}
    </div>

    <script type="text/javascript">
      $(function () {
        // When we open a popover, hide any others that may be open.
        $('a.game-link').click(function (e) {
          e.preventDefault();
          var $others = $('[data-toggle="popover"]').not(e.target);
          $others.popover('hide');
          $(e.target).popover('toggle');
        });

        // Make popovers wider.
        $('a.game-link').on('show.bs.popover', function () {
          $(this).data("bs.popover").tip().css("max-width", "600px");
        });

        // Hide a popover whenever someone clicks off.
        $('body').click(function (e) {
          var $anchor = $(e.target).closest('a.game-link');
          var $popover = $(e.target).closest('.popover');
          if ($anchor.length > 0 || $popover.length > 0) return;
          $('[data-toggle="popover"]').popover('hide');
        });

        $('[data-toggle="popover"]').popover({
          html: true,
          trigger: 'manual',
          container: 'body',
          placement: 'auto bottom',
          content: function () {
            var text = $(this).closest('td').find('.game-description').text();
            text = "<p>" + text + "</p>";
            text = text.replace(/\\n\\s*\\n/g, "</p>\\n<p>");
            return text;
          }
        });
      });
    </script>
  </body>
</html>
HTML

puts content

Call it with a list of systems that you want to include on the page in the order they should be shown, and redirect STDOUT to a file to save it to disk — e.g., make-game-list arcade daphne > gamelist.html. If you want the heading above a particular system to say something nicer than its bare name, add a key‐value pair to NAME_MAP near the top. (For instance, without its entry in NAME_MAP, the daphne section would have a heading of “daphne” rather than “Laserdisc Games.”)

Once you’ve got an HTML file you can do whatever you please with it. My version of this script also scps the HTML file to andrewdupont.net so that I can update the game list with one command.

The code

Here’s a gist with all three scripts.

The end is in sight

One more installment and I’ll be done with this series and onto writing about a different stupid hardware project of mine. Next time I’ll finally cover a safe, idiot‐proof way to power up your monitor and marquee light when your Pi is on and power them off when your Pi is off.