Ever since Apple TV came into my life, I have had a dream. The dream, simply stated, has been to have every episode that I own of "The Simpsons" available on my Apple TV, for random access viewing as I please. And over the last year, I have poked on various aspects of this project, gone down a bunch of dark alleys, performed research, and overcame a ton of obstacles. But I'm proud to say that as the new year dawned, I finally hunkered down and actually did it, the final act being this piece of documentation that I'm writing now.
The Motivation:
I own seasons 1 through 9 of "The Simpsons" on DVD, buying them diligently as they became available, stopping only because I found I wasn't watching them. As it turns out, while I dearly love being able to cue up full, uncut versions of any episode that I please -- I don't like having to go and find the proper disc, put it in the player, deal with the unskippable legal notices and menus, just to be able to watch a single episode. I know that it isn't all that much work, but I am so very lazy, and even those small speedbumps were enough of a hurdle to stop my Simpsons viewing entirely. Thus, I knew that I needed these episodes to be free of physical DVDs -- in that way, I could just plop down on the cough, grab the remote, and be seconds away from Simpsons bliss.
The obstacles:
If you ignore the grey-area of ripping CSS-encrypted DVDs, the problems with this project centered on the number of individual Simpsons episodes that I was targeting. At 203 episodes, I knew that it was going to be a lot of work just to rip and encode each episode in Handbrake, but I was really dreading all of the work that came after. In order of iTunes to properly recognize each episode, I would have to manually enter in the following metadata:
- Episode title
- Episode number (so that I could watch them in order if I wished)
- Episode type (set to TV show, so that iTunes would recognize it as such)
- Episode season
And to be pedantic, although it wasn't strictly needed by iTunes, I wanted to have the following information in each episode file:
- The writer(s)
- The director(s)
- The episode description - I tend to know episodes more by description than by title
- The Episode ID
- Year of production
- Proper album art
And I was just dreading manually entering in all of that information. I spent some time experimenting programs like MetaX -- but I couldn't find anything that met my needs. So, I fired up my text editor, and wrote some code of my own to do this, which I'll cover in more detail after the jump.
The other set of obstacles was in determining just the right way to convert each episode from it's DVD version, to a file on my hard drive, so that I could get it into iTunes (and by extension, Apple TV). I cover this as well in the technical section after the jump.
The Payoff:
And without further ado, here is the payoff:
The first 9 seasons of "The Simpsons", in iTunes.
I have found that having random access to this many Simpsons episodes really has been transformative. Whenever I have 22 minutes to spare, I can find a nice escape and watch an episode. This includes when I'm on the go -- I now always have 10 Simspons episodes in my pocket, ready for viewing at a moments notice:
Watching "Itchy & Scratchy Land" on my first-generation iPhone
The put it bluntly, I am very, very happy with the results of my new Simpsons diet! It was a lot of work, but the payoff has definitely been worth it. If you're interested in more of the nitty-gritty technincal details, you can continue reading after the break.
The Nitty-Gritty Details:
The other aspect of this project that I spent a long time on was deciding the encoding settings in Handbrake. I knew that I had to encode using the h.264 video codec in an mp4 container, so that the file would be view-able on Apple TV. But I also wanted the episodes to be viewable on my iPhone, and I didn't want the file size to be very big (since there are over 200 of these suckers), and I wanted the quality to look great -- whether I was watching on a 46" TV or a 3.5" phone.
So, I experimented with pretty much every "preset" and setting in Handbrake, trying to arrive at the best mix of settings to achieve all of the factors that I set down above. I found that the "Animation" preset did almost everything that I wanted -- the encoded episode looked great, the file size was small (~170MB per episode), and it was pretty fast to encode. There was just one small problem -- I couldn't get files encoded with the "Animation" preset to play properly on Apple TV or on my iPhone, and iTunes sometimes crashed when playing them as well.
Ripping with the "Apple TV" preset made files that worked on Apple TV, but they were huge, and didn't work on iPhone. Ripping with the "iPhone" preset made files that did everything I wanted -- except that they didn't look so great when played on a TV.
Luckily, with the advent of Handbrake 0.9.3, a new profile showed up, called "Universal", which did almost everything that I wanted. It produces files that look fantastic on both my iPhone and Apple TV, and also work well in iTunes. The only problem is that Handbrake is really slow to encode with the Universal preset, and the resulting files are pretty big (~350MB per episode). I decided to trade time and disk space so that I could have better quality. I assume that my devices will have more storage space in the future -- but I don't ever want to have to re-encode all of these episodes again!
After deciding upon my Handbrake settings, and I made sure that I included the unique "Episode ID" somewhere in the filename for each episode. Then I wrote two scripts -- one that scraped Wikipedia for Simpsons episode metadata, and a second that took said metadata, and wrote it into each file in an iTunes-compatible way. Details of these scripts follow.
mk_simpsons_episode_db.pl:
The mk_simpsons_episode_db.pl script scrapes the individual season pages from Wikipedia, extracting the episode title, description, writer(s), director(s), year, season, and episode number. The results of this extraction are written to a file on-disk, in YAML format. I did this so that this metadata could be in a format easily read by a computer, and thus be useful for other projects. If you don't care a whit about iTunes or iPhones, but have your eye on some other Simpsons-related project, you can download the YAML file, and write your own code to take that metadata and do something else with it. Plus, I have been wanting to do something with YAML for like, ever.
The other trick that this script used was the HTML::TableExtract module from CPAN. After wasting some time screwing around with parsing Wikipedia's HTML by hand (which I know, you should never do), I stumbled upon this CPAN module which does exactly what I want -- given a mess of HTML, this module will pull out the matching tables, and make it easy to iterate over them row-by-row. Using this module made the task of parsing out the episode information from the Wikipedia page almost trivial.
Running this script requires the LWP, YAML::XS, and aforementioned HTML::TableExtract perl modules, which can all be found on CPAN. Because getting this all going isn't exactly trivial for the uninitiated, I am also putting the YAML file that I generated online. It only covers seasons 1 through 11, but you should get the idea.
In terms of running mk_simpsons_episode_db.pl itself, simply invoke it with the '--help' argument, and you'll get a list of all the parameters, as well as some execution examples.
add_metadata_to_simpsons_episodes.pl:
Next, with my YAML file in place, I had to write a script that given an episode of "The Simposons", write the proper metadata into the file. The program that I produced, add_metadata_to_simpsons_episodes.pl, is in reality just a fancy front-end to an external program, AtomicParsley, which does all of the heavy-lifting of writing the metadata to the video file.
This script understands the Simpsons Episode ID scheme, so it looks for files that it can extract an Episode ID from, and looks up the Episode ID in the YAML database. If it finds a match, it constructs a list of arguments to AtomicParsely, and invokes it. A new file will be produced, which will be the exact same video and audio as the input, but with the proper metadata inserted. Then the resulting video files can be dragged and dropped onto iTunes, in order to add it to your library, and from there synced to your devices.
The add_metadata_to_simpsons_episodes.pl script only requires the YAML::XS perl module, so it should be fairly easy to satisfy the pre-requisites. Invoking the script with the '--help' argument will produce a list of all the parameters, as well as some execution examples.
Putting It All Together:
My basic workflow for processing Simpsons episodes was:
- Use mk_simpsons_episode_db.pl to produce a YAML file for the Sesasons that I knew I was going to encode.
- Use Handbrake to encode an entire season of "The Simpsons", into it's own directory. Each file was named with the Episode ID (which you can find in the booklet that comes with the DVD set -- except for Season 1, which doesn't include this information).
- Use add_metadata_to_simpsons_episodes.pl, with the '--dir' option, to add metadata to each file in the Season. Note that this step will take awhile, because by default AtomicParsley makes a copy of each (~350MB) file as it goes.
- Using the Finder, drag all of the processed files (they'll be postfixed with the string "-temp-", followed by a random number) into iTunes. You can sort by "Date Modified" in Finder to make it easy to grab all of the processed files in one go.
- Go back to step #2, and repeat for the next season.
- Enjoy lots of Simpsons on any device that you choose!