How to encode quality PlayStation 1 video

Recently I've run across a couple of groups trying to replace the video of a Playstation 1 game. As we have to do the same thing with S.E. Lain, I thought I would share some insight as to how it might be done with the highest quality output.

Since there are many things that go into encoding mpeg video, that step is best left to programs that know how to juggle those options, such as ffmpeg. It also allows you to tweak those options, and use its advanced features to get the best results.

So here is how you do it, broken down into 6 ridiculously complicated steps.

1. Convert your video into 4:2:0 yuv4mpeg2, but using the PSX specific rgb->yuv conversion.
2. Feed the yuv4mpeg2 video into ffmpeg at an allowed fps and create a mpeg1 movie with only I-frames (-intra). This should be done at a variety of different quality levels (-qscale from 1 to n).
3. Parse the I-frames out of the movie and parse their macro-blocks
4. Convert the mpeg1 VLCs to PSX VLCs
5. Do this for every mpeg1 quality and pick the one that fits best within the amount of space available to the frame
6. Multiplex the frames and construct all the sectors

If the new video is mostly just the old video with some changes (e.g. subtitles) then quality can improved tremendously with this variation.

5. Use some method to determine which macro blocks to replace (manually picking them, or performing a diff on lossless video data, or a fuzzy diff on lossy video data)
6. Only replace those macro blocks with new ones of the same qscale
7. If the replaced data makes the frame too big, then either replace the entire frame with an ffmpeg created frame that will fit (qscale will probably be bigger), or remove some quality in the frame to make it smaller
8. Multiplex the frames and construct all the sectors

Alternatively have ffmpeg write an AVI using MJPG codec and parse the JPEG frames and convert the JPEG VLCs to PSX VLCs (I'm not sure how to get ffmpeg to produce quality variations with the MJPG codec).

zOMG psx.lain.pl is down!

And it has been for quite awhile. I thought it was just temporary, but finally learned yesterday that the good fellow that was nice enough to host the page has chosen to close the account. We are grateful for the time we could use the server.

Don't worry, we still have all data and work done so far. Unfortunately we're currently without a host, and the project is still stuck at around 14% completion.

I personally would like to thank everyone that has contributed thus far...

Hikari - Translated dialogs from video sequences to Polish.
phm - Extracted the media with jpsxdec and setup the wiki. Retranslated video sequences dialogs from Polish to English. Advertised for translators, and coordinated with their changes.
MercuryTW - Provided some translations from a translator of his own (Keigo). Edited/corrected translated files.
stalker-kun - Provided hosting for the wiki.
otakufish - Provided some translations.
toruvinn - Provided nice domain.
farhan - Ruthlessly proofread random translations, utilising his qualifications in the field of Applied Pedantry with Grammar Nazism.
fishy - Proofread translations, created the public project page.
arc - Provided some translations.
Jossos - Provided some translations.
utakata - corrected a Japanese transcription.
Najica - Provided some translations.
Quibbage - Created subtitle files for videos.

...and everyone else (sorry if I've missed your name here)--thank you all very much.

jPSXdec development has been pretty quiet as well. I should submit the bug that fails to decode some of the Lain videos, or just commit the fix. The next things to do with jPSXdec require some major changes, and things have been busy (my toon won't hit level 60 by itself ;). I'm still here and interested in the project and related topics. I don't plan on that changing anytime soon.

Sonic

AVI
Fixed the obvious problems with AVI writing, but still problems persist.
* My Windows Quicktime install must be bad because it crashes from like everything (but Mac Quicktime plays fine)
* My Linux mplayer must be bad because it plays all movies stretched, one way or another
* All programs on my lappy crash from my movies except Windows Media Player
* My Linux Totem still plays it with green garbage

So I think I may be doing MJPG slightly wrong, but I'm not sure what. Plus I finally figured out why I could never find any documentation on the MJPG codec: "there is no document that defines a single exact format that is universally recognized as a complete specification of “Motion JPEG” for use in all contexts" Wikipedia

Native decoder
Got the native decoder working great on Windows, Linux and Mac. CMake has made cross-platform building a breeze :D It's really exciting to see the decoding plow through dozens of frames per second.

Indeed, native decoding is fast...

Java is faster
The heck?? Contrary to what I thought I saw in my initial tests, my swift Java decoding module is significantly faster than using the native decoder. How? Not sure, but I suspect it's due to all the overhead needed when calling a JNI function (such as changing big-endian to little-endian). It already melts through 150 sectors/second like a hot knife through butter--but since it spends half its time with garbage collection, could it be faster with some object pooling?

v0.34

I just needed to get something for a public release. It was the re-implementation of the command-line that kept hanging me up (it was boring and I wanted to work on other things ;). I admit in my rush to just get something released, I sorta skimped on the code commenting. :P

Testing

Before releasing, I randomly tested jpsxdec on the dozen or so games that my friends let me borrow. As a result I threw in some last-minute changes to handle a couple new movie types. 'Last minute' means these may not even work right (I look forward to finding a way to save variable frame rate movies ^^).

I ran some quick tests on my Linux Mint machine. Compiz was often rendering the progress window without controls, so there was no way to close the program! And there's an annoying JFileChooser 'All Files' filter bug... :P (just wrote a work-around a moment ago).

My tests on a Mac were a little more promising. Rendering was slick--even looked better than Windows rendering.

After seeing how jpsxdec worked on Linux, I was worried about the quality of jpsxdec--then I saw iTunes for Windows and was feeling pretty good again.

AVI writing is faulty

Seems my AVI writer is getting something wrong because it is crashing or otherwise playing poorly on Linux by VLC ('this avi is broken'), mplayer (plays upside-down, stretched, or just crashes), Totem player (green gunk on the screen). On Windows and Mac, Quicktime is having problems (plays video too slowly or crashes). Real Player also crashes the good crash.

Speed

Since much of the core decoding work for jpsxdec is finished, I started looking for something a little more exciting to work on. So, following MrVacBob's suggestion, I began writing a crazy fast Java decoder. After doing everything I could think of, and examining the work of previous coders, I came to the conclusion that a real-time PSX Java decoder may be possible, but the resulting code would be so ugly, long, and annoying, it just isn't worth it. I hope to complete a faster Java decoder that'll cut the decoding time by maybe half, but also make a small, cross-platform native library will be fast enough for real-time playback.

My friends keep dropping games on me--more than I can keep up with. It takes so long to rip games that I went ahead and threw together raw cd reading on Windows. It's pretty buggy, and I'm sure I'm doing something wrong, but it mostly worked. If ever I can get it more stable, I'll include it. For my Linux coding friends, the JNI Java class is in the source if you feel like writing a Linux version of the lib :)

Licenses

The LGPL of the Swing libraries included with jpsxdec don't require that I distribute the source code of the libraries (I haven't changed them, and I am linking them separately). But since jpsxdec is GPL, does that require me to include the LGPL sources with the distribution? I really didn't want the hassle of it, but just to be safe I included the source with the pre-combiled binaries together in the zip files.

Incoming

Been wanting to get some stats on jpsxdec code for a long time (Google code doesn't have such features), finally ran across StatSVN and took a quick look at the results.


Now running over 20,000 lines of code, and almost 100 files. Compared to Oct 21, 2007's 7,000 lines, that's 13,000 LOC over 144 days, for a measly 90 lines per day.

Anyway, I'm guesstimating next week will be the next jpsxdec release. It will have some exciting features:

• Saving to AVI, both uncompressed and MJPG compressed your mileage may vary
• A very nice dialog box providing the many decoding options available it's arrite
• Final Fantasy 9 finally working properly done!
• Chrono Cross is also working! mostly done!
• Cleaned up command-line done!
• and lots of delicious code documentation :)

lain? lain!

The Serial Experiments Lain PSX Game Translation Project is really coming along. A few more wonderful translators have generously offered some help. We have roughly 7% of the game translated already.

Here are some exciting fruits of everyone's efforts so far (actual screenshots).

nihongo niban

Much awesomeness has transpired recently. But first, the dull ramblings of jpsxdec status.

Looked up how to encode uncompressed (RGB) AVI files. It's not too complicated--Microsoft and other sites have usable documentation. Even found a Java class to get me started. Of course if we're gonna write AVI files, we need to know the frame-rate. Just a simple 400 lines of code is all that took. Needed to also change the overall code design to a 'push' architecture with listeners. This had the additional bonus of pulling the actual file writing to the top-level interface--right where it belongs.

Overall things are looking pretty good. For the next release I want to add a new popup window showing the many decoding options.

Also in the future I want to allow for piping out uncompressed AVI to a program like ffmpeg or mencoder. For Windows I'll also include the infamous Video for Windows. Still need to add raw cd reading.

But while merrily carrying on with all this, out of nowhere came a fishy fellow with the powers of language transformation! He generously offered to spend some time with the S.E. Lain game. Sooo... efforts on jpsxdec are delayed as the Serial Experiments Lain PSX Game Translation Project commences (SELPGTP?).

But with 5.5 hours of audio, he has his work cut out for him. Do you share the powers of language transformation? We could really use your help.

On the technical side, we are working on adding the eventual translation into the game. And I present to you an actual screen shot based on an earlier idea.

Amalgamation

Been busy with a bunch of different aspects of jpsxdec. Only the GUI and Code Design will be in the next release (v0.31 today!). The rest will show up in releases after that.

GUI
Even though it's a pretty lame GUI now, I hope it will drop the bar of entry a little.


Code Design
I had difficulty deciding on highest-level interface between the GUI/CLI and the logic. So I set it aside and worked on other things.

PSX YUV->yuv4mpeg2 conversion matrix
This will allow jpsxdec to output more accurate .yuv image sequences.

Frame Rate Calculation
I thought it was a good idea to guess the exact FF7 frame rate algorithm via trial-and-error. 8 hours later I finally saw that FF7 frame lengths do actually follow a pattern, but a ridiculous one. I hoped it would come to a frame rate of exactly 15/100.1% (half the rate of NTSC). That is close, but there was no way I could guess the underlying algorithm of this...

    if (iMovieType == WACKED_NTSC1 && iThisFrame == 101)
        iThisFrameLength = 11;
    if (iMovieType == WACKED_NTSC2 && iThisFrame == 100)
        iThisFrameLength = 11;
    else if (iThisFrame == 200)
        iThisFrameLength = 11;
    else if (iThisFrame >= 203 && iThisFrame <= 298) {
        if (((iThisFrame - 203) % 4) == 0)
            iThisFrameLength = 11;
        else if (((iThisFrame - 203) % 4) == 1)
            iThisFrameLength = 9;
        else
            iThisFrameLength = 10;
    }
    else if (iThisFrame == 299)
        iThisFrameLength = 11;
    else if (iThisFrame >= 303 && iThisFrame <= 398) {
        if (((iThisFrame - 303) % 4) == 0)
            iThisFrameLength = 11;
        else if (((iThisFrame - 303) % 4) == 1)
            iThisFrameLength = 9;
        else
            iThisFrameLength = 10;
    } 
    else if (iThisFrame == 399)
        iThisFrameLength = 11;
    else if (iThisFrame >= 402 && iThisFrame < 497) {
        if (((iThisFrame - 402) % 4) == 0)
            iThisFrameLength = 11;
        else if (((iThisFrame - 402) % 4) == 2)
            iThisFrameLength = 9;
        else
            iThisFrameLength = 10;
    }
    else if (iThisFrame == 498)
        iThisFrameLength = 11;
    else if (iThisFrame == 501)
        iThisFrameLength = 11;
    else if (iThisFrame >= 504 && iThisFrame <= 600) {
        if (((iThisFrame - 504) % 4) == 0)
            iThisFrameLength = 9;
        else if (((iThisFrame - 504) % 4) == 2)
            iThisFrameLength = 11;
        else
            iThisFrameLength = 10;
    } 
    else
        iThisFrameLength = 10;

Raw CD reading (Windows)
Made a native program to read raw CD sectors via DeviceIoControl. It's a simple interactive prompt that accepts commands to open/close a CD and read sectors. I even setup a Java program to communicate with the native program via stdin/out. Unfortunately it was slow, and was very vulnerable to deadlock while reading from stdin. I also started adding ASPI for raw CD reading, but there seems to be a lot more to using that API.

Encoding via VFW (Video for Windows)
I worked out a little program to convert a series of raw image and audio data into an AVI, using the oh-so-familiar Video for Windows popup. I wonder if it can handle accepting all the audio at once, then all the video at once....


JNI
Even though I said it would be a pain in the butt, I guess it's really not so bad. It's kinda like translating VB6 COM code into C++. This will likely speed up raw CD reading, and save me from communicating via stdin/out.

Code Design
Finally got back to this again. Since no solution was particularly attractive, I went with the simplest one. This will only use one optional callback listener (for progress status).

However, I have begun to think that the 'pull' architecture is not the best approach to begin with. A 'push' architecture is more applicable to this situation: as sectors are read from the disc, they need to be 'pushed' into the appropriate streams. Listeners can be added to collect the streams of interest. If jpsxdec was to ever play media in real time (not likely), this would be the necessary design. In any case, the 'push' approach would make decoding a little faster because there is less seek time.

PSX YCbCr to yuv4mpeg2

PSX YCbCr

Given Y, Cb, and Cr color range of [-128, 127].

                              PSX
[ 1   0        1.402  ]   [ Y + 128 ]    [ r ]
[ 1  -0.3437  -0.7143 ] * [ Cb      ] =  [ g ]
[ 1   1.772    0      ]   [ Cr      ]    [ b ]

yuv4mpeg2 (i.e. Rec.601) YCbCr

Given Y color range of [16, 235] and Cb,Cr color range of [16, 240].

                              Rec.601
[ 1.164   0       1.59  ]   [ Y  - 16  ]     [ r ]
[ 1.164  -0.391  -0.813 ] * [ Cb - 128 ]  =  [ g ]
[ 1.164   2.018   0     ]   [ Cr - 128 ]     [ b ]

To convert a PSX YCbCr color to a Rec.601 YCbCr color:

[ 1  -3415973/13224846875  1242172/13224846875 ]
[ 0   105814197/105798775      -5608/105798775 ]
[ 0       19492/105798775  105791687/105798775 ]

Answer:  yuv4mpeg2^-1 * PSX =
[ 1  3415973/13225888625  -1242172/13225888625 ]
[ 0  105791687/105807109        5608/105807109 ]
[ 0     -19492/105807109   105814197/105807109 ]

Rec601_YCbCr = 
                                     PSX
                                     [Y + 128]   [ 16]
     Rec601_Matrix^-1 * PSX_Matrix * [Cb     ] + [128]
                                     [Cr     ]   [128]

Where

Rec601_Matrix^-1 * PSX_Matrix =

     [250/291 -488509/2660418030 -82738/1330209015]
     [0          4014411/4571165       164/4571165]
     [0            3673/27426990   8031459/9142330]

Thanks to toruvinn for refreshing me on matrix math, and this site and Maxima for saving me from lots of raw calculations.

Update 27Feb2010: Fixed.
Update 23Mar2010: Really fixed.

HATSUHINODE

Hope all had a pleasant new year. As we enter 2008, it is interesting to note that this year marks the 10th anniversary of the Serial Experiments Lain anime series, along with the Playstation game.

Now that holidays are over, I finally managed to release the latest compiled executable on the download page. No changes to the program interface (same wonderful command-line that you're used to ;), but it has all the latest internal goodness that I hope actually works--most notable is it now [edit] mostly supports FF8 (it would help to fully test it before making such calims :P).