Reverse engineering 3D Movie Maker - Part 5

• January 26, 2021
• blog  
• Reverse Engineering  
• 3dmm  

A while ago, I started reverse engineering Microsoft 3D Movie Maker to understand how it works and to develop my game reversing skills. This blog series is about my adventures in reversing 3D Movie Maker and some of the interesting things I learnt along the way.

Previously, on “Reverse engineering 3D Movie Maker”:

• I recovered the C++ class hierarchy by reversing the custom runtime type identification system and wrote a Ghidra script to automate the process.
• I reverse engineered the message handling system and found an Easter egg that had gone unnoticed for about 20 years.
• I reverse engineered the scripting engine and wrote my own disassembler to disassemble the scripts that drive the user interface.
• I wrote my own assembler for the scripting engine and patched vtables to re-enable menu support.

As I continued to reverse engineer 3D Movie Maker, I kept finding pieces of unused functionality in the binary. These features provide some insight into design decisions that were made during the development of 3DMM, Creative Writer 2 and other applications using the same engine.

File format curiosities

One of the first things I looked at when I started reversing 3DMM was the file format. To recap, 3DMM uses a custom file format called “chunky” files which stores chunks of data that are used to serialize and deserialize objects. Chunks have a chunk tag, chunk number, the chunk data, and a list of child chunk references. Chunky files also have an index which makes it quick to find a chunk by tag/number, and to find a chunk’s children. To parse a 3DMM file, you need to be able to parse both the chunky file format and the structure of the individual chunks that make up the data. Foone Turing created a wiki that lists some of the chunk types used in 3DMM. There is also a patent that describes the data structures used by the file format.

While looking at the chunky files in a hex editor, I kept seeing the magic bytes 01 00 03 03 or 01 00 05 05 in the file header, the chunk indexes and the chunk data. I was curious to know what these bytes were used for. I figured that if these bytes were being checked by the program, there would be a compare instruction with the value as an operand, so I decided to search for any instructions with these magic numbers as operands. This is easy to do in Ghidra using the Scalar Search. Sure enough, searching for 16-bit parts of the magic values found a bunch of CMP instructions in the middle of functions that appeared to perform serialization and deserialization of chunk data.

A common pattern I noticed in deserialization functions was that the first WORD of the data would be read and compared to 01 00. If the value is not equal, a function is called that goes through the serialized data and swaps the endianness of all of the embedded offsets in the serialized data. I thought this was interesting, as 3DMM has only ever been released to run on Intel x86 systems which are little endian.

The 03 03 and 05 05 bytes show up inside string deserialization code, and are used to identify the character set used to store embedded strings. The 03 03 value represents the default Windows ANSI codepage, and the 05 05 represents the Unicode UCS2 encoding used by the Japanese release. The string deserialization code will convert Unicode strings to ANSI using the standard Win32 WideCharToMultiByte function.

The string conversion code also contained string types 02 02 and 04 04 which I hadn’t seen used in any of the data files I looked at. The 04 04 string type was easily identified as the big endian version of UCS2. The characters are handled the same way as the little endian Unicode characters, just with an extra call to a function to swap the endianness of WORDs in the buffer. The 02 02 type appeared to be handled by the codepage string handler, but with a small change: any characters above 0x7F were converted using a lookup table. The lookup table didn’t look familiar to me, so I Googled some of the bytes. This led me to some sample code for converting characters from the MacRoman character set, the character set used by the Macintosh.

This was an interesting find: it suggests that the application engine was designed with cross-platform compatibility in mind. This would make sense: in the mid-90s most multimedia applications were released for both Windows and Macintosh. The 68k and PowerPC processors used in mid-90s Macintoshes are both big endian. Microsoft released the original Creative Writer and Fine Artist products for both 16-bit Windows and the 68k Macintosh. If they had expected to use this engine with a bunch of products it would make sense to design it to support cross-platform releases.

As far as I know, there was no Macintosh version of 3D Movie Maker. I imagine porting it would be non-trivial as the Argonaut BRender 3D rendering engine used by the application contains hand-rolled x86 assembly language code, so either Argonaut would have to port BRender to 68k/PowerPC or Microsoft would have to use a different rendering engine. Perhaps slightly more likely is a Macintosh version of Creative Writer 2, given the original Creative Writer was released for both Windows and Macintosh - but as far as I know Creative Writer 2 for Macintosh doesn’t exist.

Debugging window messages

In Windows, when you register a new window class you specify a bunch of options including a pointer to a function in your program called the window procedure. This function is called whenever a message is sent to instances of your window class. You can choose to handle messages or pass them to the default window procedure. There are a lot of standard messages but you can also define your own custom messages. This is a quick and easy way to implement basic inter-process communication. To send a message to an application you just get a handle to the application’s window and call SendMessage to send a message.

The window procedure for 3DMM is implemented as virtual functions on the APP and APPB classes. There are several functions that handle window messages. If the first function can’t handle a specific type of message, it will chain to the next function. If none of the functions can handle the message, it is dispatched to Windows’ default message handler.

Most of the message handlers aren’t very interesting, and are the usual things you would expect from the window procedure: for example, if you send a WM_PAINT message the window is repainted. There are some custom messages that are interesting though. It appears there are some debugging messages that allow an external process to query the app engine for the state of graphical objects (GOBs). These messages can be used to get the type tag of a GOB, get the GOB’s parent, first child, or next sibling ID, and check if a GOB is a subclass of a particular class.

These messages can be used to display a list of GOBs that are currently rendered by the engine. I wrote a Python script that starts at the root GOB, which always has a fixed ID, and recursively visits all of the child GOBs to produce an object tree. If you run the script while the main menu is open, it shows all of the HBTN objects that represent the menu items:

There are also messages for querying app properties, getting the current cursor ID, seeking to a position in a movie, and changing the global clock’s time scale. Changing the global clock is fun as it affects not just the framerate of your movies but every animation that isn’t a pre-rendered AVI cutscene. It’s like overclocking the game!

I would guess these messages were added to facilitate automated testing or debugging of the application. The 3DMM team used automated testing during the product’s development, so I guess these messages could have been used by an external test tool to validate the state of the UI.

Multiple document support

Another interesting find in the window procedure was multiple document interface (MDI) support. MDI is a Windows feature that lets you create child windows under your main application window for each document that the user has opened. This type of interface has mostly been replaced with tabs in modern applications.

Both 3DMM and CW2 are designed as single-document interface (SDI) applications - you can only watch or edit one movie at a time, or edit one document at a time. But, for whatever reason, both executables have a bunch of code that implements all of the scaffolding required for an MDI user interface. This code is never used in the application. I wondered if there was a way to enable any of the MDI code.

While playing around with the unused menu support, I started looking at any other messages that could be triggered from a menu. Most of the app engine’s messages cannot be used with menus, as there is a callback that has to return a specific value for the menu item to be enabled. This callback could be patched to always return the specific value that says “enable this menu item”, but I was more interested in the messages that already had callbacks that resulted in the menu items being enabled.

Looking through the message map for the APPB class I found a message handler which would eventually send a WM_MDIACTIVATE message. This message is used to activate an MDI client window. The app doesn’t create any MDI client windows, so I figured that maybe some of the code around it might create the window before activating it. The rest of the code appeared to be calling into the global clipboard object and reading clipboard info.

Rather than try and understand a bunch of clipboard viewer code that I wasn’t particularly interested in, I decided to just patch the binary to enable menu support, add a menu item that would send message 0x7A, run it, and see what happens. So I did that, and 3DMM switched into MDI mode:

It looks like the developers created a clipboard viewer in order to test the MDI features of the engine. The edit boxes and scroll bars are functional, and the text editor can be split by dragging the black rectangle next to the scroll bar buttons. The rest of the application continues to run but you can’t easily interact with it.

The end… or is it?

I feel like I’m running out of cool stuff to blog about from my adventures in reverse engineering 3DMM, so I’m going to take a break from blogging about it for now. I’m not going to stop reverse engineering 3DMM though. If there is one thing I have learnt from reversing 3DMM is that it has hidden depths: I didn’t expect a relatively obscure children’s program to have its own custom scripting language, C++ runtime type identification or multiple Easter eggs. I’ll publish another post if I find anything else that I think is interesting.

If you’ve made it this far, thanks for reading! I’m hoping to post some more blogs soon about other rabbit holes I went down while reverse engineering 3DMM and a few other old, obscure applications. If you want to be notified of these posts, you can follow me on Twitter or subscribe to my RSS feed.