XMODITS a Year in Review
Contents
It has been over year since I made the first commit to my first big project (Jun 20 2022).
I feel incredibly proud of the work I put in, and I’m glad that a lot of people found it useful. ^_^
What is it XMODITS?⌗
XMODITS is a tool made to bulk extract samples from tracker modules with convenience.
What is a tracker module?
A tracker module is a family of music formats that store arrangements of notes (like MIDI) with embedded audio samples. These sequences are used to instruct how those samples should be played out. Think of it like an orchestra stored in a file.
The samples are typically stored as a PCM + some metadata. With that in mind, they’re almost identical to a .wav, however, the way that metadata is stored varies across formats.
Programs like OpenMPT have no problem dealing with these formats, and can, in fact, be used to export those samples to something like a wav.
This is fine for a couple of modules, but gets unmanageable when you have thousands of them. They’re simply not made to do that sort of thing (maybe in the future?).
What my program does is automate that process with convenience, speed and a few extra features.
So what motivated me to make this tool in the first place?
The Seeds⌗
It started back in 2020 - during the height of the pandemic - when I played Deus Ex for the first time. One level that stuck with me was “NYC Streets”. I fell in love with the atmosphere and most importantly, the music.
Wanting to listen to the music outside of gaming, I looked through the game files. After a bit of digging, I discovered an interesting file called NYC_Streets.umx (link).
Instinctively, I opened it with VLC and… It plays!
Gosh, VLC can play anything!
I called it a day and never thought much about it.
Fast forward 2 years later, I wanted to recreate the exact soundtrack in LMMS, a free digital audio workstation.
Deus Ex’s soundtrack does sound like it was made with a tracker, which, after a simple search confirms it.
Great! I could use the original samples!
Ok, let’s load NYC_Streets.umx in Schism Tracker and… It doesn’t work.
Oh.
Looks like Schism Tracker can’t open umx files.
The umx format is technically a container format, and Deus Ex’s soundtrack uses Impulse Tracker - which can be opened by Schism.
I wrote a simple program to strip the umx header revealing the inner module. This allowed me to load the modules in schism tracker and export the samples manually.
The program was simple. Impulse tracker files always start with “IMPM”, so it’s just a matter of locating where that is and truncate anything before it.
And then I found that OpenMPT can open umx files just fine.
Oh well..
I got the samples I needed, but exporting them manually was arduous.
Why don’t I make a program that does this?
Writing the Program⌗
I chose to write my project in Rust, because it’s the language I feel the most comfortable writing in for this project.
Comparing my past projects written in Rust and Python. I’ve spent less time cursing when writing in Rust.
I also wanted to get better at the language.
Choosing a Name⌗
Coming up with a good name is pretty hard.
Originally, the project was given a placeholder name “tracker dump”, until I gave it a decent name 26 commits later.
Scrapped Ideas:
- Tracker Dumper
- XMODSD - eXtreme MODule Sample Dumper
- XMODITS3
As you can see, “xmodits3” was the ‘winner’.
Why is it called “xmodits”?⌗
I came up with the name by listing the formats I wanted to rip:
- .it
- .xm
- .mod
- .s3m
- .umx
Then I combined them to make something readable, and got this:
UMX XM MOD IT S3M
UMX
XM
MOD IT S3M
XMOD IT S
XMODITS
Personally, I think it’s a decent name - short, snappy and easy to pronounce.
The Core⌗
The Impulse Tracker (.it) was the first format I worked on. This was mainly due to it (no pun intended) being exclusively used in Deus Ex’s soundtrack.
The it format was quite nice to implement. There was an abundance of documentation covering the format, and they were quite easy to understand.
The first milestone I felt proud of was being able to extract compressed samples. I don’t remember how long it took, but it felt like forever implementing the decompression routine. (Thank you itsex.c)
After successfully extracting the samples for the first time, I implemented S3M, and MOD formats. (XM and UMX was implemented a little later after publishing)
Reading the source code of Schism, Milky Tracker, ModPlug and OpenMPT despite not knowing C/C++ at the time was, to say the least, an interesting experience.
ModPlug has the nicest looking codebase.
My only gripe with OpenMPT is that it extends existing formats with hacks. So I needed to resort to reading its source code.
Out of all the formats implemented, the .xm format was the hardest to do.
Its binary structure was incredibly weird from a parsing point of view and its documentation was scarce.
Impulse Tracker was the most fun.
Now that I’ve implemented the core, I need to make an interface.
The CLI⌗
This version was super basic. All it did was extract the samples to a self contained folder. The way samples were named was hard coded.
I was satisfied with the cli, but using it on Windows felt off. For a predominantly graphical workflow, using the command prompt felt like a chore use.
At that stage I wasn’t sure how to make a GUI for it, so I made a basic version that lets you drag and drop trackers onto of the binary.
The first release:
██╗░░██╗███╗░░░███╗░█████╗░██████╗░██╗████████╗░██████╗
╚██╗██╔╝████╗░████║██╔══██╗██╔══██╗██║╚══██╔══╝██╔════╝
░╚███╔╝░██╔████╔██║██║░░██║██║░░██║██║░░░██║░░░╚█████╗░
░██╔██╗░██║╚██╔╝██║██║░░██║██║░░██║██║░░░██║░░░░╚═══██╗
██╔╝╚██╗██║░╚═╝░██║╚█████╔╝██████╔╝██║░░░██║░░░██████╔╝
╚═╝░░╚═╝╚═╝░░░░░╚═╝░╚════╝░╚═════╝░╚═╝░░░╚═╝░░░╚═════╝░-0.8.8
By B0ney - https://github.com/B0ney
USAGE:
xmodits <module>... [destination folder]
FLAGS:
-h, --help Prints help information
-v, --version Prints version
EXAMPLES:
xmodits song1.s3m
xmodits song1.s3m ~/Downloads/
xmodits song1.s3m song2.it
xmodits song1.s3m song2.it ~/Downloads/
At that point, I considered the project done. I wasn’t really interested in working on it anymore (apart from adding XM an UMX support).
Porting XMODITS to a Python Library⌗
One project I discovered on a discord server was called DawVert, (originally called Daw Conversion Tools).
It’s a tool that can convert different music project formats (such as impulse tracker) into other project formats like FL Studio or LMMS.
Where does xmodits fit in?
After using DawVert to convert Impulse Tracker modules to LMMS’ .mmp format, I noticed that it didn’t include the samples. So I used xmodits to extract them. Since it was also written in python, this gave me the idea to make a python library with xmodits.
Rust has a really convenient library called Maturin that lets you write Python libraries in Rust.
In less than a week, xmodits_py was born!
While making the library, I quickly noticed that DawVert’s method of naming samples was vastly different from XMODITS’, so I added the ability to customise how samples are named.
Realising the potential, I’ve also implemented this feature back into the original project. I got two birds with one stone :D
The GUI⌗
A month after publishing xmodits_py, I wanted to finally tackle writing a GUI for it.
Making the GUI version of XMODITS has been considered since it was first published, but was put on the back-burner due to its difficulty.
I had two main candidates:
After dabbling with egui, I went with iced. Primarily because of its styling system and Elm inspired architecture made the most sense to me.
Early Prototypes
First Prototype - 30th October
Prototype 2 - 3rd November
Prototype 6 - 4th November
Prototype 8 - 6th November
Prototype 10 - 9th December
Prototype 12 - 14th December
After roughly 2 months of work, I published version 0.9.8:
My Thoughts After Writing the GUI.⌗
Overall, it’s not hard, it’s just that there is SO MUCH code to write.
I used a tool called tokei to count how many lines of code the gui has:
cd ./xmodits-0.9.8/app/gui
tokei
===============================================================================
Language Files Lines Code Comments Blanks
===============================================================================
Markdown 1 15 0 8 7
SVG 6 11 11 0 0
TOML 1 56 47 3 6
-------------------------------------------------------------------------------
Rust 29 2478 2013 240 225
|- Markdown 3 14 0 12 2
(Total) 2492 2013 252 227
===============================================================================
Total 39 2560 2071 251 238
===============================================================================
2560 lines is quite a lot of code for the gui. (There’re a lot more lines for later versions too!)
The most challenging part of this was bridging the logic and gui together.
Furthermore, you’ll need a decent computer with more than 4 cores if you want to be productive. Rust is infamous for its slow compile times. Clean compiling xmodits took 2 minutes on my main system and, for a quick laugh, took 15 minutes on my old dual core Pentium. But after compiling everything, it only takes a few seconds to see the changes I make.
Rewriting the Core⌗
After publishing 0.9.8 I took a break.
I received a couple of requests, and realised how difficult it was to add new features with the old code.
So I began rewriting it.
I took the time to note down the overall design of the core library. One of the goals is to make the code easy to modify, and less about writing “perfect” code.
The rewrite took about 2-3 months. And I must say, I am incredibly pleased with how it turned out; the new code is, *chef’s kiss*, several orders of magnitude better than the old one. :D
If you’re curious, you can browse the old code and new one (at the time of writing)
- (old) xmodits-lib
- (new) xmodits-lib
Making the Program Faster⌗
Version 0.9.8 was single threaded, so it was pretty slow. Not to mention the additional overhead introduced by the GUI.
I needed to speed things up.
At one point I experimented with ripping trackers asynchronously. It was pretty slow and consumed about 1.2GB of memory. Safe to say I was trying to solve the wrong problem and unsurprisingly, got the wrong results.
The optimal approach was parallelism. With my 12 core laptop, it took xmodits 12 seconds to rip 2700 trackers (with Windows Defender temporarily disabled). That’s 225 trackers per second per core. Originally, this took ~1 minute.
My obsession with benchmarking ultimately made me add a feature to the GUI so that it shows how long it took to extract the samples:
While making the program faster, I also made 2 major memory optimisations to the program:
-
The first optimisation targets how xmodits traverses folders. Folder traversal is quite literally a can of worms, you don’t know how many files you’ll encounter. This blows up the deeper you go. My solution was to store the file entries to a temporary file.
-
The second optimisation targets how xmodits handles errors. When xmodits receives an error from the workers responsible for ripping trackers, they’re stored in memory. This can be a problem when there’s thousands of errors. I solved this by streaming the errors to a file when there’s too many of them to store in memory.
With these improvements, I published version 0.10.0.
Closing Thoughts⌗
It was super fun to develop! I cannot underestimate how many things I’ve learned from this project. It almost feels like cheating.
XMODITS is my first big project. With this blog, I hope to look back and smile at what probably is one of the many important moments in my programming journey.
Thanks for reading!