Metal Shading Language Compiler
Background
When I first met Johan, and he initially hired me to work on Fugl, before I became a partner on the game, he had the initial version running on iOS using Metal and MSL. I did the initial port to OpenGL, and made the game run first on Windows and then in VR with Oculus. Later on Johan continued the work on Metal, as it was quite performant, had a more modern programming interface than OpenGL, and is less complicated to setup for error handling and debugging. Naively, I said, “Ok, don’t worry, I’ll port it over to Windows”. I had already done the initial port of Fugl from iOS to Desktop (Windows / Mac) and Oculus VR, so with typical programmer hybris I jumped at the challenge. However, the shaders and the code were a lot more complicated at this point, than the first time around. It took me quite a while to do this port to OpenGL, and as Johan continued to work on the Mac, keeping track of the changes became quite time consuming. So, I did what any sensible programmer would do, I tried to automate the task.
Previous Work
There is a lot of tools already that can convert between shader languages, such as Khronos' SPIRV-Cross. However, while there are lot of projects that can convert to Metal and msl, I havent' been able to find one that allows conversion from msl.
Technical Details
Msl2glsl has been built using all the methods I remember from my compiler course at uni 20 years ago. Of course, it turns out I had forgotten most of it, but it was interesting relearning it. The compiler currently only builds on Windows and Linux, and requires Flex and Bison. On Windows, it currently requires WinflexBison and Visual Studio. I am using Meson as my preferred build system on Linux.
Like many automatic code generators, Msl2glsl does produce code that is overly verbose, and as such the code generated is not suitably for production. Given enough time, it is my belief that a human will nearly always be able to produce faster code than a machine, by hand-optimising. However, for Fugl I use a 4 step approach:
- Convert MSL to GLSL. Run Msl2glsl to extract glsl shaders from a msl shader file
- Convert GLSL to SPIRV. Run glslangValidator to convert glsl to spirv
- Optimise SPIRV Code. Run spirv-opt on each shader spirv file
- Convert SPIRV to GLSL. Run spirv-cross to convert spirv files backto glsl
These 4 steps are packaged into 2 different python scripts, both of which can be found in a different github repo here.:
- The first script is just a little python wrapper msl2glsl.exe that takes msl shader file as input and copies the extracted glsl files to an output directory.
- The second script is conceptually independent from msl2glsl and simply runs step 2-4 above, by converting glsl to spirv, optimising it and converting it back again.
What the compiler doesn’t do
The compiler doesn’t generate any of the cpu side code, it only helps with generating shaders. On the cpu side, the user still needs to write a fully fledged OpenGL renderer; loading models, generate VAOs, bind textures, compile shaders, upload uniform buffers, and so.
Future Work
While, I suspect for my part, the lifetime of Msl2glsl will closely linked with that of Fugl, we live in a time where products need to be constantly updated, or they will eventually dissappear into the ether of the internet. As such, I suspect Msl2glsl will kept up-to-date for quite a while. A full list of future improvements can be found on the repository, though the primary limitation at the moment, is that the compiler really only supports a c-like subset of MSL, plus the templated samplers, textures, etc. The lexer file should really be replaced with one based on c++14, which is what Msl is officially based on.
Repo
The code for the msl2glsl compiler can be found here on Github.