I moved the shader compiling outside of the game. Previously we'd compile them in process - which worked but it meant that the entire process froze up while it was doing it. Which was not good.
I also tweaked things so that reloading the shaders is a bit more reliable. The engine does have dynamic shader compiling, which compiles just the combos you're using.. which is really useful. Unfortunately it isn't that stable, so I've bypassed all that stuff until we have the stamina to work through it and make it reliable.
Shader compiling is a particular pain point right now due to the huge amount of combos they generate. It's something we really need to think about and experiment with. Why have the shaders got so many combos, are they needed, do modern gpus branch efficiently enough for us to avoid them, do render states need combos. It's all stuff we're going to think about at some point.