Since we were, and still are unable to make use of the Navmesh due to it just refusing to work on bigger maps, we were forced to handle NPCs as if they were bots, they can see around them and we simulate their movement rather than following premade Navmesh paths.
Unfortunately the default character controller implementation in S&box is just too laggy, your fps is cut in half with 10 players, so imagine what it would do with 50 NPCs that use it.
It took me about 2 weeks but I managed to implement our own Character Controller using the Collide and Slide algorythm, you can find more about it here: https://www.peroxide.dk/papers/collision/collision.pdf
I'm seeing anywhere from 55% to 210% performance improvement with it and we're thinking of releasing a streamlined version soon.
As a side note, we finally increase the amount of NPCs spawned in the city since that doesn't drop your FPS (as much) anymore, and we made them killable this time, so have fun!