That surprises me, I would have expected that their wouldn't have been that much of a drag on processing since most of the tests would have resulted in no change in position ( which should avoided work ). usually if else comparisons are fast even though you would need to check against the volume of the box, so it would be a few per box. It may be that every object has to compare to every other object in the environment making it at least a O(n^2) algorithm of how many comparisons must be done per the main run loop, along with another a O(n) algorithm calculations done on every object during the main run loop. I may be discounting that the collision could run some code affecting every box contacting another box. I still would have expected performance to degrade long before crashing.
It would be compounded if it has to relay all positions to the clients, not just changes in position.