How long to load is too long to load depends on you, how much are you willing to wait for it to load?
And how many units is too many also depends on you. You might have a beefy system, but are you planning to publish the template? If yes, you might have to consider to remove unneeded units.