I kinda have the same issue, only my fps is terrible at all times while hosting.
I have an i5-2500k @ 4.2, 8GB RAM, 780ti classified and a EVO SSD at 4800x900
Hosting a mission for my buddies I get 11fps, that same mission that one of them hosts I get 50fps
I get that same 50fps if I am the only one connected to my server, when just one person joins, it tanks.
They see no performance drop on their end when joined to my server.