Ok, I managed to find the culprit.
It was not the view distance, it is the shadows.
Shadows turned off make my FPS go up to 80's on the ground near a big city and 120+ frames in the sky.
Shadows on low make my FPS on the ground fall to 40ish near a city.
Shadows on medium or higher cause FPS to fall to 20 or lower.
All other settings are on high at 1600x900, 2x MSAA, 2x Anistropic filtering. Trees at 15k, preload radious at 100k.
Computer: Win7 64bit, I5 3550 3.30GHZ, 8GB RAM, GTX 560. Computer is 4 years old, so it's not cutting edge. Time to upgrade RAM and video card.