Warthog Stick Bad Stiction Problem

[Madexi]

Allright this is very bad. I tried regreasing but the biggest problem is this part:

It just refuses to move in those pins - it rubs them very badly and adds friction. I tried reinstalling the pins and turning the big piece 90-270 degrees but it's still very bad.

 

I have a temporary solution: I removed one pin so that it moves on 3 pins only, and it's better but not as stable as before. What do you think, should I use sandpaper to enlarge the hole so that it would move better.

Using EM-30L btw.