I think what it boils down to and this is of course just a theory...
Anyone with a reasonable amount of real flight experience finds that the AP fights them. I cannot get comfortable with the APs and find myself in a similar situation to OP.
Other squad members are just fine with the APs.
This could be a number of any issues, but the most likely culprit is real experience. Having never flown a Ka-50 nor an autopilot assisted helicopter, but having 150+ hours of fixed wing time with some heli time thrown in for good measure that is the root problem for me...
Just a theory...