I've noticed sometimes that the thing can drift very very slowly, thought it might be accumulative errors in the INU causing the drift. The comment about hover hold seems very appropriate.
One thing I've noticed is that I can sometimes have it trimmed up etc, engage the hover and it starts going all over the place. Afterwards it feels like the autopilot is off (I've started to fly with the AP off to train up flying skills) even though the AP buttons are lit. Might have engaged auto hover under 4m though so that could be a trigger.