Let's see, maybe you're not understanding how this switch works.
The function of the switch is to disable the aileron actuators, but even then you're able to move them, just this time with your own force by direct control. This means that ailerons will become harder to move at greater speeds, but you're still able to control them.