I find the same time lag and I use key presses. The in-cockpit wheel turns, but it takes a long time before I can feel any + or - effect taking place. Then, when the effect is felt, it is a sharp change that does not feel like any trim response I have known before. I also use a FF stick and thought this was just an issue that I had, so I am interested to find that I am not the only one.
Happy landings,
Talisman