It works because you now have defined the S3 button to be the modifier. But it will still output the DX command you have assigned to it.
The flag thing is a bit more work and not necessarily better. Depends what you want to do.
Here is an example:
//define flag
char shift;
MapKey(&Joystick, S3, TEMPO( DX5, EXEC(" shift=1;"), 500));
MapKeyR(&Joystick, S3, EXEC(" shift=0;"));
MapKey(&Joystick, TG1, EXEC(" if (shift) ActKey( KEYON+ PULSE+ 'a') else ActKey( KEYON+ PULSE+ 'b')));
Not sure the code is flawless, I need to go to the other pc to check, but i hope you can get the idea.