If you use target script editor, you can set it up like this:
SetShiftButton(&Joystick, S4, &Throttle, BSF, BSB);// Pinky paddle as shift + Boat switch activates layers (mode1, mode2, mode3 as you call it)
Examples how to setup buttons (different output when different mode/layer selected):
example1
MapKeyUMD(&Joystick, S4, 'u', 'm', 'd'); //when pressing the S4 button, if layer UMD master switch (boat switch on right throttle) is Up, generates a “u”, if middle generates an “m” and if Down, generates a “d"
example2 - using layers (using boat swtch) + submodes (using s4 as shift button) - I means "IN" as shift pressed, O means "out" as shift not pressed
MapKeyIOUMD(&Joystick, S4, KP1, KP2, KP3, KP4, KP5, KP6);
This one is clearly not easy to read, but we can change the way we display it.
MapKeyIOUMD(&Joystick, S3,
KP1, // BSF button, if shift button In generate Keypad 1
KP2, //BSF, if shift button Out generate Keypad 2
KP3, // BSM, if shift button In generate Keypad 3
KP4, // BSM, if shift button Out generate Keypad 4
KP5, //BSB, if shift button In, generate Keypad 5
KP6); //BSB, if shift button Out, generate Keypad 6
I know its a lot, but ask if you need any more help