There is no way a mode 4/6 IFF can respond a hostile (red) contact(Yes, you can see the red contact on F/A-18, but it's not from your MODE 4 IFF reply, its a combination from multi-source) .
And friendly F-18 will always respond the correct reply (green) regardless your M6 key settings, planes other than JF-17 always use magic IFF unless in Mode 2. And there is no A+B key settings for M6, only ONE key at a time. The only way a friendly F-18 show as unknow(yellow) on your radar, is that you forget to turn on you interrogator, and your IFF symbol on RDR page will not be boxed if you try to interrogate.
The Deka's IFF tutorial on Youtube can give U more information on how the system works.