So I think I found the solution, by searching the forum some more. Just played for over an hour with no issues what so ever. Using other posts, I found that RAM timing issues might be the problem. I've got a AMD B650 Asus motherboard, which I set the ram to EXPO1 for. This should make for a stable RAM usage, although on OC, running on 6400 Mhz, which is the spec for these two RAM modules.
However, there is something going on that trains the memory once in a while when setting it to EXPO1, apparently. Setting it to EXPO2, should train the memory at each boot. Thus, explaining how I got the issue, the training did not take place for a while, making for an unstable system.
Now, for the testing I've just reverted to the AUTO setting, which runs my memory on 5300 Mhz for now. It's stable, and I might experiment with the EXPO2 (which makes for a slower boot, but a more stable system apparently...)
I'm also checking if the RAM combined with my motherboard even supports EXPO, or only XMP. I might have gotten those two mixed up.