I agree that the replay system is broken and needs to be addressed. As a youtube content creator, the current state of the replay system is very frustrating and extensively limits what I am able to create.
I don't know what the fix is, but from a user standpoint I want a stable system that allows you to fast forward and reverse at various speeds.