It actually does "push harder" at the tail, since there is nothing to "grab onto" anywhere at the nose of an airplane for a crosswind component. In other terms, the profile shape of the tailfin generates lift pointing in a horizontal direction (left or right, one might call it), thus generating torque around the center of gravity of the plane, resulting in rotational acceleration around the planes vertical axis (also some roll, since the tailfin is usually above the CoG). Lift is generated, since the crosswind component lets the air hit the tailfin at an angle of attack >0, so that even a symmetrical profile as used on tailfins generates lift (beware this is angle of attack on the profile of the tailfin as a result of the sideslip angle beta, not the typical angle of attack alpha).
So yeah, theoretically, a plane would, without any inputs, weathervane into the wind (provided it is aerodynamically stable). But since an airplane usually gets controlled (and trimmed) by a pilot, it does not turn into the wind, but flys into another direction accordingly. Thereafter, only navigational corrections due to a potential crosswind component need to be applied.