This is because a the hitchhiker needs a "way out" if they pick up a bad vibe from the driver and decide they (politely) want to decline the ride. If you decide that you'd rather not have this person in your car, don't feel like you have to just because you've pulled over and had a few words with them.
It's more dangerous for them. If you're driving and youre alone you have control of the situation. Although most hitchhikers only want a ride and