I wash my hands every time, but when I'm in a public restroom I almost invariably use just water due to allergen issues with chemicals in most soaps*. That said, it's rarely a "quick rinse"; I still do all the scrubbing I do at home, I just don't use soap. At home I have soap that doesn't leave my hands an itchy, scaly mess.
*At my office, the building owners have two soaps available. I'm allergic to both, for different reasons. Since I'm probably the only person there with the particular combination of allergies to make that an issue, I haven't complained. (For the curious, citrus and aloe. The dispensers have citrus-scented soap, the bottles on the counter aloe.)
Also, I open most bathroom doors with only part (of my left hand. It's the one that I don't really use, so if it gets contaminated the chances of issues are much reduced.