I am about to get flamed from both sides. After giving this issue some serious thought, I have come up with what I believe to be the only logical conclusion.
Preventing a person from entering whichever restroom that they choose is not about crime. It is about possible crime. It is about speculation. There is no evidence that a man entering the same restroom as a woman is indicative that a crime is about to be committed, unless the act of entering the restroom labelled for the opposite sex is in itself illegal, of course. This means that the only real objection to this act is the unfounded and speculative fear that a crime may be committed in the future.
Contrast this fear with another common fear:
There is no evidence to indicate that an armed person who is entering a building is indicative that a crime is about to be committed, unless the act of entering the building while armed is in itself illegal, of course. This means that the only real objection to this act is the unfounded and speculative fear that a crime may be committed in the future.
For this reason, my position on transgender bathrooms is the same as for firearms: A law that prevents a person from entering a building (restroom) of his or her choosing (as long as said building is open to the public) while they are transgender or armed, is the antithesis of liberty, as long as said people are otherwise not committing a crime.
With that being said, a private business is free to prohibit people from doing either, as long as it is understood that the business is then responsible for providing reasonable alternative arrangements for the security/restroom needs of those whom they would deny access.