Who said anything about "special privileges?" That is the code-phrase that right-wingers typically use any time an argument of equal rights for sexual minorities comes up, since they can't muster any real argument against all adult human beings having equal rights under the law.
Indeed you are right that government should not recognize or discriminate against any persons. This means no restricting marriage to certain combinations of persons, no race, gender, or sexual preference check boxes on forms, and no government facilities segregated on the basis of gender. Especially when government is trying to force someone to identify with a gender which is not the individual's gender of choice.
As the Advocates for Self-government might put it: Who should decide which gender you belong to, you or the government?
Yours in liberty,
<<< Starchild >>>