The main reason is that sex and gender have been intrinsically linked due to the prime directive of societies which is survival through procreation. This allowed little room for variants since they did not contribute to this basic need.
In addition religious societies did not take a kind view of this kind of aberration. Anyone inclined to differ from this model would have extreme difficulty in living authentic lives; this was especially true in Judeo-Christian societies. Some ancient cultures hold some reverence for a third gender and the fate of these people has been markedly different.
The truth is that the vast majority of people expressing gender variance are intelligent and highly functional so the issue of mental illness begins to lose traction.
The question of sexual deviance was partly answered by Alfred Kinsey during his study of the sexual habits of American Society. What came out of this major work was that people had a wider range of sexual preferences and practices than previously understood. Kinsey’s work found that individual fantasies and tastes of many veered significantly from the accepted understanding of what was then considered to be normal human sexuality. His findings were published as Sexual Behavior in the Human Male in 1948, followed in 1953 by Sexual Behavior in the Human Female.
In a separate but parallel stream was the work begun by Magnus Hirschfeld in 1910 and continued by Harry Benjamin which in addition to sexuality studied gender identity. This culminated in the 1966 publishing by Benjamin of “The Transsexual Phenomenon” which raised awareness but most of all empathy for transgender and transsexual people everywhere.
Where we began to veer in the wrong direction was with the work of one particular 20th century sexologist named Kurt Freund. He was the first to employ the use of plethysmography (measurement of bloodflow to the penis) in his native Czechoslovakia as a way to have an objective measurement of sexual arousal in males. Freund used the device on sex offenders and on homosexuals and over his career, refined its use as part of his broader research on male sexual interest.
Other researchers and activists disputed this method as the best measure of orientation, pointing out that neither identity nor behavior are perfectly correlated with measured or self-reported arousal. Freund acknowledged this, and in fact demonstrated it in his studies, but maintained that orientation per se was best defined as the object of arousal.
Freund was mentor to one Ray Blanchard who took his position at the now infamous Clarke Institute in Toronto. Blanchard collaborated with Freund and formulated a similar approach only that now he was applying it in his reviewing the legitimacy of candidates for gender reassignment surgery. Blanchard's focus was now on sexual arousal as an indicator and primary driver for the desire to have such a procedure since pre-transition transsexuals, by virtue of their identity issue, sometimes experienced body conversion fantasies that one could easily exploit. The detailed questioning on the sexual fantasies of these candidates became the focus of his published work in the late 1980’s and gave birth to the much contested term “Autogynephilia”.
Blanchard’s work fell into virtual obscurity until it was picked up by Anne Lawrence in 1998 and by J Michael Bailey who in 2003 resurrected it in his book ‘The Man who would be Queen”. It understandably raised the ire of many in the transgender community due its "pull no punches" lack of tact, lack of credible science and insinuations that transgender people were basically sexual deviants.
As this work falls justifiably back into obscurity it still serves as a cautionary tale for many today.