Are you a man or a woman? The chances are you have a strong feeling either way; probably one that's been with you since infancy and corresponds to your genitalia. But not necessarily. Some people have always believed they were born into the wrong gender; in others, these feelings didn't develop until later life. Some may opt to surgically change their gender; others may occasionally cross-dress; or do nothing at all. Then there are people who feel neither masculine nor feminine, but somewhere in between.  It takes all sorts to make a world, they say, and this is certainly true of gender.

But where do these innate feelings of male and femaleness come from? And to what extent do they shape the person we will eventually become?

Debate around the origins of male and female behaviour has been raging for decades. In the 1970s, feminists hoping to create a more equal society dressed boys in dresses and encouraged little girls to play with trucks. Then, in the early '90s, public interest shifted towards the apparent differences between the male and female brain – largely thanks to the international bestseller Men Are From Mars, Women Are From Venus. Today these debates continue, for example in the discussion around whether toy stores should segregate their wares into pink and blue stereotypes, but growing research into the biological underpinnings of trans-sexuality is also shedding new light on the matter.

You can't look at an individual brain and know if it is male or female

People often talk about the male and female brain, and it's true there are some differences – though their significance is often over-hyped. On average, men have a larger total brain volume, and there's also variation in the size of certain brain regions; for instance, the cortex, or outermost layer of the brain tends to be slightly thicker in women, while the emotion processing amygdala tends to be slightly larger in men. Even so, there's a lot of overlap.

"You can't look at an individual brain and know if it is male or female," says Lise Eliot, associate professor of neuroscience at Rosalind Franklin University in Chicago and the author of Pink Brain, Blue Brain. The same applies to psychological traits, such as mathematical ability or empathy. "It's just not the case that men lack empathy and women are all universally supersensitive," adds Eliot. "There are statistical differences, but not predictive individual differences. There is tremendous overlap."

The brains of adult transgender people fit even less neatly into male or female categories. "It is not a matter of the size, it is a matter of the pattern – or how the brain has been built," says Antonio Guillamon at the National Distance Education University (Uned) in Madrid, who has been involved in some of these studies. At the whole brain level, women who identify as men tend to have female-sized brains, and men who identify as women tend to have male-sized brains, but when Guillamon and his colleagues scanned the brains of men, women and transgender individuals, they found subtle differences in four regions of white matter – fatty tissue harbouring the long spindly projections of nerve cells that connect different brain areas together – between each of these groups.

In women who identified as men, these regions more closely resembled those of male controls, whereas in male-to-female transsexual people, the structure of these regions was halfway between that of control men and women. "The brains of male to female transsexuals are not male brains exactly, and the brains of female to male transsexual are not exactly female," says Guillamon. However, he cautions that it's still too early to say whether these differences explain why people feel male or female.

The Lady Gaga hypothesis

Yet other studies have found similar reversals in male and female-typical patterns among transsexuals in small areas of grey matter. One of them is a tiny area of the hypothalamus called INAH3; the animal equivalent of which seems to influence sexual behaviour in rats.   

Some claim such findings as evidence that transsexuality has a physical basis rather than being a choice, but not everyone is convinced. One issue is the brain's innate plasticity, or its ability to rewire itself in response to experience.

"There may be hints of brain differences in transgender people, but you'd expect that because their life experience is going to have been quite different," says Eliot. "How long they have identified as the other gender; the way they talk; who they played with as children; what sort of jobs they're involved in – all of these things could affect those same pathways in subtle ways. There is certainly no proof of the Lady Gaga hypothesis that I was 'Born This Way'."

That's not to take anything away from individuals' profound sense of having the wrong brain for their body, she emphasises. But when it comes to identifying the biological underpinnings of gender identity, possibly the adult brain isn't the best place to look.

What about infants?  On the surface at least, the development of gender seems simple: babies born with two X chromosomes will develop female genitals, while those with an X and a Y will grow male ones. The reason is testosterone, and a gene on the Y chromosome which kicks off its production in male foetuses. Even by week twelve of pregnancy – around the time of a woman's first ultrasound scan – the presence or absence of this hormone will have sculpted the baby's genitals into either testes or ovaries. But its psychological gender is thought to develop later – and precisely when is still very much up for debate.

David Reimer lost his penis during a botched operation when he was eight months old – and doctors assumed he'd be better off raised as a girl

For a long time babies were thought to be blank slates as far as gender identity is concerned. "The idea in the 1970s was that it was the social environment that would push the child in the male or female direction," says Dick Swaab, a professor of neurobiology at the University of Amsterdam. But this view was challenged by individuals like David Reimer, who lost his penis during a botched operation when he was eight months old. Doctors assumed he'd be better off if he was raised as a girl, so when he was 17 months old his testicles were also removed, he was renamed Brenda, dressed in girls' clothes, and given female hormones at puberty.

However, it turned out that Brenda had always felt masculine, and as an adult reverted back to his birth sex. This supports the idea that a person's sense of gender is established very early in life. Indeed, Swaab and many others believe it is hardwired before a baby is even born.

Unfortunately, we can't ask babies whether they identify as boys or girls, and it's difficult to scan their brains because they move about so much. Instead, scientists have focused on studying their behaviour. A major clue is the toys they prefer to play with. "Play behaviour shows a big sex difference – at least as large as the sex difference in height," says Melissa Hines, director of the hormones and behaviour research lab at the University of Cambridge. Although social conditioning may exaggerate such differences, similar preferences for vehicles and dolls have been found among male and female monkeys.

A key piece of evidence supporting the prenatal hardwiring of gender identity comes from studies of girls who were exposed to high levels of testosterone in the womb because of a genetic condition called congenital adrenal hyperplasia (CAH). Unlike most girls, who tend to show a preference for stereotypically female toys like dolls, "if we put these girls with CAH into a playroom with lots of different toys they will spend more time with the toys that boys normally choose, like vehicles," says Hines.

They're also far more likely to identify as male when they grow up, compared to the rest of the female population. But it's still only a small minority – roughly 1% of women with CAH – who feel this way. 

This suggests that, although prenatal exposure to testosterone may be one part of the puzzle, it's not the only piece. Other studies have suggested that genetic variation in the receptor for testosterone and the female hormone, oestrogen can increase the chances of being transsexual.

Quite a number of children have gender identity questions, but only 10% of them or so will develop along the trajectory of transsexualism  

But while identical twins are more likely to both be transsexual than non-identical twins, there are plenty of exceptions, such as Jonas and Nicole (born Wyatt) Mains, whose story is described in the book Becoming Nicole: The Transformation of an American Family, published last year. Even at the age of two, Wyatt was asking his mother when his penis would fall off and when he'd get to be a girl.

Such gender confusion can be common in young children, and by no means guarantees that those feelings will persist into adulthood – although in Nicole's case they did; she took drugs to delay puberty and underwent a sex change before starting college. "Quite a number of children have gender identity questions at early ages, but only 10% of them or so will develop along the trajectory of transsexualism," says Swaab.

Take Debra Soh, a sex writer and neuroscientist at York University in Toronto. "As a very young child, I remember feeling that I should have been born as a boy: I preferred to look like a boy, all of my friends were boys, and I was only interested in male-typical activities, like rough-housing and playing with trucks," she says. "My parents allowed me to shave my head and behave as I wanted, while maintaining a female identity."

Her understanding of neuroscience has led her to the conclusion that her brain was probably partially masculinised during development. However, in her late teens, Soh grew more comfortable with the idea of being female. "It was a combination of emotional maturity and realising that one does not have to be stereotypically feminine in order to be, or identify, as female," she says. "To this day, as an adult, I still have times when I feel more masculine than feminine, but this doesn't cause me distress. This isn't to say that transgenderism doesn't exist, but I believe it is possible to notfeel 100% female (or male), and still feel comfortable in your birth sex."

There are also plenty of transsexuals whose feelings that they're in the wrong body don't start until later on. "We often hear about kids who have felt that there in the wrong gender from an early age, but there's plenty of people that decide they are transgender in puberty, in early adulthood, even in late adulthood," says Eliot.

That takes us back to environment – both before and after birth. Although you might assume that identical twins have identical experiences in the womb, they often don't. Where in the uterus they attach; what proportion of nutrients they receive; physical interactions with their twin – these are just some of the differences that may alter their experiences in ways that we are only just starting to understand. One possibility is that so-called "epigenetic" factors might shape our sense of gender. For instance, recent studies have revealed that chemical modifications to genes in another area of the brain that has been linked to transsexualism – the BSTc – can change the play of male rat pups, reducing the amount they playfully bite, chase and pounce on one another, so that they look more like females.

Meanwhile after birth, social factors may yet exaggerate innate differences between males and females. "The take-home message right now is that there are undoubtedly genetic and hormonal tweaks that may push you in the male or female direction behaviourally and in terms of the brain's microstructure, but there is a tremendous amount of plasticity," says Eliot. "I believe one's psychological identity – your sense of your own gender – is influenced by how your brain trains itself across life. These little kids who are trying to figure this all out: they have to essentially pick a team, and if a little XY individual just has much more affinity for the pink team he/she will decide, I'm actually a girl and then will play with girl things, identify with girl things and essentially wire themselves up into what we think of as solidly female identity."

So where does this leave us? Although we understand a few factors that shape our sense of gender, we're far from a complete picture. We're also a long way from being able to screen young children for transsexualism on the basis of a brain scan, their behaviour, or other factors. That doesn't mean we should stop trying to understand it: a child who holds onto the belief that they're wrong gender, particularly once puberty kicks off, is at high risk of depression and even suicide.

But rather than viewing gender as a binary thing, perhaps it's time we start thinking of it as a spectrum – the way we do other psychological traits, such as tendency towards autism. If you look at male and female brains, there is a huge amount of overlap in our skills and abilities, even as adults. The same is possibly true of our sense of gender, particularly when it is still developing.

"The spectrum notion doesn't necessarily appeal to, or help everybody, but I sometimes almost feel a sigh of relief in the room when I mention it as a possibility," says Michelle Bridgman, a British psychotherapist who specialises in gender dysphoria, and herself transitioned from male to female in mid-life. "When I work with clients I encourage them to think of it as a journey. It's not a one-off event where you go from A to Z; there's a whole range of positions in between these two polarities, which means we need to explore and see what it means for people."

We humans have a tendency to see things in black or white, but our physical gender belies the complexity of our brains. Gender may well be a constellation of different traits written in multiple dimensions. We're not just pink or blue, but all the beautiful shades of purple in between.

—

This story is part of our Sexual Revolutions series on our evolving understanding of sex and gender.

Linda Geddes is a freelance writer in Bristol and the author of Bumpology: The Myth-busting Pregnancy Book for Curious Parents-to-be. Emma Leslie is a photographer based in Sydney. An exhibition of her Transcend series of portraits will appear at GkJE Galleries in Sidney between 1 and 28 September.

Join 600,000+ Future fans by liking us on Facebook, or follow us on Twitter, Google+, LinkedIn and Instagram. If you liked this story, sign up for the weekly bbc.com features newsletter, called "If You Only Read 6 Things This Week". A handpicked selection of stories from BBC Future, Earth, Culture, Capital, Travel and Autos, delivered to your inbox every Friday.