Did you know about the Open Source Psychometrics project? It has more than two dozen personality tests that are all free to use with a creative commons license and posts large, open access datasets for their validation? Wow. What’s even stranger is that this site has no university affiliation, so far as I can tell and I can’t find any info on the site’s administrator. They’ve collected data from ~300,000 people here, these datasets are massive and an incredible resource, but are also virtually untouched by the academy. It’s downright criminal.
Case in point, the Open Sex Roles Inventory (OSRI). I started looking up data on Bem’s Sex Role inventory to help enrich the gender section of my intro psychology class, when I came across this measure. The open dataset had ~20,000 data points in it and I thought it’d be fun to play around with. I sent an email to the site administrator asking about how to score it and validation info. No email response, but the website was updated four days later with a TON of info and modifications to the scale to make a 20-item, more well-validated short form from the original 44 items. Seriously, who is this wonderful person? These data could easily be published in any personality journal they wanted, but they’re just free for all to use with no glory or publishing prestige. Searching online, I found a single publication using the OSRI, and it was a Ph.D. dissertation that used the data to test out a new method of measurement invariance, for which this particular data was mostly incidental (finding that the 44-item version of the OSRI didn’t have measurement invariance across heterosexual vs. non-heterosexual groups.
Anyway, back to the point. I wanted to have a visualization for my intro psychology class to clarify the distinction between sex (i.e., biological/physiological) and gender (i.e., socio-cultural). Specifically, I wanted to talk about Sandra Bem’s theory published way back in 1974 that gender roles are spectrum with at least two dimensions (masculinity vs. femininity). I’m aware of challenges to this distinction (e.g., that sex/gender are inseparable, and that sex is also a spectrum), but broadly won’t get into that here. In any event, the dataset I have only has sex measured as “Male, Female, Other,” so a categorical model will need to do for now. My goal was to show that there is a sizable minority of people beyond cis-gendered men and women. The dataset is pretty large (~16,000 after missing data was removed), so I decided to use geom_hex() with the viridis color scheme. This crushes my computer, and takes a long time to run but in the end, after a lot of obsessing over details, I am happy with the result. Special thanks to this post on Stack Overflow that helped me solve an graphical glitch that stumped me for a long time.
ggplot(mydata2, aes(x = FemSF, y = MascSF)) + geom_hex(aes(colour = ..count..), bins = 22) + theme_classic() + facet_grid(. ~ gender) + scale_fill_viridis() + geom_hline(yintercept = mean(mydata$MascSF)) + geom_vline(xintercept = mean(mydata$FemSF)) + labs(y = "Masculinity", x = "Femininity" ) + geom_text(x=4.25, y=4.25, label="Androgynous", col = "darkslategray2") + geom_text(x=2.0, y=1.75, label="Undifferentiated", col = "darkslategray2") + geom_text(x=2.0, y=4.25, label="Masculine", col = "darkslategray2") + geom_text(x=4.25, y=1.75, label="Feminine", col = "darkslategray2")
I like the way this graph came out. It shows that indeed, you can predict self-identified male or female status with the items on this questionnaire. But, it also shows that there’s a lot of variation, with folks that are androgynous (i.e., high in both) or undifferentiated (i.e., low in both) in for males and females. The “Other” gender category looks more similar to the female category than the male category, but a little more in the center overall. When I did some t-tests comparing men vs. women, femininity had a cohen’s d of 1.14 and masculinity had a cohen’s d of 0.82. Those are huge effect sizes! Much larger than the majority of sex differences in the literature.
Out of general interest, I decided to do a few more quick plots to break separate out the heterosexual vs. the non-heterosexual participants (I binned homosexual, bixsexual, asexual and other into one category for comparison in this analysis). Nothing fancy this time around, just some simple density plots:
ggplot(mydata2, aes(x = Fem, fill = gender)) + geom_density(alpha = .4) + facet_grid(. ~ orientation.d) + scale_fill_manual(values = pal) + labs(x = "Femininity", fill ="") ggplot(mydata2, aes(x = MascSF, fill = gender)) + geom_density(alpha = .4) + facet_grid(. ~ orientation.d) + scale_fill_manual(values = pal) + labs(x ="Masculinity", fill ="")
One thing that is pretty striking about these plots is that the femininity and masculinity scales predict identifying as a heterosexual man or woman REALLY well, with massive effect sizes. Folks identifying as “other” fell somewhere in between, though a little closer to the female profile. However, for the non-heterosexual folks it’s a different story. The sex difference is still there for femininity, but less pronounced, mostly accounted for by men being more feminine. For the masculinity scale though, there’s basically no discrimination at all! Men, women and non-binary folks can’t really be distinguished based on their masculinity scores. I suppose that’s consistent with the measurement invariance issue the this Ph.D. thesis picked up on. Not sure what it means in a theoretical sense, but it’s definitely interesting.
I could stop here, but I wanted to also do a quick factor analysis on the short form data, just to see if the factor structure looked good. Nothing fancy or in-depth, just an exploratory factor analysis extracting two factors, with principal axis factoring, and an oblique rotation. I also fussed around until I could make a nice plot of the factor loadings, using the ggrepel package.
#Factor Analysis mydata4 <- select(mydata2, Q4, Q26, Q28, Q30, Q34, Q36, Q38, Q40, Q42, Q44, Q1, Q3, Q5, Q7, Q9, Q17, Q29, Q35, Q39, Q43) corMatsf <- cor(mydata4) solutionsf <- fa(r = corMatsf, nfactors = 2, rotate = "oblimin", fm = "pa") #Made a .csv file manually by copying into excel (can't figure out better solution) print(solution$loadings, digits=2, cutoff=0) loadings.sf <- read.csv("efa.loadingsSF2.csv") #The plot ggplot(loadings.sf, aes(x = PA1, y = PA2, label = item, col = FemMasc)) + geom_label_repel() + geom_point(size = 3) + scale_y_continuous(limits = c(-.80, .80), breaks = seq(-.80, .80, .10)) + scale_x_continuous(limits = c(-.80, .80), breaks = seq(-.80, .80, .10)) + geom_hline(yintercept = 0) + geom_vline(xintercept = 0) + scale_color_manual(values = c("hotpink", "royalblue")) + labs(x = "Factor 1 (Masculinity)", y = "Factor 2 (Femininity)", col = "") + guides(col=FALSE)
That looks pretty excellent, items are clearly distinguished by their factors and the factors are mostly uncorrelated. So the measurement seems solid. No wonder, given the amount of data that went into validating it! Overall, very cool dataset and measure. Might even use this myself some day in a published study…
If you want to take the test yourself, take a look here: https://openpsychometrics.org/tests/OSRI/