Sentiment analysis gone awry

Prooffreader has recently published an ‘attitude analysis’ of tweets related to feminism. One of his claims is that sentiment, as measured by traditional sentiment analysis techniques, is a poor predictor of attitude. To make this claim, he had a team (of humans) hand-classify 1000 tweets into one of three categories: pro-feminist, anti-feminist, or neither. The full analysis is available on Github, and it includes this plot:

Sentiment analysis cannot separate attitudes
Sentiment analysis cannot separate attitudes

This plot shows that there is basically no relationship between the sentiment of a tweet and whether or not the tweet is pro or anti feminist. This is an important lesson for organizations that use sentiment analysis to gauge the public’s attitude about their brand. What’s interesting is to look at the specific tweets that have positive sentiment but anti-feminist attitude and those that have negative sentiment but pro-feminist attitude. Prooffreader gives these hypothetical examples of the latter case:

    1. Man, do I ever hate feminists.

2. I hate that my mom does not like the word ‘feminism’.

The data set contains cases showing real-world break down in usefulness of basic sentiment analysis when the real goal is to capture attitudes. In all of the following examples, the attitude was determined by human classifiers, but the sentiment by the textblob package for Python.

Examples of anti-feminist attitudes expressed using positive sentiment

Any men shedding a tear at the end of #page3 , make sure you bottle them. #Feminists get thirsty….

I liked a @YouTube video http://t.co/r6xWBbR1ER Re: Potty-Mouthed Baby Feminists

Feminist article: don’t call me “working mommy” because we don’t call men “working daddy.” So? My MOM title is my most esteemed title. ~Mom

@DoeringNorman @Oneiorosgrip let’s be accurate here norm- when dealing with feminists- polite or not if you disagree,you are their enemy

Wait.. So the feminists siding with amber rose and slandering the lil kardashian for being in a relationship with a pedophile?

Slate Sex Trafficking Post Proves Feminists Can\u2019t Be Happy Even When You Agree With Them http://t.co/VvbbiR9Vk3 #tcot

Feminist gon kill Chuck lmao

@officialncc1701 ALRIGHT I GUESS COULD BE BETTER IF THEY WOULDNT SPOIL MY TV SERIES AND NOT ARGUE ABOUT FEMINISM WITH ME

@Sepko01 @kav_p Someone needs a pat on the shoulder. “It’s ok, bud. Just sit still and the feminists can’t see you.”

@elliottkrista @HisFeministMama The more you dumbass Feminists fight amongst yourselves, the more the MRAs laugh at you. Don’t you get that?

I just realized, the way that the anti-gamers and feminists promote Towerfall is a perfect allegory to their crumbling narrative. #GamerGate

@ONLINEPIXI3 but the “feminists” will love it.

I’m sure that over 100 feminists have me blocked.

#ThankYouEllenPao you have proven that women and minorities can be in power and can achieve amazing things and didn’t need #feminism for it.

Lol see these women are trolling, no other explanation “@Ntiana_: So real quick does feminism throw culture out the window ?   Yes or No”

So many otherwise intelligent people completely lose their minds when it comes to feminism. So many parallels between that and religion.

@Ashykneesirwin Okay so if you don’t feel ashamed, why do you need feminism? To stop men from making jokes about women?

be peddled in the name of feminism. I am glad we all could see those nuances. However, where is this vigilant mass when it comes to 3/n

twitter feminism in a nutshell lol http://t.co/8g3MhdOGEq

So many feminists on twitter so keen on protecting the rights of people who can pop out babies they think it’s OK to trample my boundaries.

@RaasAFC @Faisal_RedDevil Tbf what we said was true. And we shut her up. Feminism is a disgrace, mostly so for women like yourself.

too many attention whores on instagram all in the name of feminism

we need meninism more than feminism because men are the ones who found this world so \u2615\ufe0f

@Niki8954 I admire your optimism, but in my experience, feminism is about convincing women that they’re more oppressed than they are.

Examples of pro-feminist attitudes expressed using negative sentiment:

‘Isn’t feminism when u hate females’ – my brother, 2015

@siIentwords @honestfandom HeForShe is about gender equality/feminism. He wouldn’t have gone if he was against it. What’s so wrong with him+

It’s sad. People are so arrogant and insecure that they think feminism is about not needing a man.

Why would you date a boy who thinks feminism is stupid

@jessepurtell and you should not say you hate all feminists because one video you saw of a women being disrespectful.

Real #feminist will show support for @RealCytherea. This Violent Gang Rape I’d atrocious.  http://t.co/5nnhEr3HUt via @returnofkings

I don’t understand why feminists are being ignored or hated upon, they’re fighting for something that has been ignored for decades????

@BigFashionista @Steve___Miller I hate when anyone does this! Feminism is NOT the root of all evil. Discrimination is!

@PrisonPlanet Snowden still works for Booz/CIA, if you weren’t so obsessed with bashing feminism and Islam you might realize that, dumbass.

@drunkonjbxcbxag @tropicaljustxn @Kayfeminist @Grxnde__Butera she has to be a troll account trying to be a feminist but insulting females+

@Eyebrows_13 ctfu I explained what feminism was and she was like “but all the women I see are angry at men, don’t shave, are spelly hippies”

The lesson from this is that a sentiment analysis that is too simplistic does not provide reliable information about attitudes toward a certain topic.

Advertisements

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s