Book Review: Weapons of Math Destruction (Cathy O'Neil)

This post marks my first attempt in trying to force myself to gain a better understanding of the books that I've read. Previously, I find myself reading books after books without being able to recall the important things that I've learned earlier.

It's rather frustrating to be honest.

So I'm trying this out as a way for me to push myself to understand the book and synthesize the various concept and ideas that are conveyed from the book.

A disclaimer: My reviews will not attempt to be neutral or unbiased - as I feel that any attempt for me to try and write such kind of a blog post would result in a dry and boring outcome.

Guess you could say that it'd probably be much more of a rant rather than review.

Moving on.

I bought the book from Amazon quite awhile back in April and it has been on the shelf for quite sometime as I was another book at that time. The outline is rather interesting, as it highlights the pitfalls of big data implementation from a first person point of view. I wouldn't say that I was hooked on it's premise (hence why I didn't read it immediately), but I was quite curious as to what the author had in mind.

The author, Cathy O'Neil; is herself a data scientist. Some description of her from the book's cover:
She earned a Ph. D in mathematics from Harvard and taught at Barnard College before moving to the private sector, where she worked for the hedge fund D. E. Shaw. She then worked as a data scientist at various startups, building models that predict people's purchases and clicks. O'Neil started the Lede Program in Data Journalism at Columbia and is the author of Doing Data Science. She appears weekly on the Slate Money podcast
The book starts of with the author recounting her experience making a move from academia to D.E Shaw, filling the reader with stories of her excitement on being able to practice what she was good at. It's wasn't until the big financial crash of 2008, that she finally saw how what hedge funds such as the one that she was working in - was partly to be blamed for the outcome that wrecked havoc in people's lives.

She eventually left the company, leaving it to join a company that does risk assessment for banks. After all, if lack of oversight were to be blamed for the mismanagement that caused the 2008 crash; then assessing risk for future transactions would surely make the world a better place.

Things didn't turn out so well though, as she felt that the industry is simply running a rubber stamp business. Her experiences in these and other companies - left her disillusioned about how big data was being implemented and abused to optimize profitability disregarding ethical and moral concern.

2/3 of the book then talks about the various examples of where algorithms, deployed in a real life setting, had unintended consequences, impacting people lives due to the lack of foresight during it's inception.

One of the example talks about how the university ranking system created by U.S News started of an arms race where universities; in trying to climb up the ladders in ranking system, were only optimizing the exact features (such as academic citation) that were used by the algorithm (hey now....doesn't that remind you of stuff back home.. ). What the system fail to include in it's algorithm however, were items such as tuition and fees. This basically gave private entities all the incentives to hike up the fees and channel it to things that will bring them higher in the ranking - all of which while may contribute in a better learning environment to the students, would also burden them with heavy debts upon graduation.

The author calls them (the models) a Weapon of Math Destruction (WMD) - which as far as I can recall are algorithms/model that are:

  1. Has big target audience
  2. Doesn't apply a feedback loop to correct itself using future outcome
  3. Uses various proxies to account for the predicted outcome, which may in certain cases be discriminatory.
  4. Is not transparent to the audience

The book has various examples of these WMDs. So much so there were times while I was reading I really felt the urge to just skip a few chapters and just read the conclusion. But *ahem* sunken cost prevailed yet again. While it does get dry after awhile, it did prove that models that are built without item #2 and are filled with item #3 as in the above list would suffer from the biasness of the modeller or society as a whole. And what's worse about it is how it would reinforce that "perspective of the world" again to society.

Hence in the case of a hypothetical model created to predict targets for a loan shark - people with low income, mostly black, who lives in certain neighbourhood; are preyed upon again and again, thus ensuring that they will never be free of their predicament. They are now a victim of their situation.

That, to me, was what the whole book is all about. Proving that these WMDs exist and are deployed at various institution in our daily lives.

So what then - should we do with all these WMDs?

  1. In the case of models used in a non-commerial setting, she proposed that better consideration should be used when coming up with the features being used in making prediction models. While removing them could lead to a lower accuracy score of the model, she argues that a lower accuracy score is a better reflection of how the model performs as compared to one which perpetuates the bias that exists in society.
  2. Policing of the models by the public.
  3. Regulation, such as ones that being used in Europe, where customers have to opt-in before companies can use their data for other purposes.
  4. Transparency of the models to the public. 
My take on it

Ethical concern about big data is nothing new. But typically the angle of which ethics have been highlighted previously (at least in my experience) has been more about transparency and confidentiality. I find the angle about bias and self reinforcing bias quite interesting as it does have a far wider and continuing impact to the victims of the model. In my previous experience, revising your model has always been about making sure that your model does not go stale - since market trend changes and your model have to be tweaked to adjust for those changes. I guess now I'd have a new motivation to update my models.

I also wonder whether an explore-exploit strategy (from reinforcement learning) should be used in these instances. Meaning that, we should let our model produce a few errors and observe whether the result reinforces the bias or not. This would allow us to continue using the existing (bias-laden) features, but correct our model's assumption if the result show otherwise.


Popular posts from this blog

HIVE: Both Left and Right Aliases Encountered in Join

Assign select result to variable in Netezza stored procedure

Splitting value in Netezza using array_split