Weapons of Math Destruction

Key Takeaways

Under Consideration — to be added.

Connections

No cross-book interconnections identified yet.

Highlights

Roam capture metadata — Author: Cathy O’Neil · Reading Status: Books Read · Source: https://medium.com/@kwharrison13/2018-in-books-part-i-7d3590e7f5e8 · Recommended By: Minna Wang Folio Books · Tags: #Books

• ==Highlighted Quotes==
  • ==START REVIEWING HERE>>>==
  • “But instead, in the wake of the crisis, new mathematical techniques were hotter than ever, and expanding into still more domains. They churned 24/7 through petabytes of information, much of it scraped from social media or e-commerce websites. And increasingly they focused not on the movements of global financial markets but on human beings, on us. Mathematicians and statisticians were studying our desires, movements, and spending power. They were predicting our trustworthiness and calculating our potential as students, workers, lovers, criminals.”
  • “The math-powered applications powering the data economy were based on choices made by fallible human beings. Some of these choices were no doubt made with the best intentions. Nevertheless, many of these models encoded human prejudice, misunderstanding, and bias into the software systems that increasingly managed our lives. Like gods, these mathematical models were opaque, their workings invisible to all but the highest priests in their domain: mathematicians and computer scientists. Their verdicts, even when wrong or harmful, were beyond dispute or appeal. And they tended to punish the poor and the oppressed in our society, while making the rich richer.”
  • “In Weapons of Math Destruction (WMD), many poisonous assumptions are camouflaged by math and go largely untested and unquestioned.”
  • “Do you see the paradox? An algorithm processes a slew of statistics and comes up with a probability that a certain person might be a bad hire, a risky borrower, a terrorist, or a miserable teacher. That probability is distilled into a score, which can turn someone’s life upside down. And yet when the person fights back, ‘suggestive’ countervailing evidence simply won’t cut it. The case must be ironclad. The human victims of WMDs, we’ll see time and again, are held to a far higher standard of evidence than the algorithms themselves.”
  • “But there’s one important distinction between a school district’s value-added model and, say, a WMD that scouts out prospects for extortionate payday loans. They have different payoffs. For the school district, the payoff is a kind of political currency, a sense that problems are being fixed. But for businesses it’s just the standard currency: money. For many of the businesses running these rogue algorithms, the money pouring in seems to prove that their models are working. Look at it through their eyes and it makes sense. When they’re building statistical systems to find customers or manipulate desperate borrowers, growing revenue appears to show that they’re on the right track. The software is doing its job. The trouble is that profits end up serving as a stand-in, or proxy for truth. We’ll see this dangerous confusion crop up again and again.”
  • “The folks building WMDs routinely lack data for the behaviors they’re most interested in. So they substitute stand-in data, or proxies. They draw statistical correlations between a person’s zip code or language patterns and her potential to pay back a loan or handle a job. These correlations are discriminatory, and some of them are illegal. Baseball models, for the most part, don’t use proxies because they have pertinent inputs like balls, strikes, and hits.”
  • “Our own values and desires influence our choices, from the data we choose to collect to the questions we ask. Models are opinions embedded in mathematics.”
  • “Needless to say, racists don’t spend a lot of time hunting down reliable data to train their twisted models. And once their model morphs into a belief, it becomes hardwired. It generates poisonous assumptions, yet rarely tests them, settling instead for data that seems to confirm and fortify them. Consequently, racism is the most slovenly of predictive models. It is powered by haphazard data gathering and spurious correlations, reinforced by institutional inequities, and polluted by confirmation bias.”
  • “The first question is (1) even if the participant is aware of being modeled, or what the model is used for, is the model opaque, or even invisible? (2) does that model work against the subject’s interest? In short, is it unfair? Does it damage or destroy lives? (3) Is whether a model has the capacity to grow exponentially. As a statistician would put it, can it scale?”
  • “I got the sense that the people warning about risk were viewed as party poopers or, worse, a threat to the bank’s bottom line. This was true even after the cataclysmic crash of 2008, and it’s not hard to understand why. If they survived that one — because they were too big to fail — why were they going to fret over risk in their portfolio now?”
  • “I wondered what the analogue to the credit crisis might be in Big Data. Instead of a bust, I saw a growing dystopia, with inequality rising. The algorithms would make sure that those deemed losers would remain that way. A lucky minority would gain ever more control over the data economy, raking in outrageous fortunes and convincing themselves all the while that they deserved it.”
  • “When you create a model from proxies, it is far simpler for people to game it. This is because proxies are easier to manipulate than the complicated reality they represent.”
  • “It’s here that we find the greatest shortcoming of the U.S. News college ranking. The proxies the journalists chose for educational excellence make sense, after all. Their spectacular failure comes, instead, from what they chose not to count: tuition and fees. Student financing was left out of the model.”
  • “The victims, of course, are the vast majority of Americans, the poor and middle-class families who don’t have thousands of dollars to spend on courses and consultants. They miss out on precious insider knowledge. The result is an education system that favors the privileged. It tilts against needy students, locking out the great majority o them — and pushing them down a path toward poverty. It deepens the social divide.”
  • “So the government capitulated. And the result might be better. Instead of a ranking, the Education Department released loads of data on a website. The result is that students can ask their own questions about the things that matter to them — including class size, graduation rates, and the average debt held by graduating students. They don’t need to know anything about statistics or the weighting of variables. The software itself, much like an online travel site, creates individual models for each person. Think of it: transparent, controlled by the user, and personal. You might call it the opposite of a WMD.”
  • “The for-profit colleges do not bother targeting rich students. They and their parents know too much.”
  • “The feedback loop for this WMD is far less complicated than it is nefarious. The poorest 40 percent of the US population is in desperate straits. Many industrial jobs have disappeared, either replaced by technology or shipped overseas. Unions have lost their punch. The top 20 percent of the population controls 89 percent of the wealth in the country, and the bottom 40 percent controls none of it. Their assets are negative: the average household in this enormous and struggling underclass has a net debt of $14,800, much of it in extortionate credit card accounts. What these people need is money. And the key to earning more money, they hear again and again, is education.”
  • “… bankers are virtually invulnerable. They spend heavily on our politicians, which always helps, and are also viewed as crucial to our economy. That protects them. If their banks go south, our economy could go with them. (The poor have no such argument.)”
  • “The Constitution’s implicit judgment is that freeing someone who may well have committed a crime, for lack of evidence, poses less of a danger to our society than jailing or executing an innocent person. WMDs, by contrast, tend to favor efficiency. By their very nature, they feed on data that can be measured and counted. But fairness is squishy and hard to quantify.”
  • “The unquestioned assumption is that locking away “high-risk” prisoners for more time makes society safer. It is true, of course, that prisoners don’t commit crimes against society while behind bars. But is it possible that their time in prison has an effect on their behavior once they step out? Is there a chance that years in a brutal environment surrounded by felons might make them more likely, and not less, to commit another crime?”
  • “And this is precisely what happens. Consider a recidivism study by Michigan economics professor Michael Mueller-Smith. After studying 2.6 million criminal court records in Harris County, Texas, he concluded that the longer inmates in Harris County, Texas, spent locked up, the greater the chance that they would fail to find employment upon release, would require food stamps and other public assistance, and would commit further crimes. But to turn those conclusions into smart policy and better justice, politicians will have to take a stand on behalf of a feared minority that many (if not most) voters would much prefer to ignore. It’s a tough sell.”
  • “Not a chance, I’d say. The difference is this: Basketball teams are managing individuals, each one potentially worth millions of dollars. Their analytics engines are crucial to their competitive advantage, and they are hungry for data. Without constant feedback, their systems grow outdated and dumb. The companies hiring minimum-wage workers, by contrast, are managing herds. They slash expenses by replacing human resources professionals with machines, and those machines filter large populations into more manageable groups. Unless something goes haywire in the workforce — an outbreak of kleptomania, say, or plummeting productivity — the company has little reason to tweak the filtering model. It’s doing its job — even if it misses out on potential stars.”
  • “… We’ve seen time and again that mathematical models can sift through data to locate people who are likely to face great challenges, whether from crime, poverty, or education. It’s up to society whether to use that intelligence to reject and punish them — or to reach out to them with the resources they need. We can use the scale and efficiency that make WMDs so pernicious in order to help people. It all depends on the objective we choose.”
  • “Scheduling software can be seen as an extension of the just-in-time economy. But instead of lawn mower blades or cell phone screens showing up right on cue, it’s people, usually people who badly need money. And because they need money so desperately, the companies can bend their lives to the dictates of a mathematical model.”
  • “That’s all the more reason to spread the word about these and other WMDs. Once people recognize them and understand their statistical flaws, they’ll demand evaluations that are fairer for both students and teachers. However, if the goal of the testing is to find someone to blame, and to intimidate workers, then, as we’ve seen, a WMD that spews out meaningless scores gets an A-plus.”
  • “In other words, the modelers for e-scores have to make do with trying to answer the question, “How have people like you behaved in the past?” when ideally they would ask, “How have __you __behaved in the past?”
  • “…The world is dominated by automatic systems chomping away on our error-ridden dossiers. They urgently require human context, common sense, and fairness that only humans can provide. However, if we leave this issue to the marketplace, which prizes efficiency, growth, and cash flow (while tolerating a certain degree of errors), meddling humans will be instructed to stand clear of the machinery.”
  • “I have no reason to believe that the social scientists at Facebook are actively gaming the political system. Most of them are serious academics carrying out research on a platform that they could only have dreamed about two decades ago. But what they have demonstrated is Facebook’s enormous power to affect what we learn, how we feel, and whether we vote. Its platform is massive, powerful, and opaque. The algorithms are hidden from us, and we see only the results of the experiment researchers choose to publish.”
  • “In any case, the entire political system — the money, the attention, the fawning — turns to targeted voters like a flower following the sun. The rest of us are virtually ignored (except for fund-raising come-ons). The programs have already predicted our voting behavior, and any attempt to change it is not worth the investment. This creates a nefarious feedback loop. The disregarded voters are more likely to grow disenchanted. The winners know how to play the game. They get the inside story, while the vast majority of consumers receive only market-tested scraps.”
  • “It’s a silent war that hits the poor hardest but also hammers the middle class. Its victims, for the most part, lack economic power, access to lawyers, or well-funded political organizations to fight their battles. The result is widespread damage that all too often passes for inevitability.”
  • “Big Data processes codify the past. They do not invent the future. Doing that requires moral imagination, and that’s something only humans can provide. We have to explicitly embed better values into our algorithms, creating Big Data models that follow our ethical lead. Sometimes that will mean putting fairness ahead of profit.”
  • “How do we start to regulate the mathematical models that run more and more of our lives? I would like to suggest that the process begin with the modelers themselves. Like doctors, data scientists should pledge a Hippocratic Oath, one that focuses on the possible misuses and misinterpretations of their models. Following the market crash of 2008, two financial engineers, Emanuel Derman and Paul Wilmott, drew up such an oath. It reads:
    • I will remember that I didn’t make the world, and it doesn’t satisfy my equations.
    • Though I will use models boldly to estimate value, I will not be overly impressed by mathematics.
    • I will never sacrifice reality for elegance without explaining why I have done so.
    • Nor will I give the people who use my model false comfort about its accuracy. Instead, I will make explicit its assumptions and oversights.
    • I understand that my work may have enormous effects on society and the economy, many of them beyond my comprehension.
  • “The achievement gap, mass incarceration, and voter apathy are big, nationwide problems that no free market nor mathematical algorithm will fix. So the first step is to get a grip on our techno-utopia, that unbounded and unwarranted hope in what algorithms and technology can accomplish. Before asking them to do better, we have to admit they can’t do everything.”
  • “Finally, models that have significant impact on our lives, including credit scores and e-scores, should be open and available to the public. Ideally, we could navigate them at the level of an app on our phones. In a tight month, for example, a consumer could use such an app to compare the impact of unpaid phone and electricity bills on her credit score and see how much a lower score would affect her plans to buy a car. The technology already exists. It’s only the will we’re lacking.”
  • “The researcher’s’ finding was not welcome. For a meeting with important public officials, our group prepared a PowerPoint presentation about homelessness in New York. After the slide with statistics about recidivism and the effectiveness of Section 8 was put up, an extremely awkward and brief conversation took place. Someone demanded the slide be taken down. The party line prevailed.”
  • “Models like this will abound in coming years, assessing our risk of osteoporosis or strokes, swooping in to help struggling students with calculus II, even predicting the people most likely to suffer life-altering falls. Many of these models, like some of the WMDs we’ve discussed, will arrive with the best intentions. But they must also deliver transparency, disclosing the input data they’re using as well as the results of their targeting. And they must be open to audits. These are powerful engines, after all. We must keep our eyes on them.”
  • “The Facebook debacle raises some interesting questions. First, what do we mean by “bias”? Given how polarized our newspapers and public discussions have become — there is now widespread disagreement about the facts themselves, and about the limits of appropriate debate — what policy could Facebook adopt that would be seen as universally fair?”
  • “But in the last couple of years, we have seen the darker consequences of its algorithm. For example, after the presidential election, if you asked Google “who won the popular vote,” the very first result was a link to a conspiracy blog claiming Trump had won. Dylann Roof, the young man who murdered nine church goers in Charleston in 2015, was by his own account radicalized after searching the term “black on White crime” and believing the results verbatim. This should not surprise us. Machine learning and artificial intelligence algorithms do not have an embedded model of the world that can reliably distinguish between truth and lies.”
  • “We need to ensure that data effectively and comprehensively represents the world — even, we hope, bears witness to the world and its suffering, rather than shaping the world — especially in ways that exacerbate misery. To accomplish this, we’d need to expand the role of data scientist.”
  • “This brings us to a larger question about the nature of evidence itself. If there’s one thing that I’ve learned in researching this book, it’s that different fields have very different concepts evidence of what constitute evidence, what elements of an argument need to be “proven,” and when accumulated evidence amounts to persuasive argument. What constitutes observable truth in law or philosophy may not in psychology.”