13 September 2022 by Richard

The Quality of Quantitative Analysis

I was asked by a colleague here in Leipzig to participate in a panel discussion of “What unites quantitative and qualitative research approaches?” I don’t have well-formed opinions on this topic, and I’ve never written about it. So naturally I immediately agreed to participate.

There’s a storied and diverse literature on the distinctions between qualitative scholarship and quantitative scholarship. Can anything new or interesting be said? That’s not a rhetorical question. The anthropologist in me—I don’t let him out often, because he asks uncomfortable questions—expects that the best thing to do with a dichotomy is to defy it. So I’m going to try out some defiance and hope to learn from any reactions and suggestions that it provokes in readers. I’m not sure yet how much I endorse any of what follows. But that’s how thinking works.

Game Theory Is Usually Qualitative

Game theory is the mathematical study of strategic interaction. (Selfish advert: I wrote a book about it.) It might seem like quantitative analysis, because it uses formal mathematics. But it is deeply qualitative as well, because its goal is not usually to analyze any exact situation, but rather to deliver changes in how we perceive classes of interaction, to produce a qualitative change in how we understand social life.

Consider as an example the classic Hawk-Dove or Chicken model of dyadic conflict. In this scenario, two individuals compete for a non-divisible resource like a nesting spot. Each individual can decide to escalate to fight over the resource. If both fight, then both risk injury, and the cost of injury may exceed the value of the resource. But if only one fights, then that individual wins the resource without risk of injury. Similarly, if you know your opponent will escalate, it is often not worth escalating.

The Hawk-Dove scenario favors doing the opposite of your opponent. And that is an interesting insight. But at the same time, the entire scenario is garbage. Where did these two individuals come from? Are there any outside options they could pursue instead? Do their choices have signal value to outside observers? Probably no strategic scenario in our lives is as simple as the Hawk-Dove game. But that’s okay. We still learn from it, because most people do not realize the implications of even simple strategic scenarios. It trains our thinking to explore them. The game is stated and solved entirely quantitatively. But its effect on our thinking is qualitative. We build intuitions by studying many such games. And we hope we can apply those intuitions, together, in more complex settings, using qualitative insight that may make us cautious where would have been reckless.

Qualitative Is Opaque Quantitative

A great thing about computers is that they are good at things that people find difficult. Like arithmetic. But they are terrible at things that children find easy. Like classifying pictures or holding a conversation. But these tasks are all computational. They are information processing. A big difference is that we just don’t understand how people classify pictures or produce conversation. We also don’t understand those flashes of insight that come from reading widely and holding in mind a breadth of examples and experiences. How do people recognize new kinds of things?

Qualitative changes in our thinking are often opaque. Quantitative thinking can in contrast be taught to a computer. We don’t yet know how to program a computer to do qualitative analysis, to generalize and analogize and brick together new insights from unrelated solutions. But if we could, it would be powerful. Likewise, we cannot teach people to simulate Hamiltonian manifolds in their heads. But we can teach people to use computers for that purpose, so that they become inputs into qualitative insights.

An analogy that flips the script here is that some machine learning techniques, like “deep” neural networks, are perhaps as opaque as human thinking. They do amazing things. But no one yet understands exactly how. Sure we understand the principle by which they work. But the same could be said for the human brain. So already we are training computers to perform valuable qualitative work. And that is potentially going to cause lots of disruption. But it might also produce lots of value. Because qualitative insight is extremely valuable, especially when it uses quantitative inputs.

I don’t mean to say that qualitative analysis is never traceable. It often is. Someone doing grounded theory can explain the steps very clearly. But still no one can program a computer to do grounded theory. There is a qualitative different in how opaque qualitative and quantitative research are.

That said, so much quantitative research is done so badly that no one can figure out what exactly was done. If there is no code and no data, then I tend to think it is qualitative data analysis wearing a quantitative costume. That’s okay. I don’t hate qualitative analysis and I don’t hate costumes. But such analysis risk sacrificing some of the unique advantages of quantitative research while simultaneously failing to realize the unique advantages of qualitative research.

Objectivity Is Subjective

A traditional distinction between qualitative and quantitative analysis is that the first is subjective and the second is objective. A “qualitative” change is a change in how we experience some phenomenon. A “quantitative” change is measured in a way that is the same for all observers. But it is almost too obvious to say it without embarrassment that every objective measurement depends upon many subjective choices. And different scholars will make difference choices, because their expertise and goals differ. A thermometer is objective, and in some sense different temperature scales are equivalent. Celsius and Fahrenheit measure the same thing, mostly. But different temperature scales produce different qualitative experiences, because people unlike computers don’t experience all numbers the same way. The point is that quantitative choices arise from qualitative goals and their effects are always at least partly qualitative as well, even if unintentionally.

This reminds me of some novelist—I can’t remember who but I think they are Israeli—who said something like “The category ‘non-fiction’ includes just those things we agree on. They are not necessarily more real.”

Good Thinking Is Qualiquantitative

Everyone does qualitative analysis. It’s just that not everyone has been trained to do it well. This is a looming problem in the sciences, since so much quantitative analysis is translated in dirty ways to qualitative insights. Should we try to be more professional about how we do that?