How many people have been killed, maimed, raped, imprisoned without due process, tortured, etc. in conflict Z, Country Q? News media focus on that number, and those of us who collect and analyze data on dissent and repression as part of our vocation
are should be well aware of the fact that the numbers reported are not only guesstimates, but guesstimates produced by interested actors (i.e., individuals representing organizations who want to shape the future course of events). Yet, we yearn for The Truth, and new technologies often produce the possibility that we will now be able to produce the truth. Fuhgeddaboutit. That’s not happening.
Why not? Because human beings maiming, killing, raping, torturing, and otherwise abusing other human beings sometimes want that information known (and even exaggerated), and other times do not want it known. Across different collections of human beings those interests vary, and even over time those interests may vary for a given collection. Thus, if you wanted to discern The Truth you would need to account for that, which puts you in the land of modeling, and models have error… So this is what I want you to burn into your brain: EVERYONE who tries to tell you something else is a charlatan selling a divining rod. A deity, should one exist, may know. We mortals cannot.
Please know that Christian Davenport did not listen to me when I tried to tell him that he was on a fool’s errand when he first launched the project that produced his book Media Bias, Perspective and State Repression. After a few years of trying to divine the truth from the multiple source soup, he embraced Kurosawa’s Rashomon: we cannot know the truth, yet must press on nonetheless. So, what are we to do? Davenport helps us understand that those of us who are interested in developing theoretical accounts of human behavior (in his case, state repression in response to dissent) need not concern ourselves with The Truth, but instead peoples’ perceptions of reality (I like the generic term “beliefs”), how those perceptions are produced, and why people take actions given those beliefs. Indeed, the BIG thing missing from the work of Charles Tilly’s 1978 masterpiece From Mobilization to Revolution was information (which informs beliefs). This is something that Ted Gurr did well (from a psychological vantage) in Why Men Rebel. (despite the fact that the vast majority of readers trivialize the arguments in that book by reducing it to the frustration–aggression mechanism qua relative deprivation upon which Gurr constructs his many hypotheses about different values of information, see chap 6). But I digress.
That backstory serves to set up my primary reason for posting: to help distinguish between projects like the Syria Tracker Crisis Map and the efforts of the Human Rights Data Analysis Group (HRDAG) to produce estimates of the number of casualties in civil conflicts (pdf). Note first that HRDAG produces estimates, not a number. Projects like the Syria Tracker Crisis Map do not produce estimates, but instead report counts. That is your first indication that HRDAG is using scientific inference (in this case, statistical inference) whereas the other project is not. Folks at projects like the Syria Tracker Crisis Map appear to be pursuing an accurate count (aka The Truth). Patrick Meier posted recently about an article in New Scientist, “Mapping the Human Cost of Syria’s Uprising” [seriously NS, the uprising is responsible for the human costs?]. He quotes this waffle:
the crisis map “could be the most accurate estimate yet of the death toll in Syria’s uprising [...].”
Setting aside the “could be,” we see the desire for truth (qua accuracy) before us. Meier continues quoting New Scientist:
Their approach? “A combination of automated data mining and crowdsourced human intelligence,” which “could provide a powerful means to assess the human cost of wars and disasters.”
And there we have the technological innovation: crowd sourcing is held to have promise for revealing The Truth (i.e., accurate counts). Unfortunately, that simply is false. We cannot know The True Count. Admittedly, the target target audience for such projects is news reporters, advocates, and politicians, none of whom are especially likely to take readily to the ontological and epistemic points of this post. Whether that excuses projects that report counts from trying to be forthright I leave for you to decide.
HRDAG is also interested in accuracy, but unlike other projects, they recognize that we cannot know The Truth, and since a census is not possible the best option is to estimate a count using statistical methods. In a series of posts that begin here they describe how they did that in Syria here. HRDAG’s methodology is scientifically defensible. People may develop alternative defensible methods. At present I am unaware of any, and it is important to underscore what I tried to explain to Davenport many years ago: multiple sources (including the crowd) cannot produce an accurate (ac)count. Which leads me to KIA’s use of Fatboy Slim’s tune: You can roll with this, or you can roll with that…
Did I just call Patrick Meier a charlatan? I don’t think so, in part because I haven’t seen him claiming the sorts of things he quotes New Scientist saying. I do know this: crowd sourcing is here to stay (assuming a relatively free web) and I plan to be involved in small ways along side those like Meier and the many, many others who are doing this kind of work. So this is definitely NOT A DISS of crowd sourcing work in the service of peoples’ rights, and especially not Meier’s work in particular. This post hopes to increase awareness among those of us doing this work that we are either engaged in scientific work or we are not, and it is important to know what the difference is. A record of verified claims is a very important thing: Amnesty International (AI) in particular has had an enormous impact by making public verified claims. What AI is very careful not to do, however, is claim to have an accurate count of anything. Projects like Syria Tracker Crisis Map should be very clear that they are reporting only verified counts (and should limit themselves to reporting only verified events), not an accurate count. A verified count will always be an undercount of the unknowable true number, and we need to do what we can to make journalists, policy makers, politicians, and the public aware of that.
In closing, if that KIA video makes you need to see Christopher Walken fly in Fatboy Slim’s “Weapon of Choice,” here tis…