But I Just Want an Accurate Count

How many people have been killed, maimed, raped, imprisoned without due process, tortured, etc. in conflict Z, Country Q?  News media focus on that number, and those of us who collect and analyze data on dissent and repression as part of our vocation are should be well aware of the fact that the numbers reported are not only guesstimates, but guesstimates produced by interested actors (i.e., individuals representing organizations who want to shape the future course of events).  Yet, we yearn for The Truth, and new technologies often produce the possibility that we will now be able to produce the truth.  Fuhgeddaboutit.  That’s not happening.

Why not?  Because human beings maiming, killing, raping, torturing, and otherwise abusing other human beings sometimes want that information known (and even exaggerated), and other times do not want it known.  Across different collections of human beings those interests vary, and even over time those interests may vary for a given collection.  Thus, if you wanted to discern The Truth you would need to account for that, which puts you in the land of modeling, and models have error…  So this is what I want you to burn into your brain: EVERYONE who tries to tell you something else is a charlatan selling a divining rod.  A deity, should one exist, may know.  We mortals cannot.

Please know that Christian Davenport did not listen to me when I tried to tell him that he was on a fool’s errand when he first launched the project that produced his book Media Bias, Perspective and State Repression.  After a few years of trying to divine the truth from the multiple source soup, he embraced Kurosawa’s Rashomon: we cannot know the truth, yet must press on nonetheless.  So, what are we to do?  Davenport helps us understand that those of us who are interested in developing theoretical accounts of human behavior (in his case, state repression in response to dissent) need not concern ourselves with The Truth, but instead peoples’ perceptions of reality (I like the generic term “beliefs”), how those perceptions are produced, and why people take actions given those beliefs.  Indeed, the BIG thing missing from the work of Charles Tilly’s 1978 masterpiece From Mobilization to Revolution was information (which informs beliefs).  This is something that Ted Gurr did well (from a psychological vantage) in Why Men Rebel. (despite the fact that the vast majority of readers trivialize the arguments in that book by reducing it to the frustration–aggression mechanism qua relative  deprivation upon which Gurr constructs his many hypotheses about different values of information, see chap 6).  But I digress.

That backstory serves to set up my primary reason for posting: to help distinguish between projects like the Syria Tracker Crisis Map and the efforts of the Human Rights Data Analysis Group (HRDAG) to produce estimates of the number of casualties in civil conflicts (pdf).  Note first that HRDAG produces estimates, not a number.  Projects like the Syria Tracker Crisis Map do not produce estimates, but instead report counts.  That is your first indication that HRDAG is using scientific inference (in this case, statistical inference) whereas the other project is not.  Folks at projects like the Syria Tracker Crisis Map appear to be pursuing an accurate count (aka The Truth).  Patrick Meier posted recently about an article in New Scientist, “Mapping the Human Cost of Syria’s Uprising” [seriously NS, the uprising is responsible for the human costs?]. He quotes this waffle:

the crisis map “could be the most accurate estimate yet of the death toll in Syria’s uprising […].”

Setting aside the “could be,” we see the desire for truth (qua accuracy) before us.  Meier continues quoting New Scientist:

Their approach? “A combination of automated data mining and crowdsourced human intelligence,” which “could provide a powerful means to assess the human cost of wars and disasters.”

And there we have the technological innovation: crowd sourcing is held to have promise for revealing The Truth (i.e., accurate counts).  Unfortunately, that simply is false.  We cannot know The True Count.  Admittedly, the target target audience for such projects is news reporters, advocates, and politicians, none of whom are especially likely to take readily to the ontological and epistemic points of this post.  Whether that excuses projects that report counts from trying to be forthright I leave for you to decide.

HRDAG is also interested in accuracy, but unlike other projects, they recognize that we cannot know The Truth, and since a census is not possible the best option is to estimate a count using statistical methods.  In a series of posts that begin here they describe how they did that in Syria here.  HRDAG’s methodology is scientifically defensible.  People may develop alternative defensible methods.  At present I am unaware of any, and it is important to underscore what I tried to explain to Davenport many years ago: multiple sources (including the crowd) cannot produce an accurate (ac)count.  Which leads me to KIA’s use of Fatboy Slim’s tune: You can roll with this, or you can roll with that…

http://www.youtube.com/watch?v=jOHwjjhFTac

Did I just call Patrick Meier a charlatan?  I don’t think so, in part because I haven’t seen him claiming the sorts of things he quotes New Scientist saying.  I do know this:  crowd sourcing is here to stay (assuming a relatively free web) and I plan to be involved in small ways along side those like Meier and the many, many others who are doing this kind of  work.  So this is definitely NOT A DISS of crowd sourcing work in the service of peoples’ rights, and especially not Meier’s work in particular.  This post hopes to increase awareness among those of us doing this work that we are either engaged in scientific work or we are not, and it is important to know what the difference is.  A record of verified claims is a very important thing: Amnesty International (AI) in particular has had an enormous impact by making public verified claims.  What AI is very careful not to do, however, is claim to have an accurate count of anything.   Projects like Syria Tracker Crisis Map should be very clear that they are reporting only verified counts (and should limit themselves to reporting only verified events), not an accurate count.  A verified count will always be an undercount of the unknowable true number, and we need to do what we can to make journalists, policy makers, politicians, and the public aware of that.

In closing, if that KIA video makes you need to see Christopher Walken fly in Fatboy Slim’s “Weapon of Choice,” here tis…

http://www.youtube.com/watch?v=sMZwZiU0kKs

@WilHMoo

About Will H. Moore

I am a political science professor who also contributes to Political Violence @ a Glance and sometimes to Mobilizing Ideas . Twitter: @WilHMoo
This entry was posted in Uncategorized and tagged , , , , , . Bookmark the permalink.

5 Responses to But I Just Want an Accurate Count

  1. Mike Ward says:

    The point you are missing, it seems to me, Will, is that we can know things precisely, just not without uncertainty. And while not every type of combination is equally principled, you should at least be open to methods that produce uncertain estimates from other uncertain estimates. This is what, for example, the UN does on population estimates now. It’s what others have done in different contexts. Even “verified” counts have uncertainty, I believe.

    • Will H. Moore says:

      Thanks for commenting, and yes, your revision is very helpful: even verified allegations (such as those made by AI) are uncertain. Point well taken!

  2. Thanks for your reference to Syria Tracker and our work. I would like to point out that our sole mission is to empowers citizens in Syria by giving them a voice to tell the world their story. We receive eyewitness reports, that include all sort of things, from disease spread, to rape, killing, kidnapping, etc. Only 6% (3,138 published out of ~52,000) of the eyewitness reports submitted ot Syria Tracker have been published, of which 98% have been verified. Wrt documenting killings, I want to emphasize that we’re _not_ in the business of estimation (the NewScientist mis-quoted us initially and I believe they have since corrected the statement), we simply publish what we have been able to verify and document from the ocean of reports we have received. To-date (as of May 2, 2013) we have been able to document 65,834 killings (full report available here for your examination https://syriatracker.crowdmap.com/reports/view/3261). In our response to the Benetech report, we outlined that we believe the numbers we have documented are only the tip of an iceberg and we have highlighted key limitations and challenges we face, but at the end of the day we are publishing the “ground truth” (you can see our 2-pager response here http://www.slideshare.net/humanitariantracker/syria-tracker-benetech). We published our methods and verification process a little over 16 months ago here http://www.slideshare.net/humanitariantracker/syria-tracker-crowdsourcing-crisis-information. These methods remain approximately the same, with the addition of mining social media which we started in early 2012 that wasn’t captured in the paper (so far we have mined over 60 million tweet and over 120 thousand article). You may also want to refer to this article by the Atlantic (http://www.theatlantic.com/international/archive/2013/01/60-000-dead-in-syria-why-the-death-toll-is-likely-even-higher/266777), and remember, at the end of the day it’s not about the numbers, it’s about giving a voice to citizens on the ground, their lost, injured or killed ones. Each one of the counts has a face, name, story, and much more to tell and that’s what we intend to server! Finally, here is a Prezi that hope to better tell what we do and why we do it, http://prezi.com/mkltwfgx0ufu/life-dignity-and-digital-technology. The #1 hits we get on Syria Tracker are from within Syria, searches for name of people killed or are missing, looking over the fence to see what’s going on in other neighborhoods or regions. That’s our target audience, it’s not just limited to utility by news reporters, advocates, or politicians – although those are our audiences as well by virtue of the way we provide all our information publicly.

    Regards, ST

    • Will H. Moore says:

      Thanks for taking the time to comment. I will dig into the reports and either respond here, or in a follow up post.

  3. Pingback: Counts of Violence & Syria Tracker | Will Opines

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s