A post yesterday at openGlobalRights has generated frustration and ire among several of us who work in the data production (and analysis) salt mines. Lawrence Sáez, a political economist at U London’s SOAS, opined thusly under the headline “Human rights datasets are pointless without methodological rigour.”
In general, I am very supportive of the effort to analyze changes in human rights protections from a cross-national perspective. I believe that dataset initiatives, like CIRI, can help us develop a more nuanced understanding about such trends. However, at present, the CIRI dataset suffers from significant methodological problems that may make it useless for any meaningful statistical analysis.
Pointless? Useless? Yup, those be fightin’ words.
The hyperbolic tone of Sáez‘s post is unfortunate. Sure, hyperbole generates clicks, but it also polarizes. Polarization is fine when one is preaching to the choir and mobilizing one’s side against an opposing side. While I have no interest in peering into Sáez’s head and backing out his goals, as one who produces and uses the human rights data it is challenging to read his post as something other than an attack. And the unfortunate part is that all the reasonable/useful points he makes get lost in his presentation. They further reinforce inaccurate biases that people who neither understand data collection nor believe that data, and especially the statistical analysis of it, can teach us anything useful about human rights. When one includes phrases like “As a quantitative political scientist…” and concludes that data are pointless and useless, well, you be the judge.
You can’t post that!
Several of my colleagues have posted comments to the post, and Sáez added a response. I want to call attention to some criticism leveled at oGR for posting the piece. Amanda Murdie concludes her comment:
It is disappointing to me that a piece like this would be published at OpenGlobalRights.
Chad Clay, who recently contributed a post to the series this one is a part of, wrote:
Overall, this post feels beneath the standard of openGlobalRights. The claims made here would not withstand peer-review at the vast majority of journals, and yet, it was published on this popular website without any real vetting of the information it contains.
Those remarks made me wince a bit. Why?
As the commenters point out, Sáez is ignorant. If we set aside his tone, the post is a sophomoric effort to describe problems that definitely do exist in conflict data generally and human rights data specifically. The problem, from the vantage of experts like myself and my colleagues who commented, is that scholars like Sáez do not avail themselves of the opportunity to move beyond the sophomoric critiques that all of us who use human rights data develop when we first engage such data.
But should we call on a blog like oGR not to publish such posts? I don’t think so, and have weighed in on that here. Many of my colleagues will disagree with my view, but it seems to me that we are better off viewing such posts as an indicator of the challenge we face educating our colleagues who do not study human rights data (about which I write below). I get frustrated by posts like Sáez‘s, but I view them as teaching opportunities. Calling for censorship strikes me as a bad idea: doing so will drive the ignorant underground and make it easier for us to think we are doing a great job teaching others about how to think profitably about human rights data. Yes, I very much want people who read oGR to understand what we do, but as I argue below, what we do is complex and technical, and those who are new to the topic–even when they are trained quantitative scientists–will inevitably have reactions akin to those Sáez expressed in his post.
So, yes, Professor Sáez is a quantitative political economist and he has the training and skill needed to use statistics and data to do useful science. However, like all of us when we first encounter a new topic domain, he has a Pollyannaish expectation of what data collection is likely to look like, and is thus horrified by the apparent compromises made by those who have collected data. One hopes Professor Sáez would not walk into a chemistry lab working on measurement at the nano scale, and then write an incredulous blog post about the fantastic assumptions made to assign values to observations, but perhaps he would. He made that error here, and those of us who see it have a responsibility to point that out.
But I cannot support arguments that a blog post is “below standards” and should not have been posted. When it comes to peer review publication, yes, that is absolutely correct. But blog posts are not refereed and it is not reasonable, in my view, to imagine that they might be.
Then there is the issue of hyperbolic critique.
In a post on the issue of “mutual respect” in academic blogging I wrote:
what do you think about professional ethical conduct in the blogosphere? Quixotically wishing it away is not a conversation that interests me, but I want to encourage you to think about whether the CA problem should be addressed, and if so, how?
My own rule, thus far, has been to permit myself wide latitude when I am posting on my own blog, but limit myself to a more professional (less conversational) and more collegial (less snark) tone when writing on a collective blog. Is that a good/useful rule?
Of course, we are the norm entrepreneurs of the academic blogosphere. None of us know what function academic blogging should perform, nor what norms should govern its conduct. It seems to me it will emerge from practice. But it also seems to me useful to use this kerfuffle (and others like it) to raise the question and encourage others to weigh in. What norms do you advocate?
Newbies and human rights data
Let me now return to the issue of researchers familiar with data who have never thought about collecting human rights data. Happily, there is both blogged discussions and published research that investigate the topic and offer guidance to those who are new.
Back in 2014 I wrote a pair of snarky posts under the title “Two Rubes Walk into a Bar, Order Event Data” and followed it with another pair of posts I titled “No More Fountains of Youth/Pots of Gold: Conceptualization and Events Data.” I also co-organized (with Christian Davenport) a pair of meetings on creating conflict data, and in 2015 we released a Creating Conflict Data: Standards & Best Practices document for researchers to follow (link to PDF).
The four posts above, and the standards and best practices document, are too esoteric to be of value to people who do not have basic training in statistics, theorizing in science, and an interest in conflict data. But they are precisely the sort of thing that someone like Sáez could usefully engage. He would learn, among other things, that researchers who study human rights and use data have been publishing peer reviewed work on the types of concerns he describes since 1986.
That said, I am but one person who has contributed useful blog posts of these issues! Happily, Anita Gohdes tweeted a nice dust devil in response to Sáez. and identified several of them (though hardly a comprehensive list).
If I can muster the energy and block the time I will draft a specific response to to Sáez to try to find some wheat in his pile o’ chaff. Don’t hold your breath, but hopefully I’ll follow through.
Correction: In the initial post I botched my capitalization of the acronym for openGlobalRights. I’ve corrected that, replacing “OgR” with “oGR.”
 Not all events data are human rights data, nor are all human rights data events data. Nevertheless, there is considerable overlap, and the issues I discuss in those posts are relevant for human rights data.