The other day I posted about counts of violence, and discussed the Syria Tracker project. Souraya Tafrah, who works on the project, was kind enough to write a comment to the post, and correct some errors I had made. I wrote that ” the target audience for such projects is news reporters, advocates, and politicians,” and Tafrah wrote:
I would like to point out that our sole mission is to empowers citizens in Syria by giving them a voice to tell the world their story… at the end of the day it’s not about the numbers, it’s about giving a voice to citizens on the ground, their lost, injured or killed ones… The #1 hits we get on Syria Tracker are from within Syria, searches for name of people killed or are missing, looking over the fence to see what’s going on in other neighborhoods or regions. That’s our target audience, it’s not just limited to utility by news reporters, advocates, or politicians – although those are our audiences as well by virtue of the way we provide all our information publicly.
I very much appreciate that information as I really was thinking about it strictly from the perspective of external consumption, having given little consideration about how it might be used by family members of victims. I assume Tafrah’s point hits home with you as it did with me. Tafrah also provided a link to a Prezi that is a combination of narration and slides, writing that it better tells “what we do and why we do it.” I urge you to check it out.
That said, I have a handful of points I wish to make. First, Syria Tracker can do a much better job clarifying what they do. The About Us page does not explain what Tafrah wrote in the comment, saying only “We are strictly dedicated to supporting a humanitarian cause.” I would like to see them add content to that page, add a link to the Prezi and encourage visitors to view it.
Second, I assumed that the Project’s primary objective was “naming and shaming,” by which I mean documenting human rights violations. Thought the Prezi does not mention it, in a Jan 2012 report Tafrah references (more below) they write
A single place to pull disparate reports on human rights violations together in a crowdsourced way is crucial to make sense of the broader pattern of abuse and demonstrate a widespread and systematic attack on civilians, the legal qualification of crimes against humanity.
In the comment Tafrah continues:
I want to emphasize that we’re _not_ in the business of estimation (the NewScientist mis-quoted us initially and I believe they have since corrected the statement), we simply publish what we have been able to verify and document from the ocean of reports we have received.
and provides a link to the Full Report on their website. Looking back at this report I realized that they are careful to label all of their graphs and maps with “Reported Deaths.” I paid more attention to the New Scientist quote than I did to this page (though I did look at it), and could have been more careful.
That said, continuing the theme, I encourage Syria Tracker to add more text to their Full Report. It is almost exclusively figures: there is no discussion of methodology, no attempt to help readers understand how to interpret the figures, and what they might mean. This is a nontrivial shortcoming.
From their Tafrah writes:
we believe the numbers we have documented are only the tip of an iceberg and we have highlighted key limitations and challenges we face, but at the end of the day we are publishing the “ground truth”
I find this confusing. First, Tafrah is correct: despite the shortcomings of the Full Report, Syria Tracker has posted a set of slides that respond to the HRDAG report, and in it the first bullet point it “The Overall Number of Deaths listed by most sites may be greatly underreported.” They have also posted their methodology in a Jan 2012 paper on slideshare.net, but the three paragraphs in the methodology section discuss only technical aspects of crowdsourcing side of the project. In the results section there is no discussion of the issues I raised in my post, nor that Tafrah made in the comment.
Further, I am not sure how one navigates to that from the project website. In other word, if the highlighting that Tafrah references are those two sets of slides set, then the two of us have a different sense of the verb “to highlight.” Why do I say that? Because the average visitor to the project website (more on this below) is not going to poke around all of the reports, and is thus not terribly likely to learn that point. Here is a screenshot of the Navigation menu on the main page of the site:
It would be great if they would not only revise the About Us page to better describe their goals, but also highlight their challenges and limitations. Further, I would like to see a Methodology link in which they lay out the discussions found in some reports, and a Use & Misuse link in which they specifically identify what they take to be proper use of their data, and illustrate some misuses. Perhaps a FAQ page makes more sense–there is more than one way to address this–but the project has opportunities to better educate visitors on these issues.
The part of that quote that really confuses me is Tafrah’s claim that the project is getting at “ground truth.” This must be a disagreement about what that term means. I use it as a synonym for The Truth (what really happened), and as I explain in my post, that quantity is unknowable. Perhaps Tafrah is using it as a term to indicate that Syria Tracker is able to collect and vet information collected on the ground in Syria. If so, then my confusion disappears as Syria Tracker certainly does that, and they are quite correct to point out that this fact is one of the truly awesome aspects of the project (e.g., consider what we might know had Rwandan’s been able to access the Internet during April-June 1994 and post there).
Lastly, I want to return to the primary point of my original post and two points raised by Syria Tracker (in the Prezi and Tafrah’s comment, and then the Jan 2012 report):
remember, at the end of the day it’s not about the numbers, it’s about giving a voice to citizens on the ground, their lost, injured or killed ones. [comment] / A single place to pull disparate reports on human rights violations together in a crowdsourced way is crucial to make sense of the broader pattern of abuse [report]
We all want to have our cake and eat it to. But we owe it to one another to call each other out when we behave this way. As I noted in the opening above, I had completely missed the value of Syria Tracker. I stand corrected, and am indebted to Tafrah for taking the time to gently school me. If I may return the favor, I offer this: we should only search for patterns in biased data when we have a sense of the bias. Syria Tracker should not only improve its presentation of how and why it does what it does (hopefully I have offered some fairly easy to implement ideas), but just as importantly to take greater care in how it presents its aggregated information. If it is not about the numbers, then why present so many numbers on eye catching (and very cool!) graphs and maps?
In closing, let me be clear: projects like Syria Tracker are awesome. I am delighted that they exist, and wish to publicly thank Tafrah and the others who have worked on the project. The commitment, foresight, energy, and willingness to do hard work that it took to launch Syria Tracker is extraordinary, and I am pleased beyond words that cyberspace not only makes projects like it possible, but has permitted me to connect (via this blog) with these folks. That said, the project can be better. Should folks wish to cast stones in my direction for critiquing something without offering an alternative, I am certainly guilty as charged. But just because is done for good reasons does not mean we should shy from pointing out shortcomings. Those of us interested in improving human being’s ability to hold states accountable cannot afford to create a mutual admiration society in which we do not point out the shortcomings of ongoing efforts. I trust that the folks at Syria Tracker, and similar projects, understand that.