The Quantitative Peace

Historical Cartography

Uri Friedman at the Atlantic has a nice piece entitled "12 Maps that Changed the World." It's based on a book by Jerry Brotton, professor of Renaissance Studies at Queen Mary University. It's worth checking out if you haven't seen it already. This article is interesting for a few reasons. First, it gives you some sense as to how views of the earth have changed over time. Second, Friedman's snippets for each map help to show how politics, culture, and religion all influenced the evolution of these views. And related to that previous idea, it points to that ever-present issue of measurement. Social scientists aren't the only ones who confront these kinds of issues, and information we now tend to assume away as being relatively objective and simple to define/convey (e.g. geography) can actually be more complicated that we might first think:

"All cultures have always believed that the map they valorize is real and true and objective and transparent," Brotton, a professor of Renaissance studies at Queen Mary University of London, told me. "All maps are always subjective.... Even today’s online geospatial applications on all your mobile devices and tablets, be they produced by Google or Apple or whoever, are still to some extent subjective maps."

There are, in other words, no perfect maps—just maps that (more-or-less) perfectly capture our understanding of the world at discrete moments in time. 

At this point in time I suspect the subjectivity associated with cartography is probably less a matter of relative size and space, and more a matter of content (i.e. borders, landmarks, etc.). Friedman addresses this a bit when he talks about Google's map-making efforts in #12.

Map #4 also stood out to me:

What's most striking about this Korean map, designed by a team of royal astronomers led by Kwon Kun, is that north is at top. "It's strange because the first map that looks recognizable to us as a Western map is a map from Korea in 1402," Brotton notes. He chalks this up to power politics in the region at the time. "In South Asian and Chinese imperial ideology, you look up northwards in respect to the emperor, and the emperor looks south to his subjects," Brotton explains.

There's also a clip from the West Wing attached to Map #11, the Peter's Projection. Something for everyone.

Peer Performance in the Review Process: Reviewer Elo

The peer review process is an imperfect construction that helps lend credibility to the publication of research. It is not the final arbiter of who is right, as cumulative research should encourage further discussion, but it is an important barrier that offers a check on research and also provides feedback as authors work to make their findings available to the scientific community and the public in general.  

Non-academic peer review exists in a variety of venues beyond just scientific research.  One such arena, online video games, may offer an insight of how to encourage active and meaningful participation in the peer review process by providing feedback to not just authors and editors, but to reviewers.  

--

The game League of Legends, created and operated by Riot Games, is one of the most popular competitive video game across the globe. It boasts a userbase of tens of millions (stats from October, 2012) and will be having its third world championship in less than a month with over $8 million at stake. The event will have thousands watching live in California and be watched live online by millions. Of note, the North American and European teams are expected to fair quite poorly against teams from China and Korea.   

With this large community, one of the persistent problems the community deals with is toxic player behavior. That is, behavior that includes aggression, foul language, threats, and taking actions that ruin the experience of the game for other players.  Such behavior, done by individuals that are typically anonymous except for a username that can be changed, is detrimental to the health of the game and deters other players from continuing to participate in the game or community. In fact, Riot just released a video yesterday to try to combat the problem by appealing to statistics and performance.  In one of their multiple methods to try and combat this problem, Riot hired social scientists to see what inducements they can offer to curb poor behavior.  One strategy they have adopted is the Tribunal. In the tribunal, other players read through the logs of an individual, who has been reported repeatedly, and vote to either pardon the person or punish them for their behavior in the game. Once enough people vote to pardon or punish an individual, the majority decision holds and that person either has nothing done to them due to being pardoned or they receive punishment for their behavior.  This punishment escalates as the player is found guilty in front of multiple tribunals and includes punishments such as a warning, losing the ability to chat in game, being temporarily banned, or having their account permanently banned from the game.

Now, the part of this whole story that is related to academic peer review is that the users who participate in this process receive a score that changes with their voting in the majority or the minority.  This score, an Elo rating, goes up when they are in the majority, and down when they are in the minority. For the tribunal, the system is a modification of the same system that calculates an individual's chess rating based on their wins and losses.  Generally, a higher score indicates better performance as judge.  This gives feedback to the judge about their consistency in evaluations relative to the rest of the community and there are even leaderboards that rank the top few hundred judges. 

--

 

This mechanism has potential for use within academic publishing.  Essentially, when a reviewer makes a recommendation to an editor about a particular manuscript, they are casting a vote for rejection, revise and resubmit, or some level of acceptance.  Naturally, these votes are not in isolation.  Given that any academic is likely to review multiple articles over the course of a year, as well as a lifetime, it would be possible to rank reviewers along a similar metric.  In academic publishing, I recommend two such scores to mark performance over binary outcomes: Reviewer agrees with Peers and Reviewer Agrees with Editor.  The first score is an indication as to whether or not a given reviewer is generally in the majority or minority on reviewing papers.  The second indication is a score that determines whether or not a reviewer is providing useful information to the editor.  Thus, it is plausible that generally a reviewer agrees with his or her peers and this informs the editor to their agreement. It is also possible that a reviewer consistently does not agree with his/her peers but editors use their views to inform their decisions.  

To provide an Elo system for reviewers would require the cooperation of one or more journals within a discipline. It is feasible for a journal to provide the scores of reviewers for just that journal if there were not cooperation within a discipline, though the sample and movement of those scores would be small as a result.  If several journals within a discipline worked together, then the information would be data rich.  Given that peer review is anonymized on two fronts, the data would likewise have to be anonymized in terms of which papers that reviewer reviewed, but this can be done by simply issuing a unique ID for a manuscript.  Then, all the journal would have to provide is a simple yes or no as to whether the reviewer was in the majority and if the reviewer was in line with the reviewers decision.  It is possible to make the data more informative if we included a range of possible deviations from the majority, but it would make sense to make the data collection as simple as possible.  Ultimately, after the scores have been computed, such scores should be made public to the discipline. 

Benefits:

To go through the process and provide such information requires some level of benefits to the discipline to be worth adopting.  There are a few different benefits for providing such data and assembling it.  First, it would provide a benchmark for newer members of the community to gauge how their decisions are related to other people in the community in a systematic way.  This is also true for members are established within a discipline.  As an Elo score goes up or down, the academic can gauge systematically how their decisions have related to others within the community.  This allows individuals to move beyond anecdotal evidence that could very well be filtered through confirmation bias and, instead, provide a direct assessment of their reviewing capabilities.

Second, it can offer a level of prestige.  Such Elo measurements can and should be broken down into lifetime and yearly scores.  These numbers then give participating reviewers a service-based metric of their performance.  If you are the highest Elo reviewer within a discipline for a given year, or over a lifetime, this would send a strong signal about your consistency. If your Elo is consistently low, this may be because you have a higher standard than your peers or, perhaps, a lower standard if you are often the lone wolf that recommends publication among two other rejection recommendations.

The third notable benefit would be to editors.  Editors have networks of reviewers that they request reviews from and have some idea of how those reviewers have performed in the past. This evaluation is generally based on memory as well as whether or not a particular article falls within the expertise domain of a reviewer and may be prone to error. In a ranking system as proposed above, a high Elo ranking on either metric (community or editor Elo) will give editors more information about whether or not they should approach new individuals (or old ones) for additional reviews.  A community wide Elo system would also give editors access to individuals within the discipline that they may not have known were active reviewers.  

There are other benefits to having a public Elo system, but I will stop at these three.

Costs:

Naturally, such Elo computations are not free.  There are two major costs I see associated with such a plan.

First is the task of data gathering as a literal cost of time and money. It would likely put another burden on editors to provide this information to those who calculate the Elo rankings.  Editors already have to tally up votes and read through decisions and convey this information to the author(s) of manuscripts. As such, I imagine this would be a small additional task for each reviewed article, but it still would be another task.  As such, editors would have to choose to take on this additional data reporting burden. However, it can be done on a case by case basis or a journal can give the data to the compiler in a lump sum. Likewise, an organization or individual would have to take the task of collecting this information and providing it publicly (likely online). 

The second potential drawback is that such a system may encourage conformity where we do not want it to exist. If reviewers become motivated to achieve a high Elo, their goal in critiquing an article may not be to determine if it is fit to be published, but may instead be to determine what their peers think of such a manuscript. This would depress individual incentives to strongly attack or defend an article.  

A more serious problem in this regard is a decision become a focal point for reviewers.  If it is common for articles to be rejected (as it is the norm for most peer reviewed journals), then reviewers that strongly care about their Elo score will reject articles as their default position.  In general, I am not convinced that such a system would override the other incentives reviewers have for reviewing scholarly work, but it is certainly possible. 

Along this line, there can be an incentive for discomformity.  If you really do not want to review articles, you can purposely tank your Elo to send a signal to Editors.  However, I imagine that people who would do this already send similar signals to editors in the present through how they review or comment on a manuscript.

--

The review system is a standard of service within academia and is expected of scholars that publish within journals and books.  Having a systematic way of showcasing consistent reviewers, in effect, offering a public incentive for continued, strong reviewers may be an interesting experiment in providing systematized feedback to reviewers.  At the very least, it is an interesting thought experiment.

Political Science and Scientists in the News

Just a quick link to this Atlantic article. It features several works by political scientists on why and when the public is likely to support military action. Complete with graphs and everything. 

 

*Picture taken from the aforementioned and linked Atlantic article. 

Follow-up

Just a quick follow-up to yesterday's post. Chad sent me a link to this paper that was recently posted on AJPS' early view. I've not read the entire article yet, but it seems apropos given the subject matter of yesterday's piece. 

History vs. Political Science? Temporally Constrained Studies and Generalizability.

So this is my first post back from a prolonged break. As I mentioned in a previous—albeit brief—entry, I’ve had a busy but enjoyable summer. I got married, defended my dissertation at the beginning of July, and my wife and I have since relocated to Tuscaloosa, Alabama where I’ve accepted a position as a post-doc. We’re both pretty excited about the move and we’re really enjoying ourselves. Having grown up in the Adirondacks in upstate New York, summers that extend beyond a two month window have a certain appeal. And now that we’re getting settled in down here, I’m slowly getting back into a routine for work. I’ve been sitting on some papers for a while now that I’m finally turning my attention to, and it’s my experience with one of these papers that motivated me to write this particular post.

I’ve been thinking recently about how we view the place that history holds in our discipline. Now, I suspect there is an initial reaction that many people (particularly quantitative political scientists) are likely to have to the word “history,” and I’m guessing that is probably one of derision. It’s not that political scientists necessarily dislike history or historians, but this reaction is conceivable given that these disciplines are marked by some important epistemological differences, and many quantitative political scientists are often taught from the outset to avoid relying on single cases when drawing more generalizable lessons and conclusions (with good reason). When we think of historians, it is often in the context of someone who provides an excruciating level of detail about one particular event, and then tries to explain its origins and broader implications/lessons. My issue is not in fact with this particular point—as social scientists, we want to be able to assess just how generalizable the relationships that we’re looking at really are. We are typically interested in exploring systematic relationships, trends, and patterns that hold over broader periods of time and apply to a wide range of actors than are many historical studies. So to be clear from the outset, I am not advocating that quantitative political science should attempt to emulate the professional approach or methods that many historians employ.

My issue, really, stems more from the assertion by many quantitative political scientists that a study that is time bound in some way is “history”. This does not apply to all political scientists, but it’s been my experience that reviewers will often look at the temporal range of a study and dismiss its if it is explicitly bound to a particular period of time that does not overlap with the present (allowing a few years of slack on just what constitutes the “present”, that is). Indeed, some seem to be under the impression that it’s not even political science if the gap between the time period covered by a study and the present is sufficiently large. For example, let’s say I’m interested in understanding the domestic determinants of pre-hegemonic US foreign policy behavior. In this particular instance, there are theoretical reasons to suppose that the behavior of the US could be different in this time period as compared to later periods. Let’s further suppose that I have theoretical expectations regarding the relationship between X and Y, and I write a paper on this relationship for the period between 1900 and 1945. I would not at all be surprised to receive reviews rejecting the paper at least in part because the reviewer(s) questioned the study’s broader relevance, given that it’s bound to a 46-year time period.  

I think these sorts of reactions expose some problems and some important assumptions that we often make as political scientists. First, not every study necessarily needs to neatly map onto the present time period—or any other arbitrarily chosen time period. There is nothing inherently unscientific about the notion that certain relationships or phenomena can only be found in a particular temporal context. I’m fairly certain people have not abandoned the study of dinosaurs simply because we no longer see T-Rex roaming around the countryside. And our field is rife with examples of research, the temporal context of which is fundamentally limited in some way. Scholars of American politics provide perhaps the clearest example of an entire sub-field that cannot be held to apply to a period extending back beyond 1789. The sub-field of international relations is similarly dependent in many ways upon the existence of the modern nation-state, which we typically trace back to 1648, and some of the most widely used only go back to 1815. Do we consider our endeavors in these areas “history” because our studies are bound to these time periods?

Also on this point, there seems to be a double standard when we consider the broader implications of our research. Taking my example from earlier, reviewers will commonly ask how the study of US foreign policy between 1900 and 1945 is relevant for today, but rarely do we consider how a study of US foreign policy from 1945–2013 informs our understanding of the 1900–1945 period. It strikes me that this is (1) perpetually moving the goalpost, and (2) that it may be the wrong goalpost. This may seem like an odd point, but what it “relevant” by current standards is something that obviously changes on a day-to-day basis, and it’s a standard that says nothing about the quality of a paper as a piece of social science, or whether or not a paper helps us to understand a particular question about a given set of relationships. Really, this only represents our own innate temporal biases, but it says nothing about how scientific a piece of research is.  If the goal is truly generalizable theoretical and empirical knowledge, in a temporal sense, then this kind of consideration should apply just as much as thinking about how a study informs our understanding of the present. Conducting a study with the purpose of expanding our knowledge of systematic relationships between societal actors is not synonymous with expanding our knowledge of systematic relationships between societal actors for the purposes of informing our understanding of the present.

And it’s not as though we don’t attempt to deal with “unique” time periods and cases in our current research. However, it’s often the case that the manner in which we deal with these cases is fairly crude. For example, we might include a dummy variable in a model to control for a time period (or characteristics of a time period) that we believe to be unique in some way. Bipolarity during the Cold War, for example. Similarly, we might include a dummy variable in our model to control for states that we believe to be unique—depending on our topic, it might be a state like Israel, Egypt, the US, or maybe a group of states like the “Great Powers”. Sometimes we might also include an interaction term to account for how the effect of one variable might be conditional upon another. 

But these approaches are not always appropriate methods for dealing with the questions that we want to answer. Dummying out a particular time period is only going to tell us whether or not a given time period or group has a higher or lower intercept than the alternative time period or group. This approach is also often atheoretical. For example, we might have a belief that a time period is somehow different, but cannot fully articulate why or how. Similarly, interaction terms with a variable capturing a particular time period, for example, are implicitly suggesting that a given relationship is time-bound in some way. However, these approaches don’t allow us to examine whether or not the remaining variables in our models also have different effects in the context of a particular time period. For example, maybe we’re interested in whether or not both regime type and economic interests have a different effect on conflict propensity during the Cold War as compared to after. This, then, would suggest that maybe splitting our sample into two time periods is the more appropriate means of addressing our question. In fact, the notion that we would dummy out a particular time period because we suspect that it’s “different” in some way, but don’t exactly know how, is exactly the reason why we would want to conduct a study that is temporally bound in the first place.

This points to another issue. I think these biases are somewhat rooted in, and reinforced by, our relatively limited access to “good” data. Almost every journal article contains passages wherein the authors attempt to assert the broader relevance of their work. In the case of international relations, it is also quite common for these articles to then proceed to test their arguments using data that is only available for the post-World War II period. Sometimes, these tests will use data for a single country—often the US. Yet the arguments the authors make are often asserted as applicable to a broader set of countries than just the US, and rarely do such papers even address their own temporal limitations. We implicitly accept the generalizability of papers in which the tests of broader theoretical arguments rely on data from an incredibly narrow and often unrepresentative set of states, but push back when a study openly acknowledges its more narrow temporal confines. Why should we automatically assume that such studies inform our understanding of international relations and state behavior in the 1800s?

This is understandable. Particularly in the field of international relations, the availability of data is exponentially greater in the post-World War II period than before. Commonly used indicators like GDP and trade either don’t exist, are often missing, or are highly inaccurate for earlier time periods. Accordingly, many of our studies focus on this 50–60 year time period—not because it somehow matters more, or because we are interested only in this particular time period, but because this is the period for which we have access to relatively abundant data sources. But even in the post-World War II time period some of the data we use can still be of questionable reliability. Accordingly, when we see an article that focuses on a much earlier time period it sticks out like a sore thumb, and reviewers will often proceed to subject that paper to a different standard than other papers—a standard that really has nothing to do with the execution of the paper or the soundness of its argument.

This knee-jerk reaction against studies that are temporally bound in some way can also have deleterious consequences for our ability to understand the world. Finding that a particular relationship between two variables holds only for a given time period can reveal new and interesting questions. For example, we have evidence that Republicans and Democrats have switched their positions on military spending over the course of the Cold War. If we were looking for a relationship between Republicans and higher military spending over the entirety of this time period, we might erroneously conclude that there is no relationship. Alternatively, the finding that this relationship is temporally bound in some way raises new questions: Why did they switch? What caused the switch? Etc.

If our goal is to continuously develop and refine our understanding of how the world works then we must think carefully about the standards we set. If that standard is that we must only examine relationships that hold for centuries at a time, then we are imposing some very serious limitations on ourselves as researchers. These kinds of “big” systematic relationships are clearly important, but the march of scientific progress is not marked exclusively in these terms. 

Tom Coburn's Grudge, Part II

Via Jennifer Diascro's Twitter feed: The Economist has an article up addressing the specific targeting of Political Science Research by Senator Coburn. The article addresses a point that I've brought up a couple of times in the past (see here for example). Specifically, why only political science?

The Economist:

The real debate seems to be over what is of value. For many Republicans the answer is nothing having to do with political science. Yet it is hard to see how that discipline is any more frivolous than—or even all that different from—economics, sociology or anthropology.Mr Coburn's measure wouldn’t touch NSF funding for other social sciences. But this is the fourth time since 2006 a Republican has singled out political science for cuts.

If we look closer at those efforts, the picture becomes a bit clearer. Mr Coburn, for example, has particularly objected to studies on American attitudes to the filibuster, at a time when Republicans are using the filibuster routinely. Mr Flake had likewise complained about research on climate-change negotiations. When Mr Coburn first proposed cutting off funding in 2009, he complained that NSF money had gone to fund The Human Rights Data Project, which, he noted, had “concluded that the United States has been ‘increasingly willing to torture enemy combatants and imprison suspected terrorists,’ leading to a worldwide increase in ‘human rights violations’ as others followed-suit.” Mr Coburn even groused that Paul Krugman had received NSF grants more than 15 years earlier. Presumably that was because Mr Krugman went on to become a liberal columnist and not because the research won him a Nobel prize.

In all of these cases, the research risked calling into question the wisdom of policies supported by the Republican Party. In none of the cases did Republicans argue that the studies were flawed. They appeared to simply object to financing research that might contradict their point of view.

I would add that my impression is that political science research is probably the one field that is most likely to draw upon and combine insights from all of these other fields. Perhaps I'm mischaracterizing the work of scholars from these other disciplines, but if this view is accurate then it further hurts the credibility of Coburn's claims to be going after wasteful spending. If the idea is to get as much bang for our buck as taxpayers, political science seems like exactly the right field to invest in. But does this matter? Probably not. Most people just don't know or care enough about this to really see the problem, and for these people Coburn's credibility is really a non-issue, so he can still claim to be simply fighting the good fight against bloated government budgets. 

Finally, I'd urge all of my Americanist and comparativist colleagues to start building up the national security implications of their work. After all, we must remain ever vigilant to safeguard our democratic institutions from infiltration by the communist scourge. Maybe ANES can just add a question asking "Are you a communist spy?" or "Are you a terrorist?" and Coburn will be satisfied. 

Tom Coburn's Grudge

So I'm a little late getting to this, but Tom Coburn is at it again—more successfully this time, it appears. The short version is that Coburn was able to get an amendment through that prohibits the National Science Foundation from funding political science research, with the exception of research that "promotes the national security or economic interests of the United States." I just have a couple of thoughts:

1. Whose economic interests are we talking about, Tom? I mean thank god government contributes such a negligible portion of Oklahoma's economy (See page 13). Sure this is a petty point, but it does speak to an important issue that political scientists have spent some time working on. What is "good" in terms of the economy—and security for that matter—is often defined in very different ways by different groups across society. So perhaps the Senator would be so kind as to clarify who will be the ultimate arbiter on that issue in the case of political science funding.  
2. Why political science again? This is not the first time we've seen this show, yet I remain unclear as to why political science specifically is being targeted. Unless I've missed something, there is no comparable amendment to eliminate funding for economics or sociological research. If this were genuinely about cutting "unnecessary" spending, I'd have to imagine that with just a little more effort on Coburn's part, he could be extend his watchful anti-waste gaze to other fields too.  Maybe I just missed this though...

Again, I've benefited personally from NSF funding (though not any longer) Though I do think the particular project I worked on was important, I realize that's not a satisfactory defense for a lot of people. But I'm not entirely sure that such a defense is required. The NSF's purpose is to advance our collective knowledge and understanding of a whole range of subjects—even if we as individuals may not understand the specific research that is being conducted at any given time in any given field, or see its immediate practical applications for that matter. And they're not just giving this money out, either. I applied for a dissertation improvement grant a couple of years ago. I spent a lot of time working on what I believe was a fairly thorough and well thought out proposal, and I was ultimately rejected. This is a highly competitive process, and the NSF isn't just handing out cash to people that put their hands out and ask for it. 

Links Galore

It's been a while since my last post, so I thought it was time to put up at least a little something:

Senate Republicans are, for the moment, blocking Chuck Hagel's nomination as Defense Secretary. Admittedly, my attention to current events has been spotty over the past couple of months, but it seems that every time I turn my attention back to this the basic rationale for opposing Hagel has changed. Initially it concerned his comments regarding Israel, then it morphed to include a means of obtaining more information about the attacks on Benghazi, and now it seems to have evolved further to include concerns over compensation Hagel received for giving some speeches since he left the Senate. Ultimately I guess I remain unsure as to how this works out for the better for Republicans—now, or in the long run. 

A meteor injured several hundred people in Russia. 

Spencer Ackerman has a piece up at Wired looking at the mistakes made by the Galactic Empire at the Battle of Hoth. There is also a broader set of responses to Ackerman's piece at Wired, and the folks at the Duck of Minerva have several followup posts debating the shortcomings of the Empire in a wider context. Itemize these posts, I will:

1. Here's Ackerman's post.
2. Wired's symposium.
3. Dan Nexon's piece for said Wired symposium.
4. Patrick Thaddeus Jackon's piece at the Duck.
5. Robert Kelly's piece at the Duck.
6. Steve Saideman piece at the Duck.
7. Patrick Thaddeus Jackson's followup at the Duck.

As these discussions have mirrored some of the lunchtime discussions I've had over the past few years with fellow bloggers Chad Clay and Michael Allen, I've enjoyed reading them immensely. I will try to update this list if there are any new additions.

 

 

"Kountry" Package for STATA

I may be late to the ball on this one, but in discussing a project that we're working on, fellow QP blogger Michael Allen recently pointed me to the "kountry" package for STATA. The package was created by Rafal Raciborski for the purposes of facilitating the creation of country codes and switching between codes provided by different sources (e.g. COW, IMF, etc.). The cite for the STATA Journal piece corresponding to the package is below:

 Raciborski, R. 2008. kountry: A Stata utility for merging cross-country data from
        multiple sources.  Stata Journal 8: 390-400

Anyone that has worked with country codes knows that this can be a pretty big pain. I've used it once so far, but it seems to work pretty well. Others may have been aware of this before now and might be aware of any bugs, problems, etc. If so, please add to the comments section any notes that you might want to make on the matter.

---

On another note, I just wanted to reiterate the earlier message that posting will continue to be sparse for a while longer. While we would all very much like to be blogging more regularly—especially in light of the recent debates, and the final upcoming foreign policy debate—everyone is currently engaged with other professional responsibilities. I believe this will prove to be time is well-spent, though, as everyone seems to be keeping busy and is pretty productive. 

NSF Funding Amendment

Via Henry Farrell at the Monkey Cage: Republican Jeff Flake of Arizona may today introduce an amendment to cut NSF funding for political science. Both APSA and Farrell call for folks to contact their representatives, and I will echo that call. I've benefitted enormously from NSF funding during my time in grad school and it funds a lot of great programs. As the Monkey Cage has previously noted, even Senator Tom Coburn, who previously campaigned to cut NSF funding for political science, has reaped the benefits of political science research, exploiting that research for his own professional/quasi-public purposes. And beyond that, I mean who cares about things like war, right? Or voting? I could go on...