The Quantitative Peace

While the IR people are away, the Americanists will play.

Editor's note: This is a guest post by Joshua N. Zingher, a doctoral candidate in the Department of Political Science at Binghamton University. The content of this post is based on Josh's paper, ‘An Analysis of the Changing Social Bases of America's Political Parties: 1952-2008’, appearing in Electoral Studies. This piece is cross-posted at the London School of Economics' USAPP Blog, which can be found here. 

 

Demographic changes mean that traditional Republican constituencies are shrinking as the Democrats’ grow.

It is difficult to discuss electoral politics in the United States without talking in terms of social groups.  Journalistic accounts of party competition often stress the important role that specific constituencies play (such as the “evangelical” or “Latino” vote) in determining the outcome of presidential elections.  A great deal of political science research has shown that these social cleavages structure party competition.  However, the political importance of these social cleavages can change, which can lead to a reshuffling of the social bases of the party coalitions.  In addition, demographic changes also have the potential to dramatically reshape the political landscape, as new groups enter the electorate and the ratio of existing groups’ changes.  I am interested in how changes in the cleavage structure and changes in the ratio of groups work in conjunction to transform electoral coalitions.  In new research, I investigated two related questions: what group memberships influence individual level voting behavior and how has the number of votes each political relevant group contributes to the party coalitions changed over time?

 

Identifying the Group Bases of Party Support 

So what groups are politically relevant?  To answer this question, I conduct a multivariate statistical analysis utilizing nearly 60 years of American National Election Survey (ANES) spanning 1952 to 2008.  I utilized group memberships as independent variables and vote choice as the dependent variable.  The battery of group memberships included variables for class, race, ethnicity, region, education, birth cohort and age. I define politically relevant group memberships as those that exert a statistically significant influence on individual level vote choice.  I ran a separate regression for each decade, which allowed me to assess whether the effect of specific group memberships have changed over time.  The substantive results of these models are displayed in Figure 1 below.

 

Figure 1 – Predicted effect of group membership on vote choice in presidential elections

 

 

The results of the analysis displayed in Figure 1 demonstrate that the following group memberships lead individuals to be consistently and statistically significantly more likely to support one party over the other: 

 

Table 1: The Core Democratic and Republican Social Groups 

Democratic Groups	Republican Groups
African Americans	Whites
Latinos	Protestants
Catholics	Weekly Church Attendees
Union Members	Southern Whites
Non-Religious	Men
College Graduates
Women Jews

 

This analysis highlights the continuity and change that characterizes the parties’ social bases.  Some groups have been stable members of a party coalition for the last 60 years.  African Americans, Catholics, union members and Jews have all been consistent members of the Democratic coalition while whites and Protestants have reliably favored the Republicans. 

Yet, other groups’ political leanings have changed considerably.  Whites in the Southern United States were once reliably Democratic voters; today they are one of the most solidly Republican constituencies.  The divide between the religiously devout and the secular has also become more politically important in recent elections.  Non-religious voters have become increasingly likely to vote for Democratic candidates while weekly church attendees have become more likely to support Republicans.  Similarly, a partisan “gender gap” emerged in the 1980s.  Men have become significantly more likely to vote for Republican candidates while women have become more likely to support Democrats.  In sum, many traditionally politically important social cleavages have endured (e.g. black/white) while other cleavages have become politically important only in recent decades (secular/religious or female/male).

           

Group Contributions and the Importance of Demographic Changes

The preceding analysis established what group memberships significantly influence vote choice, however, simply understanding what group memberships shape individual level voting behavior paints an incomplete picture of electoral change.  Winning an election is not about having core groups of loyal supporters.  Rather, parties must be able to obtain a majority of votes in order to win elections.  Thus, any process that leads to a party gaining more or less from a specific social group is potentially politically consequential.  Figures 2 and 3 show my calculations for each group’s contribution to the Democrats’ and Republicans’ party coalitions, as measured as a proportion of the party’s total votes. There have been considerable changes in the size of group contributions to the party coalitions—many of which highlight the important role that demographic shifts play in reshaping the party coalitions. 

 

Figure 2 - Group contribution as a proportion of Democrats’ total votes: 1952-2008

 

 

Figure 3 - Group contribution as a Proportion of Republicans’ total votes: 1952-2008

 

The Democratic Party has been obtaining an increasing proportion of its support from groups that are growing as proportion of the electorate, such as Latinos, college graduates and the non-religious. In addition, increasing turnout among Latinos and African Americans has also led to the size of these groups’ contributions to the Democratic coalition.  The Republican Party is being forced to confront the opposite scenario—many Republican groups, whites, Protestants and weekly church attendees, are making up a continuously shrinking proportion of the electorate.  Thus far, the Republican Party has been able to offset demographic changes via high turnout among their core groups as well as through winning an increasingly large majority of the Southern white vote.

So what are the conclusions can be drawn from this analysis? Firstly, the group sources of party support are dynamic, the political relevance of certain social cleavages ebb and flow across decades, which is evidenced by the changing relationship between group membership and vote choice.  Secondly, demographic changes can be politically consequential.  The Republican Party is currently in the undesirable position of having to find new sources of electoral support in order to remain viable in future elections.  The Republican Party’s need to find new sources of electoral support has come about not as a result of the loss of electoral support among loyal groups—rather, the Republican Party must find new sources of electoral support simply because traditionally Republican constituencies are making up a shrinking proportion of the electorate.     

 

 

 

Bipartisanship and the Foreign Policy Bureaucracy

Editor's note: A version of this piece is cross-posted at the LSE USAPP blog, and can be found here. This post is based on my recent article, The International and Domestic Sources of Bipartisanship in U.S. Foreign Policy, in Political Research Quarterly.

---

The idea that foreign policy and national security issues are somehow exempt from the partisan rancor that often characterizes domestic politics is perhaps best encapsulated by the old adage, “politics stops at the water’s edge.” In American foreign policy, nowhere is this spirit better exemplified than by President Franklin Roosevelt’s appointment of Henry L. Stimson as Secretary of War. Since World War II, Stimson and other prominent bureaucratic appointees—such as Robert Lovett and John McCloy—have come to represent this ideal. This phenomenon of bipartisan cooperation that is held to have characterized the early Cold War period has been popularly referred to as the “Cold War Consensus”, or as “Establishment” bipartisanship.

However, following the Vietnam War pronouncements of the collapse of this bipartisan consensus became commonplace. Similar claims followed the end of the Cold War, and more recently, the terrorist attacks of September 11, 2001. Indeed, such assertions have become common in discussions of American foreign policy, with the broader substantive implication being that the conduct of American foreign policy has become increasingly erratic and subject to the kind of partisan political battles that have long afflicted domestic politics. 

Some of my recent work evaluates just how closely these stylized truths match up with reality. Earlier accounts of the breakdown of bipartisanship in foreign policymaking relied largely on anecdotal accounts or individual’s assertions with little in the way of systematic evidence (see here, here, and here). Some work by political scientists has done better in approaching the concepts of bipartisanship and consensus in a more systematic way (see here, here, here, and here). Still, there are some gaps that my work tries to address. First, earlier qualitative works often focused on elite bureaucratic policymakers (like Stimson), while quantitative studies have tended to focus on congressional voting patterns or public opinion. Second, foreign policy scholars often view bipartisan cooperation through the lens of foreign threats and crises, ignoring or downplaying more basic domestic political pressures that may drive bipartisanship. My work aims to apply a more systematic form of analysis to the subject of bipartisanship in presidential appointments, while also assessing the relative importance of domestic and international factors in affecting bipartisan cooperation.

Using data on over 1,000 individual appointees to the foreign policy bureaucracy—agencies like the State Department and the Department of Defense—I identified those individual appointees who were members of the opposition party (e.g., Republican serving in a Democratic administration). Figure 1 shows the number of bipartisan appointees in a given year between 1948 and 2011. While this is only a starting point, it does not lend much support to the conventional narrative regarding the apparent collapse of bipartisanship following either Vietnam or the end of the Cold War. In fact, the early Cold War period sees some of the lowest counts observed in the data.

Figure 1 — Annual count of bipartisan appointees, 1948–2011.

However, Figure 1 does not provide us with any insights into the effects of a wider range of variables that might be influencing bipartisanship. Accordingly, I modeled the probability of observing a bipartisan appointment in a given year as a function of these different forces. Briefly stated, I find no evidence to support the idea that bipartisanship has undergone a structural shift following either Vietnam or the end of the Cold War. In fact, as is displayed in Figure 2, each party’s probability of making a bipartisan appointment has converged when comparing the pre- and post-Vietnam periods. Though it may be contrary to our expectations, this finding makes sense when we consider the domestic political environment of the early Cold War period. The McCarthy hearings, as well as the long absence of Republicans from the White House, help to explain the scarcity of bipartisan appointments during some of the early Cold War years.

 

Figure 2 — Predicted probability of a bipartisan appointment by party and time period

 

 

Similarly, I find no evidence that US conflict involvement promotes bipartisanship. On the contrary, my findings indicate conflict involvement actually has a negative effect on bipartisanship. As Figure 3 shows, the predicted probability of observing a bipartisan appointment decreases as the duration of US war involvement increases. Furthermore, these results actually match the findings of previous studies. Using different metrics, an earlier study by Peter Trubowitz and Nicole Mellow also finds a negative correlation between international crises and bipartisanship in Congressional voting. This negative relationships is likely due to the increasingly contentious and politicized nature of particularly long military engagements.

 

Figure 3 — Predicted probability of a bipartisan appointment as a function of war duration.

 

Lastly, I find that domestic political conditions play an important part in promoting bipartisanship. I argue that bipartisan appointments can serve to compensate presidents for lower levels of partisan support in Congress and aid in the construction of moderate coalitions. However, this dynamic only works when the parties are close and there are moderate legislators in Congress with whom the president and his appointees can work. But as the parties in Congress have become more polarized, moderate legislators have become increasingly scarce. Figure 4 illustrates this dynamic. At low levels of polarization, a decrease in the number of seats that the president’s party controls in Congress leads to an increase in the probability of seeing a bipartisan appointment. As polarization increases the size of this effect becomes smaller, eventually becoming negative.

 

Figure 4 — Marginal effect of a decrease in presidential support in Congress on the probability of a bipartisan appointment

Note: Gray box denotes area of statistical insignificance.

 

Overall, these findings cast doubt on some of the conventional wisdoms surrounding American foreign policy, pointing instead to the importance of the domestic political environment in promoting bipartisanship. Accordingly, my research suggests that we should expect to see less bipartisan cooperation through presidential appointments moving forward. The fact that polarization unfolds over several years at a time also suggests that this trend is unlikely to reverse itself any time soon. President Obama’s retention of Robert Gates, and his appointment of Chuck Hagel as Secretary of Defense, would seem to provide some evidence to the contrary. However, while these appointments are surely important, they are not necessarily indicative of broader trends.

 

How Do You Review?

I’ve been sending articles out for review for a few years now, and I’ve also had the opportunity to review several papers for journals. Having been through each side of the ordeal, I’ve begun to form some opinions on the peer review process as a whole. This is something other political scientists have written extensively about. For example, Michael Allen recently addressed the issue on this blog, and Nathan Jensen recently wrote an interesting piece about one of his paper’s journeys through the political science peer review process.  However, some of my recent experiences have prompted me to think about the process from the vantage point of the reviewer. Most of what I can recall reading on the subject tends to focus on the process as a whole, or discusses the process from the point of view of the submitter, rather than the reviewer. So I thought I’d reflect on my experiences and what practices I’ve tried to incorporate into my reviews.

1. Take reviews seriously: This might seem like an obvious point, and it's one Jensen makes in his post, but for anyone who has been through the process you know you can’t always take this for granted. We’ve all gotten terrible reviews in some way shape or form—some of them are condescending in their tone, some of them seem to address topics or ideas that are completely irrelevant to the paper in question (even after much reflection), some of them are incredibly short, and some of them are just wrong. And these points just concern content. Some reviewers just don’t respond to review requests from journals, and others don’t seem to get around to the review until several months after the paper was submitted.

For the most part, all of these things are well within the power of the reviewer to control. I say “for the most part” because sometimes comments or suggestions made in a review may be wrong, but I’d like to believe reviewers are not dishing out incorrect information on purpose. And maybe sometimes people just aren’t communicating their thoughts well, thereby leaving the reader with no idea as to how those thoughts apply to their paper. Also, sometimes life does get in the way, causing unexpected delays. But, quite often, you as a reviewer can control these sorts of factors.

The review process can and does put a burden on journal editors and reviewers—there’s no question about that. As Jensen notes in his post, senior scholars in particular can be swamped with review requests. But reviewing is probably one area in which scholars can have the greatest impact on the professional fortunes of other scholars. And I don’t mean to imply that reviewers should fail to reject a paper just to provide a boost to someone else’s career. Not every paper deserves to be accepted or given a revise and resubmit, but editors rely to varying degrees on the advice given to them by reviewers. We also know that there are plenty of hard working people out there who face rejection after rejection, many of which are for really silly reasons. If you’re going to reject a paper, the author deserves to know why. A flippant one-line response is not helpful to anyone.

I typically try to get my reviews back within a month or two, if possible. I'm still pretty early on in my career, at a point where not many people know much of my work, so it's probably pretty easy for me to meet this goal as compared to more senior scholars. On the one occasion that I've known in advance that I wasn't able to get reviews back in a timely manner, I’ve communicated this to the editor. It’s easy to blow this responsibility off—I mean, this isn’t your work, right? You are taking time away from your own busy schedule to review someone else’s work. Still, more likely than not you are asking others to do the same for you. 

2. Be clear and detailed: Along the lines of the previous point, a lack of detail in a review doesn’t help anyone. I once received reviews on a paper that ran the gamut from a single sentence saying to publish the article as is, to a rather lengthy, and in no uncertain terms, reject. As nice as it was to see that someone enjoyed your paper so much that they suggested accepting it, I’m not sure that such a brief reply helps. Truthfully, some of the more lengthy and critical reviews provided me with ideas for changes that I really do believe made the paper a better one.  

More often than not, papers are going to be rejected, and reviewers should know this. Even if you like a paper so much that you want to see it accepted as is, or with minor revisions, there will usually be at least one other reviewer on a given paper, and it is quite possible that this other reviewer will have a very different take. No paper is perfect, so it can be helpful to know why a reviewer believes that the strengths of a paper outweigh any weaknesses. Especially when other reviewers are less supportive. It’s also helpful to be able to compare reviews to see if there are any common themes/complaints that are popping up across readers. Given the high rate of rejection, I think it’s important to remember that it’s not really just about the one or two reviews that the author receives on a given round of submission. Setting aside for a moment the fact that one person can get the same paper to review over the course of that paper’s journey through the review process, if reviewers are clearly raising the same concerns through multiple rounds of reviews that might be a pretty good signal that something needs to be changed. Sometimes these sorts of things are easy to spot from one set of reviews, and at other times they’re not, but the authors can’t tell if the reviewer doesn’t provide the detail. Given the high probability of rejection, if you, as a reviewer, think that a paper's strengths outweigh its weaknesses, a failure to highlight those very features could result in them being erased in future rounds of reviews.

I’ve gotten to the point where I offer a brief statement at the beginning of the review in which I explicitly state whether or not I’m recommending acceptance, rejection, etc. I also try to offer a brief summation of what factors I’m considering in making this recommendation. I also try as hard as possible to not waver in my judgment—if you have “mixed feelings” or you “could be convinced” that a paper could be worthy of publication, don’t try to pass the buck to the editor and the other reviewers. Just recommend an R&R and move on. Nothing says you have to accept the second or third round of revisions, but given how competitive the review process is, this kind of waffling is effectively the same thing as rejecting the paper outright, and it's probably not all that helpful to the author.

Beyond the introduction, I also try to clearly identify each individual point that I want to address. And yes, this often involves an enumerated list and/or bullet points. To some extent I think this is just a matter of style. However, it can be difficult in a meandering review to discern the exact points that the reviewer wants you to address, and as much as it is the author’s responsibility to take their revisions seriously, it is also the reviewer’s responsibility to make clear those points that need addressing. Furthermore, we hold ourselves to a pretty high standard when it comes to organizing our thoughts and arguments in the context of an academic paper, so why shouldn’t it bethe same in reviews? It is, after all, all part of the same, broader, knowledge-generating process.

3. Offer constructive criticism: Again, this seems like an obvious one, but you can’t take it for granted. We spend an awful lot of time in college and graduate school learning/practicing how to pick apart other people’s arguments. So while pointing out the flaws in someone’s article is certainly an important part of the review process, I also think it’s equally important to be able to point out its strengths.

For the most part, my experience on this point leads me to believe that most reviewers are pretty good about this. However, I have received reviews that are highly critical, with little in the way of constructive commentary. "Complainy" might be the most appropriate made-up adjective to describe these reviews. If you as a reviewer think that there are some deficiencies with an argument or method, you need to spell that out for the author. If such deficiencies were clear to all, then they probably wouldn’t be in the paper to begin with. Further, I think the degree to which any flaw is damning to the overall article is contingent upon the availability of feasible alternatives. If you don’t like a measure that the author uses, don’t just complain and then move on. At the very least, ask the author to clarify whether or not superior measures are actually available. If you are aware of one that you believe is superior, then please share it and explain why you believe it to be better suited to the analysis at hand. Again, this is a collaborative venture, and part of the review process is being able to access the knowledge of other scholars.

I would also say that personally, as a junior scholar, I really do appreciate a good review when I get one. Some people have clearly taken the time to think about my work, and I appreciate it. As I said before, I’ve been fortunate enough to get some great feedback that I have subsequently incorporated into my work, and I do think the work is better for it. Criticism accompanied by constructive feedback and advice generally seems to be more helpful than merely pointing out the faults of a particular project.

4. Be transparent when it comes to your own limitations: As I stated above, sometimes reviewer comments are just wrong. Sometimes this is an honest mistake, and the reviewer is trying to offer sincere advice or criticism, but just gets it wrong. But, as an author this can be incredibly annoying—particularly if some factually incorrect statement could have been a major contributing factor to your paper’s rejection. If I’m reviewing a paper and I’m unclear about a measure, statistic, or method, I typically try to be entirely transparent when it comes to my own ignorance on the matter. Now I'm pretty confident in the substantive and methodological training that I received in graduate school, but this doesn’t mean I’ve mastered every analytical approach or estimation technique. Nor does it mean that I can’t learn even more about those topics and methods that I have been exposed to. Indeed, I’ve learned a lot over the past seven years, but as this SMBC comic so nicely conveys, sometimes expanding your knowledge base can raise new questions. So, when you think you don’t know, or you know you don’t know, just say, “I don’t know.” Feel free to even pose a question on the issue to the author in your review. After all, there is a certain irony in the idea that certain people, having devoted themselves to an industry that (ostensibly) is based on the discovery and accumulation of knowledge, would be so reluctant to utter those three words.

I'm sure senior scholars may have a different take on some of these issues. As I said, I'm still very early on in my career, so I'm not facing the same kinds of pressures from as many sources as they are. I don't have to deal with dissertation committees or a never ending stream of review requests. Still, I don't mind reviews all that much. They can provide me with some insights into what people in my general area of expertise are working on, and they often lead me to read works that I've not previously noticed or thought much about. But perhaps my views will change given time. 

 

 

 

 

 

Peer Performance in the Review Process: Reviewer Elo

The peer review process is an imperfect construction that helps lend credibility to the publication of research. It is not the final arbiter of who is right, as cumulative research should encourage further discussion, but it is an important barrier that offers a check on research and also provides feedback as authors work to make their findings available to the scientific community and the public in general.  

Non-academic peer review exists in a variety of venues beyond just scientific research.  One such arena, online video games, may offer an insight of how to encourage active and meaningful participation in the peer review process by providing feedback to not just authors and editors, but to reviewers.  

--

The game League of Legends, created and operated by Riot Games, is one of the most popular competitive video game across the globe. It boasts a userbase of tens of millions (stats from October, 2012) and will be having its third world championship in less than a month with over $8 million at stake. The event will have thousands watching live in California and be watched live online by millions. Of note, the North American and European teams are expected to fair quite poorly against teams from China and Korea.   

With this large community, one of the persistent problems the community deals with is toxic player behavior. That is, behavior that includes aggression, foul language, threats, and taking actions that ruin the experience of the game for other players.  Such behavior, done by individuals that are typically anonymous except for a username that can be changed, is detrimental to the health of the game and deters other players from continuing to participate in the game or community. In fact, Riot just released a video yesterday to try to combat the problem by appealing to statistics and performance.  In one of their multiple methods to try and combat this problem, Riot hired social scientists to see what inducements they can offer to curb poor behavior.  One strategy they have adopted is the Tribunal. In the tribunal, other players read through the logs of an individual, who has been reported repeatedly, and vote to either pardon the person or punish them for their behavior in the game. Once enough people vote to pardon or punish an individual, the majority decision holds and that person either has nothing done to them due to being pardoned or they receive punishment for their behavior.  This punishment escalates as the player is found guilty in front of multiple tribunals and includes punishments such as a warning, losing the ability to chat in game, being temporarily banned, or having their account permanently banned from the game.

Now, the part of this whole story that is related to academic peer review is that the users who participate in this process receive a score that changes with their voting in the majority or the minority.  This score, an Elo rating, goes up when they are in the majority, and down when they are in the minority. For the tribunal, the system is a modification of the same system that calculates an individual's chess rating based on their wins and losses.  Generally, a higher score indicates better performance as judge.  This gives feedback to the judge about their consistency in evaluations relative to the rest of the community and there are even leaderboards that rank the top few hundred judges. 

--

 

This mechanism has potential for use within academic publishing.  Essentially, when a reviewer makes a recommendation to an editor about a particular manuscript, they are casting a vote for rejection, revise and resubmit, or some level of acceptance.  Naturally, these votes are not in isolation.  Given that any academic is likely to review multiple articles over the course of a year, as well as a lifetime, it would be possible to rank reviewers along a similar metric.  In academic publishing, I recommend two such scores to mark performance over binary outcomes: Reviewer agrees with Peers and Reviewer Agrees with Editor.  The first score is an indication as to whether or not a given reviewer is generally in the majority or minority on reviewing papers.  The second indication is a score that determines whether or not a reviewer is providing useful information to the editor.  Thus, it is plausible that generally a reviewer agrees with his or her peers and this informs the editor to their agreement. It is also possible that a reviewer consistently does not agree with his/her peers but editors use their views to inform their decisions.  

To provide an Elo system for reviewers would require the cooperation of one or more journals within a discipline. It is feasible for a journal to provide the scores of reviewers for just that journal if there were not cooperation within a discipline, though the sample and movement of those scores would be small as a result.  If several journals within a discipline worked together, then the information would be data rich.  Given that peer review is anonymized on two fronts, the data would likewise have to be anonymized in terms of which papers that reviewer reviewed, but this can be done by simply issuing a unique ID for a manuscript.  Then, all the journal would have to provide is a simple yes or no as to whether the reviewer was in the majority and if the reviewer was in line with the reviewers decision.  It is possible to make the data more informative if we included a range of possible deviations from the majority, but it would make sense to make the data collection as simple as possible.  Ultimately, after the scores have been computed, such scores should be made public to the discipline. 

Benefits:

To go through the process and provide such information requires some level of benefits to the discipline to be worth adopting.  There are a few different benefits for providing such data and assembling it.  First, it would provide a benchmark for newer members of the community to gauge how their decisions are related to other people in the community in a systematic way.  This is also true for members are established within a discipline.  As an Elo score goes up or down, the academic can gauge systematically how their decisions have related to others within the community.  This allows individuals to move beyond anecdotal evidence that could very well be filtered through confirmation bias and, instead, provide a direct assessment of their reviewing capabilities.

Second, it can offer a level of prestige.  Such Elo measurements can and should be broken down into lifetime and yearly scores.  These numbers then give participating reviewers a service-based metric of their performance.  If you are the highest Elo reviewer within a discipline for a given year, or over a lifetime, this would send a strong signal about your consistency. If your Elo is consistently low, this may be because you have a higher standard than your peers or, perhaps, a lower standard if you are often the lone wolf that recommends publication among two other rejection recommendations.

The third notable benefit would be to editors.  Editors have networks of reviewers that they request reviews from and have some idea of how those reviewers have performed in the past. This evaluation is generally based on memory as well as whether or not a particular article falls within the expertise domain of a reviewer and may be prone to error. In a ranking system as proposed above, a high Elo ranking on either metric (community or editor Elo) will give editors more information about whether or not they should approach new individuals (or old ones) for additional reviews.  A community wide Elo system would also give editors access to individuals within the discipline that they may not have known were active reviewers.  

There are other benefits to having a public Elo system, but I will stop at these three.

Costs:

Naturally, such Elo computations are not free.  There are two major costs I see associated with such a plan.

First is the task of data gathering as a literal cost of time and money. It would likely put another burden on editors to provide this information to those who calculate the Elo rankings.  Editors already have to tally up votes and read through decisions and convey this information to the author(s) of manuscripts. As such, I imagine this would be a small additional task for each reviewed article, but it still would be another task.  As such, editors would have to choose to take on this additional data reporting burden. However, it can be done on a case by case basis or a journal can give the data to the compiler in a lump sum. Likewise, an organization or individual would have to take the task of collecting this information and providing it publicly (likely online). 

The second potential drawback is that such a system may encourage conformity where we do not want it to exist. If reviewers become motivated to achieve a high Elo, their goal in critiquing an article may not be to determine if it is fit to be published, but may instead be to determine what their peers think of such a manuscript. This would depress individual incentives to strongly attack or defend an article.  

A more serious problem in this regard is a decision become a focal point for reviewers.  If it is common for articles to be rejected (as it is the norm for most peer reviewed journals), then reviewers that strongly care about their Elo score will reject articles as their default position.  In general, I am not convinced that such a system would override the other incentives reviewers have for reviewing scholarly work, but it is certainly possible. 

Along this line, there can be an incentive for discomformity.  If you really do not want to review articles, you can purposely tank your Elo to send a signal to Editors.  However, I imagine that people who would do this already send similar signals to editors in the present through how they review or comment on a manuscript.

--

The review system is a standard of service within academia and is expected of scholars that publish within journals and books.  Having a systematic way of showcasing consistent reviewers, in effect, offering a public incentive for continued, strong reviewers may be an interesting experiment in providing systematized feedback to reviewers.  At the very least, it is an interesting thought experiment.

Follow-up

Just a quick follow-up to yesterday's post. Chad sent me a link to this paper that was recently posted on AJPS' early view. I've not read the entire article yet, but it seems apropos given the subject matter of yesterday's piece. 

History vs. Political Science? Temporally Constrained Studies and Generalizability.

So this is my first post back from a prolonged break. As I mentioned in a previous—albeit brief—entry, I’ve had a busy but enjoyable summer. I got married, defended my dissertation at the beginning of July, and my wife and I have since relocated to Tuscaloosa, Alabama where I’ve accepted a position as a post-doc. We’re both pretty excited about the move and we’re really enjoying ourselves. Having grown up in the Adirondacks in upstate New York, summers that extend beyond a two month window have a certain appeal. And now that we’re getting settled in down here, I’m slowly getting back into a routine for work. I’ve been sitting on some papers for a while now that I’m finally turning my attention to, and it’s my experience with one of these papers that motivated me to write this particular post.

I’ve been thinking recently about how we view the place that history holds in our discipline. Now, I suspect there is an initial reaction that many people (particularly quantitative political scientists) are likely to have to the word “history,” and I’m guessing that is probably one of derision. It’s not that political scientists necessarily dislike history or historians, but this reaction is conceivable given that these disciplines are marked by some important epistemological differences, and many quantitative political scientists are often taught from the outset to avoid relying on single cases when drawing more generalizable lessons and conclusions (with good reason). When we think of historians, it is often in the context of someone who provides an excruciating level of detail about one particular event, and then tries to explain its origins and broader implications/lessons. My issue is not in fact with this particular point—as social scientists, we want to be able to assess just how generalizable the relationships that we’re looking at really are. We are typically interested in exploring systematic relationships, trends, and patterns that hold over broader periods of time and apply to a wide range of actors than are many historical studies. So to be clear from the outset, I am not advocating that quantitative political science should attempt to emulate the professional approach or methods that many historians employ.

My issue, really, stems more from the assertion by many quantitative political scientists that a study that is time bound in some way is “history”. This does not apply to all political scientists, but it’s been my experience that reviewers will often look at the temporal range of a study and dismiss its if it is explicitly bound to a particular period of time that does not overlap with the present (allowing a few years of slack on just what constitutes the “present”, that is). Indeed, some seem to be under the impression that it’s not even political science if the gap between the time period covered by a study and the present is sufficiently large. For example, let’s say I’m interested in understanding the domestic determinants of pre-hegemonic US foreign policy behavior. In this particular instance, there are theoretical reasons to suppose that the behavior of the US could be different in this time period as compared to later periods. Let’s further suppose that I have theoretical expectations regarding the relationship between X and Y, and I write a paper on this relationship for the period between 1900 and 1945. I would not at all be surprised to receive reviews rejecting the paper at least in part because the reviewer(s) questioned the study’s broader relevance, given that it’s bound to a 46-year time period.  

I think these sorts of reactions expose some problems and some important assumptions that we often make as political scientists. First, not every study necessarily needs to neatly map onto the present time period—or any other arbitrarily chosen time period. There is nothing inherently unscientific about the notion that certain relationships or phenomena can only be found in a particular temporal context. I’m fairly certain people have not abandoned the study of dinosaurs simply because we no longer see T-Rex roaming around the countryside. And our field is rife with examples of research, the temporal context of which is fundamentally limited in some way. Scholars of American politics provide perhaps the clearest example of an entire sub-field that cannot be held to apply to a period extending back beyond 1789. The sub-field of international relations is similarly dependent in many ways upon the existence of the modern nation-state, which we typically trace back to 1648, and some of the most widely used only go back to 1815. Do we consider our endeavors in these areas “history” because our studies are bound to these time periods?

Also on this point, there seems to be a double standard when we consider the broader implications of our research. Taking my example from earlier, reviewers will commonly ask how the study of US foreign policy between 1900 and 1945 is relevant for today, but rarely do we consider how a study of US foreign policy from 1945–2013 informs our understanding of the 1900–1945 period. It strikes me that this is (1) perpetually moving the goalpost, and (2) that it may be the wrong goalpost. This may seem like an odd point, but what it “relevant” by current standards is something that obviously changes on a day-to-day basis, and it’s a standard that says nothing about the quality of a paper as a piece of social science, or whether or not a paper helps us to understand a particular question about a given set of relationships. Really, this only represents our own innate temporal biases, but it says nothing about how scientific a piece of research is.  If the goal is truly generalizable theoretical and empirical knowledge, in a temporal sense, then this kind of consideration should apply just as much as thinking about how a study informs our understanding of the present. Conducting a study with the purpose of expanding our knowledge of systematic relationships between societal actors is not synonymous with expanding our knowledge of systematic relationships between societal actors for the purposes of informing our understanding of the present.

And it’s not as though we don’t attempt to deal with “unique” time periods and cases in our current research. However, it’s often the case that the manner in which we deal with these cases is fairly crude. For example, we might include a dummy variable in a model to control for a time period (or characteristics of a time period) that we believe to be unique in some way. Bipolarity during the Cold War, for example. Similarly, we might include a dummy variable in our model to control for states that we believe to be unique—depending on our topic, it might be a state like Israel, Egypt, the US, or maybe a group of states like the “Great Powers”. Sometimes we might also include an interaction term to account for how the effect of one variable might be conditional upon another. 

But these approaches are not always appropriate methods for dealing with the questions that we want to answer. Dummying out a particular time period is only going to tell us whether or not a given time period or group has a higher or lower intercept than the alternative time period or group. This approach is also often atheoretical. For example, we might have a belief that a time period is somehow different, but cannot fully articulate why or how. Similarly, interaction terms with a variable capturing a particular time period, for example, are implicitly suggesting that a given relationship is time-bound in some way. However, these approaches don’t allow us to examine whether or not the remaining variables in our models also have different effects in the context of a particular time period. For example, maybe we’re interested in whether or not both regime type and economic interests have a different effect on conflict propensity during the Cold War as compared to after. This, then, would suggest that maybe splitting our sample into two time periods is the more appropriate means of addressing our question. In fact, the notion that we would dummy out a particular time period because we suspect that it’s “different” in some way, but don’t exactly know how, is exactly the reason why we would want to conduct a study that is temporally bound in the first place.

This points to another issue. I think these biases are somewhat rooted in, and reinforced by, our relatively limited access to “good” data. Almost every journal article contains passages wherein the authors attempt to assert the broader relevance of their work. In the case of international relations, it is also quite common for these articles to then proceed to test their arguments using data that is only available for the post-World War II period. Sometimes, these tests will use data for a single country—often the US. Yet the arguments the authors make are often asserted as applicable to a broader set of countries than just the US, and rarely do such papers even address their own temporal limitations. We implicitly accept the generalizability of papers in which the tests of broader theoretical arguments rely on data from an incredibly narrow and often unrepresentative set of states, but push back when a study openly acknowledges its more narrow temporal confines. Why should we automatically assume that such studies inform our understanding of international relations and state behavior in the 1800s?

This is understandable. Particularly in the field of international relations, the availability of data is exponentially greater in the post-World War II period than before. Commonly used indicators like GDP and trade either don’t exist, are often missing, or are highly inaccurate for earlier time periods. Accordingly, many of our studies focus on this 50–60 year time period—not because it somehow matters more, or because we are interested only in this particular time period, but because this is the period for which we have access to relatively abundant data sources. But even in the post-World War II time period some of the data we use can still be of questionable reliability. Accordingly, when we see an article that focuses on a much earlier time period it sticks out like a sore thumb, and reviewers will often proceed to subject that paper to a different standard than other papers—a standard that really has nothing to do with the execution of the paper or the soundness of its argument.

This knee-jerk reaction against studies that are temporally bound in some way can also have deleterious consequences for our ability to understand the world. Finding that a particular relationship between two variables holds only for a given time period can reveal new and interesting questions. For example, we have evidence that Republicans and Democrats have switched their positions on military spending over the course of the Cold War. If we were looking for a relationship between Republicans and higher military spending over the entirety of this time period, we might erroneously conclude that there is no relationship. Alternatively, the finding that this relationship is temporally bound in some way raises new questions: Why did they switch? What caused the switch? Etc.

If our goal is to continuously develop and refine our understanding of how the world works then we must think carefully about the standards we set. If that standard is that we must only examine relationships that hold for centuries at a time, then we are imposing some very serious limitations on ourselves as researchers. These kinds of “big” systematic relationships are clearly important, but the march of scientific progress is not marked exclusively in these terms. 

Tom Coburn's Grudge, Part II

Via Jennifer Diascro's Twitter feed: The Economist has an article up addressing the specific targeting of Political Science Research by Senator Coburn. The article addresses a point that I've brought up a couple of times in the past (see here for example). Specifically, why only political science?

The Economist:

The real debate seems to be over what is of value. For many Republicans the answer is nothing having to do with political science. Yet it is hard to see how that discipline is any more frivolous than—or even all that different from—economics, sociology or anthropology.Mr Coburn's measure wouldn’t touch NSF funding for other social sciences. But this is the fourth time since 2006 a Republican has singled out political science for cuts.

If we look closer at those efforts, the picture becomes a bit clearer. Mr Coburn, for example, has particularly objected to studies on American attitudes to the filibuster, at a time when Republicans are using the filibuster routinely. Mr Flake had likewise complained about research on climate-change negotiations. When Mr Coburn first proposed cutting off funding in 2009, he complained that NSF money had gone to fund The Human Rights Data Project, which, he noted, had “concluded that the United States has been ‘increasingly willing to torture enemy combatants and imprison suspected terrorists,’ leading to a worldwide increase in ‘human rights violations’ as others followed-suit.” Mr Coburn even groused that Paul Krugman had received NSF grants more than 15 years earlier. Presumably that was because Mr Krugman went on to become a liberal columnist and not because the research won him a Nobel prize.

In all of these cases, the research risked calling into question the wisdom of policies supported by the Republican Party. In none of the cases did Republicans argue that the studies were flawed. They appeared to simply object to financing research that might contradict their point of view.

I would add that my impression is that political science research is probably the one field that is most likely to draw upon and combine insights from all of these other fields. Perhaps I'm mischaracterizing the work of scholars from these other disciplines, but if this view is accurate then it further hurts the credibility of Coburn's claims to be going after wasteful spending. If the idea is to get as much bang for our buck as taxpayers, political science seems like exactly the right field to invest in. But does this matter? Probably not. Most people just don't know or care enough about this to really see the problem, and for these people Coburn's credibility is really a non-issue, so he can still claim to be simply fighting the good fight against bloated government budgets. 

Finally, I'd urge all of my Americanist and comparativist colleagues to start building up the national security implications of their work. After all, we must remain ever vigilant to safeguard our democratic institutions from infiltration by the communist scourge. Maybe ANES can just add a question asking "Are you a communist spy?" or "Are you a terrorist?" and Coburn will be satisfied. 

Of QP Publications and International Relations

Two new pieces from QP authors.  First, Michael Flynn has a solo-authored piece in an upcoming Foreign Policy Analysis that has an early view avaible. Abstract: 

How does political competition among domestic actors influence foreign policy choice? Studies examining these questions often focus on the role of economic or partisan interests, and how they influence the preferences of decision makers who are subject to electoral institutions and pressures of their constituents. Less attention has been paid to how the preferences of other influential but unelected actors influence state behavior. I examine the influence of one such group by looking at how American military leaders shape decisions on military spending and force structure, while also examining how these decisions have been affected by changes to the institutions governing civil–military relations. Results indicate that military leaders occupying key positions can influence defense spending priorities in favor of their respective branches. Results also suggest the influence of military leaders has changed and is conditional upon the institutions governing the relationships between civilian decision makers and military leaders.

Chad Clay, Michael Flynn, and Colin Barry have also landed an International Studies Quarterly article (currently available in early view) which argues the following: 

Nonstate actors, such as international non-governmental organizations (INGOs) and multinational corporations (MNCs), have attained an increasingly prominent role in modern world affairs. While previous research has focused on these actors’ respective interactions with states, little attention has been paid to their interactions with each other. In this paper, we examine the extent to which the decisions of private actors seeking to invest abroad are affected by the reputational costs of doing business in countries publicly targeted by human rights activists. We find that ‘‘naming and shaming’’ by human rights INGOs tends to reduce the amount of foreign direct investment received by developing states, providing evidence that INGO activities affect the behavior of MNCs. An additional implication of our findings is that shaming by INGOs can impose real costs on targeted states in the form of lost investment.

Chuck Hagel and Bipartisanship in Foreign Policy

The news surrounding Obama's nomination of Chuck Hagel and John Brennan as Secretary of Defense and Director of the CIA (respectively) has primarily been focused on the "controversial" Hagel's previous statements and positions. Republican commentators have been complaining about Hagel's nomination, claiming that his nomination sends a powerful signal that the Obama administration is putting more "daylight" between the US and Israel, among other things. Steve Clemmons at The Atlantic has also reviewed several arguments for and against Hagel's nomination. The turf on that subject has already been covered pretty thoroughly, so I'll leave the issue be for now. What has received less attention, however, is that Hagel's nomination represents a continuation of a fairly long history of Democrats appointing Republicans to positions within the foreign policy bureaucracy. 

In 1940 FDR appointed Henry Stimson and Frank Knox—both Republicans—to be his Secretary of War and Secretary of the Navy (again, respectively). Stimson's appointment in particular is generally regarded as marking the onset of the bipartisan consensus that characterized the early Cold War period. Having previously served as Secretary of State under President Hoover and Secretary of War under Taft, Stimson was already a widely recognized figure in the foreign policymaking community (such as it was back then). Along with Stimson came John McCloy and Robert Lovett, both of whom were also Republicans and both of whom served as Stimson's assistants in the War Department.

By bringing these Republicans into his administration, Roosevelt was attempting to build support among opposition Republicans for increasing US activism in World War II. Primarily this encompassed moderate internationalist Republicans from the Northeast. Even prior to his nomination, Stimson and his mentor, Elihu Root, himself previously a Secretary of War and Secretary of State, had petitioned Congressional leadership to introduce some form of draft bill to prepare the US for involvement in the growing conflicts in Europe and East Asia. Stimson and Root had both been devout internationalists in the years leading up to the Second World War, and their appointment marked what many historians have labeled as the beginning of the "Eastern Establishment" in foreign policymaking. The Establishment narrative, broadly speaking, tends to emphasize bipartisanship in the foreign policy bureaucracy specifically, but it also encompasses themes of elite influence, economic interests, and political polarization in foreign policymaking.

This general narrative has been present for several decades in the US foreign policy literature, but it's largely been isolated to the qualitative political science and historical literatures. Quantitative political science research has delved into the more narrow issue of bipartisanship, but these studies have tended to focus on Congress. Accordingly, the central focus of this large body of work has gone unexamined in a systematic sense, and part of my dissertation research has attempted to address this gap. This is really all a long-winded way of saying that the Hagel nomination provides me with a timely opportunity to briefly share some of my work on the subject. Though I don't want to give away the farm here, this seems like a fine time to chime in. 

Despite the recognition that the foreign policy bureaucracy—indeed, the bureaucracy more broadly—plays a fairly prominent role in the policymaking process, we don't really have much in the way of systematic quantitative research on the subject. I've tried to get at some of these issues in my dissertation by collecting some new data on appointees to foreign policymaking positions within the foreign policy bureaucracy. I primarily focus on the State Department and the Defense Department, but I also focus on some other agencies like the CIA. In total, I've collected data on some 1,000+ individuals serving in these posts from 1948 through the present. One of the primary characteristics that I coded for each individual was what, if any, party affiliation that individual had. As a way to get at trends in bipartisanship within the foreign policy bureaucracy, the graph below shows the percentage of appointments included in the data set that are bipartisan in nature.* 

Some interesting points: Perhaps not surprisingly, the Truman administration represents an all-time high in the data. This is partially a function of the fact that Truman inherited many of FDR's appointments, but it's also partially due to the fact that the foreign policy bureaucracy was undergoing some pretty radical changes during this time period. In particular, the size of these agencies grew dramatically during the World War II period. Coding rules are also partially responsible, as I focused on individuals occupying positions at the assistant secretary level and up (where appropriate). By the early to mid-1950s, however, the rate of growth slows a bit and, while still increasing in size, it does so more slowly than in the first few years of the Cold War. The crash at the beginning of the Eisenhower administration might reflect some of the antics of Joseph McCarthy and Secretary Dulles, both of whom were more than a little keen to go after anyone remotely suspected of being communists. It could also be a function of the Republicans having been out of power for about two decades. Also of note, the late 1970s do see a slight slump in the level of bipartisanship, which is potentially in line with arguments concerning the impact of Vietnam on bipartisanship more generally. The Clinton administration sees a fairly sharp spike near the end, and the Bush administration also sees a dip in its later years, while the incoming Obama administration sees a slight increase.

More broadly, the work that I've done so far suggests a few general trends. First, I find some evidence that, all else equal, Democrats tend to be more likely than Republicans to make bipartisan appointments to the foreign policy bureaucracy—Even controlling for the Truman administration. Other models suggest this difference is more acute in the early Cold War period (again, also while controlling for Truman). Additionally, I also find some evidence to support the conventional wisdom concerning the use of bipartisan appointments as a means of building moderate coalitions and as a response to domestic political conditions. In particular Congressional support matters, but is perhaps a little more nuanced than the historical accounts often suggest, as the availability of moderates in Congress also seems to condition this relationship. It also appears as though Presidents make these appointments with an eye towards governing more broadly, as the composition of the Senate alone appears to have little impact.

So what? Well, with respect to current news my work would suggest that Obama's retention of Bob Gates and his appointment of Chuck Hagel are actually very much in line with broader trends in Democratic foreign policy appointments since World War II. Robert Lovett—formerly Stimson's assistant at the War Department—went on to serve as Truman's Secretary of Defense. Kennedy subsequently offered Lovett his choice of State, Defense, and Treasury. Though Lovett turned him down, Kennedy did end up appointing Robert McNamara and McGeorge Bundy—both Republicans—to important posts. While there are certainly other lesser-known bipartisan appointees, these folks are some of the more prominent. Furthermore, these trends emerged well before the Republicans had fully taken on the identity of the party of national security. Still, even in the early Cold War period the Republicans did attack Democrats for being soft on Communism, so this vulnerability was present in some incarnation. Accordingly, Democratic administrations perhaps had more to gain from trying to draw in Republicans than vice versa, as such appointments could confer some measure of credibility. 

Moving forward it will be interesting to see how these trends continue to develop. Despite the much publicized opposition to Hagel's appointment it is entirely possible that this is just media hype and Republican chest-thumping. David Karol at the Monkey Cage recently wrote about the House Republicans' position-taking when voting on the recent Fiscal Cliff deal. So it's easy to imagine that the Republican outcry over Hagel's nomination is really going to wither when it comes time for the Senate to vote. Though these situations are not completely comparable, giving Obama a hard time seems to be a means of gaining currency in the Republican Party these days. Beyond the current confirmation battle, much of this will depend on how the current debate over the GOP's foreign policy positions pans out (see Drezner's latest Foreign Affairs piece, for example). If the Democrats manage to become the party of national security moving forward, perhaps we might see a reversal of these trends. Given the current Republican field, however, I wouldn't hold my breath.**

 * Given that none of this work/data is published (yet?) please don't cite this post or data. I'm still very much hoping to update and improve this material as I go. I guess this is kind of the norm, right?

** In the off chance that anyone comments, please be aware that I have had trouble leaving comments in the recent past. On the rare occasion that we do get feedback, I feel rude not replying, so apologies in advance if this should happen.

Quick Plug on New Pubs (Shameless partial self-promotion edition)

A bit of shameless self-promotion before the holidays. The kind folks at International Studies Quarterly have put a new article by Colin Barry, Chad Clay, and myself up on early view. The link to the article, entitled "Avoiding the Spotlight: Human Rights Shaming and Foreign Direct Investment", is here, and here's the abstract:

Nonstate actors, such as international non-governmental organizations (INGOs) and multinational corporations (MNCs), have attained an increasingly prominent role in modern world affairs. While previous research has focused on these actors’ respective interactions with states, little attention has been paid to their interactions with each other. In this paper, we examine the extent to which the decisions of private actors seeking to invest abroad are affected by the reputational costs of doing business in countries publicly targeted by human rights activists. We find that ‘‘naming and shaming’’ by human rights INGOs tends to reduce the amount of foreign direct investment received by developing states, providing evidence that INGO activities affect the behavior of MNCs. An additional implication of our findings is that shaming by INGOs can impose real costs on targeted states in the form of lost investment.

We have a few projects along these lines that link our respective core research agendas in various ways, so (editorial and reviewer gods willing) be on the lookout for more in the future. 

Happy holidays to all!