Qualitative Transparency Deliberations

[Hillel Soifer Temple University]

What should scholars say about how cases were selected, and the point in the research process at which such selection took place? Should qualitative researchers always identify a logic (typical, most-likely etc.) for their selection of cases? What other information should scholars provide to make case selection choices transparent?

[Ryan Saylor University of Tulsa]

In general, I think making one’s case selection strategy explicit is useful for clarifying what one hopes to accomplish with a given comparison. At the same time, however, I’m not sure that it’s necessary for executing good research. And I worry about placing undue emphasis on specifying a case selection strategy at an early stage of research. I agree with Sean Yom’s 2015 CPS article, that we engage in inductive iteration and figure things out as we go along. This tendency probably applies to how many of us eventually arrive at our set of cases. I think a steadfast expectation that we have identified all the pertinent variables and how our cases score on them early in a research project likely overestimates both our knowledge of the cases and the specificity of our causal claims at that point. Also, we make lots of practical and pragmatic choices as researchers. These sorts of considerations, which won’t necessarily conform to a particular logic of case selection, may nonetheless be compelling features of why we analyze certain cases and may enrich our work and discipline if they are conveyed more commonly.

[Ingo Rohlfing Cologne Center for Comparative Politics, Universität zu Köln]

I read this as an argument against the use of predefined types of cases and using them in any design. Is this correct? This is an interesting thought. However, it seems to imply that there is no transparency issue per se. It would be more a matter of making transparent whether one proceeds exploratorily/iteratively, or confirmatorily? (though one can also specify types in iterative/exploratory research, I think)

[Heather Lanthorn IDinsight]

It seems useful to need to walk the reader through the case selection process, even if it happened iteratively and inductively. One can at least lay out an ex ante view of the types of cases one is seeking and why (given the research objectives). This need not become a straitjacket but the researcher can explain why deviations were taken and justify the final set of cases used.

[Hillel Soifer Temple University]

Thanks, Heather, for this post. It seems to me that the post raises a broader question: I wonder what we can say further about the grounds on which "the final set of cases" used can and should be justified: should the arguments provided to justify/defend a particular selection of cases necessarily mirror the iterative and inductive process by which they were selected, or is it appropriate to look back and justify the set of cases in other ways?

[Heather Lanthorn IDinsight]

Thanks. This is a useful question but perhaps a more concrete example will help us think it through? Do you have a (near) hypothetical in mind?

[Tasha Fairfield LSE]

A quick note to add that I agree with both Ryan and Heather. Most qualitative research is inductive, and very often cases are identified and scored during the process of research, not from the outset. I would add that inductive research of this type is fully justified within a Bayesian framework. Bayesian probability mandates a "dialog with the data"--we constantly go back and forth between theorizing, gathering data, and analyzing data, and by and large, the concerns that arise in an orthodox statistical framework just do not apply. (Timothy McKeown's 1999 article in response to KKV makes some of these points, I'm working on developing them further in my current methodological research.)

At the same time, it could be helpful to briefly walk readers through the process by which cases were identified and included in the analysis. The approach I tried in my book was to write a brief appendix describing my iterative process for selecting tax reform cases. I've posted the appendix on my webpage in case you'd like to take a skim and comment on whether you find this approach to "transparency" useful. http://tashafairfield.wixsite.com/home/publications

[Guest]

Ken Roberts, Cornelll:
Just a quick follow-up to say that I agree that it is a good idea to provide the reader with some sort of explanation of the process by which cases were selected and the methodological rationale for the case selection-- recognizing that in the real world of comparative case-based research, the latter may not be fully elaborated at the outset of the process, for the reasons that Tasha and Ryan and others lay out above. A lot of our case-based research does proceed inductively, such that theory is in constant dialogue with data analysis, in part because we are "drawn" to the study of important and interesting outcomes that we can already observe but seek to explain. Explanation then requires that we figure out "what this is a case of," and then identify the scope conditions and an appropriate set of comparative referents (ideally representing a full range of variation on the outcome of interest). In my own experience, I began a long-term study of party system stability in Latin America because I was doing some other research on parties in Peru when that country's party system, almost overnight, collapsed. Naturally, I could not help but ask "How did that happen?"-- a question that forced me to compare Peru to neighboring countries with widely varying patterns of party system stability and volatility. At the end of the day I dropped Peru from the systematic comparative analysis because even though it largely "fit" the larger patterns, the case had a number of anomalous features that made it less useful (and parsimonious) for explicating the theoretical argument. But that argument clearly emerged through intensive and inductive dialogue with the comparative data provided by the cases. Tasha's distinction between "typical," "extraordinary," and "anomalous" cases is quite helpful for thinking through these issues, in particular the role of conditional effects and the boundaries of scope conditions.

[Guest]

Andy Bennett, Georgetown. I am a member of the QTD steering committee but I am representing only my own views here.

When I review grant proposals, journal submissions, or my students' work with regard to case selection for process tracing, I often would like to see more clarity and transparency. This helps me assess whether the author has a strong rationale for their case selection, whether the case selection fits the researcher's goals for the kinds of knowledge they want to attain, whether they are claiming too much or too little on what they can infer from their cases, and what information was "use novel" to them at different points in their research process (so I can better assess the risks of confirmation bias). At the same time, I fully agree with the point that Ryan and others have made that case selection in qualitative research is usually an iterative process as our concepts and our knowledge of the cases evolve (and Ken gives a great example here where the case of party collapse in Peru led him to a new research question he later decided not to include the case of Peru itself in his study). I think being more transparent about how and when we iterate between our concepts and the cases we choose for further study would not take a lot of extra time, verbiage, or effort on the part of authors or readers. It will also clarify for students how scholars iterate between what they know or learn and how they choose their cases, and dispense with the illusion that case study selection involves easy choices made early on rather than tradeoffs among competing desiderata and often revisions as the research proceeds.

I think some reasonable transparency practices here would be: 1) identify the general attributes of the population from which you selected your cases, or if the population is small enough enumerate it, or if the population is identified in one or more extant data sets identify them; 2) identify the attributes of cases that almost fit into the population, or specific cases that almost fit if they are few and well-known, especially if they do not fit your theory (this helps convince reviewers that you have not arbitrarily excluded anomalous cases from your population); 3) identify the research design/rationale for the cases you chose (possibilities here include most-similar, least similar, deviant, most-likely, least-likely, high on independent variable, high on dependent variable, influential, "typical," etc. -- see Seawright and Gerring 2008 and Seawright's new book); 4) say something about what you knew about the cases when you chose them. Here, address the issue of the iteration between your concepts and your knowledge about the cases in the population to clarify what information was use novel for you when you started your process tracing. This involves saying something about your preliminary knowledge of the values in the cases chosen of the attributes or potential independent variables, and how these values in your cases compare to those in the general population in question (ie, you can't choose a most-similar, most-likely, high on independent variable, etc case unless you have some preliminary knowledge of the values on the independent and dependent variables of the cases in the population, even if this preliminary knowledge may change as you study the cases more closely) ; 5) briefly discuss the cases you almost chose for detailed study and why you did not choose them. Steps 1-3 are already widely practiced; steps 4-5 are less common but I think addressing them can help convince reviewers that your work is well-done and that you have taken precautions against confirmation bias. Often reviewers think there is a case that you could have selected that would suit the purposes of your study better than the cases you chose; briefly identifying the cases you almost chose and the reasons you did not choose them can help address this.

Of course, if the "cases" are individuals, in some research settings their identity and perhaps even some of their general attributes relevant to the case selection might need to be anonymized or withheld to protect them.

[Ingo Rohlfing Cologne Center for Comparative Politics, Universität zu Köln]

There is growing skepticism about what is reported in non-preregistered research. Besides that I do not see a reason to conceal the actual case selection process, there are good reasons for walking the reader through the actual case selection process because it might diffuse such concerns. For me, this is independent of the of Bayesianism/non-Bayesianism distinction and inductive/deductive distinction.

[Derek Beach University of Aarhus]

What should scholars say about how cases were selected, and the point in the research process at which such selection took place? Should qualitative researchers always identify a logic (typical, most-likely etc.) for their selection of cases? What other information should scholars provide to make case selection choices transparent?

The most important factor is a clear mapping of cases in the (bounded) population of potential cases, as this will determine our ability to generalise from what is found in the case to other causally similar cases.

Here QCA is a useful tool for finding patterns of cases that cluster into members of particular conjunctions of conditions - hopefully what then are causally homogeneous cases within the members of the particular conjunction. Once one has studied 2-3 typical cases in a particular conjunction, one can generalise with a reasonable degree of confidence that the relationship is present in all of the cases in the bounded population. For more, see recent work published in SMR on the topic.