DA-RT Vader and QCA
Reading the many interesting posts, I realize that I have little to add on the general points regarding the costs and benefits of implementing DA-RT. I therefore restrict myself to briefly sharing my thoughts on how DA-RT seems to play out for empirical research using Qualitative Comparative Analysis (QCA). For this purpose, it is useful to think of QCA-based research as composed of three different phases. (I) prior to the truth table analysis; (II) during the truth table analysis; and (III) after the truth table analysis.
Phase (I) usually resembles much of the qualitative research discussed in this forum. Scholars gather lots of (usually unstructured) data by conducting interviews, visiting archives, etc., and they pretest in an iterative manner their initial hunches about the (causal) relations between their tentative concepts. Data collection and preliminary data analysis often go hand in hand. Many, if not most of the concerns regarding the (unintended) consequences of DA-RT on this type of research equally apply to QCA-based research during phase (I). There is perhaps only one specificity: As a set-theoretic method, QCA processes data in the form of membership scores of cases in sets. The calibration of sets – that is, the translation of empirical information into set membership scores – should be transparent and replicable. For all the reasons mentioned in other contributions to this forum, this is hard to achieve if and when the raw data consists of ‘qualitative’, unstructured pieces of information. If, however, the data consists of rectangular data sets, then researchers can and should take several measures to increase transparency and replicability. They should indicate (and make available) the raw data used for the calibration of sets; report the qualitative anchors used for calibration; and indicate the functional form chosen for the translation of the raw data into set membership scores.
Things look quite different for phase (II) - the truth table analysis. In a narrow sense, here QCA very closely resembles quantitative analyses of rectangular data sets. In principle, and in practice, I think the truth table analysis can and should be fully transparent and replicable. This seems relatively easy to achieve, especially when the truth table analysis is performed with command line software. I share the concerns voiced in this forum and elsewhere that in such a narrow sense, replicability and transparency might not mean much, for it alone cannot prevent fraud, nor produce more meaningful and interesting findings. At least in QCA, whether the truth table analysis yields ‘good’ results depends on the data generated during phase (I). Yet, I see virtues and no serious vices in adhering to a replicable and transparent truth table analysis. At the risk of being incomplete, I list what I see as the minimum information that needs to be provided by any published QCA vis-à-vis the truth table analysis:
a) Consistency threshold
- What is the minimum consistency value to be achieved in order for a truth table row to be considered sufficient for the outcome?
b) Frequency threshold
- What is the minimum number of cases to be contained in a truth table row in order to consider it not as a logical remainder row?
c) Treatment of logical remainders
- Have assumptions on logical remainders been made in the course of logical minimization?
- What kind of assumptions (easy, difficult, etc.)?
- Preferably, researchers even provide a (summary) list of those remainder rows that have been included in the logical minimization
d) Any other consideration influencing the decision as to whether a truth table row is included in the logical minimization
- Sometimes specific rows are subsets of the outcome and its negation (simultaneous subset relations). They should not be included in both logical minimization procedures
e) Minimization algorithm
- Over the past years, different algorithms of analyzing a truth table have been developed. While most of them most of the time produce the same solution formulas, this is not true in all circumstances. Researchers should therefore be explicit about the algorithm used. Since differences in algorithms come with the use of different software packages, researchers should also report with software (and which version of that software) they have used
- Ideally, researchers also report how robust their findings are vis-à-vis equally plausible decisions on the above-mentioned parameters of the truth table analysis.
g) Reporting of results
- The presentation of results needs to achieve a triple goal. It must display, which conditions explain the outcomes; how well that explanation fits the data at hand; and how each case fits into this explanation. Transparency therefore requires that the solution formula is reported; that the parameters of fit are displayed; and that cases belonging to the different parts of the solution formula are made visible.
- If there is model ambiguity (more than one solution formula fitting the same truth table equally well), researchers must report about it
- Given its central role, each paper should display the truth table used for generating the solution formula
- Researchers should make available the calibrated data set
Phase (III) of applied QCA – the return to cases and their intensive study – again resembles more a typical qualitative research. The goal is to use process tracing in order to provide within-case evidence for the cross-case patterns discerned during the truth table analysis. To me, there is nothing special about post-QCA process tracing when it comes to the actual practice on the ground. Therefore all of the concerns voiced in this forum about DA-RT seem to equally apply for this phase of QCA. QCA, though, is currently making some progress towards more formalization in the selection of cases for process tracing. Under the label of set-theoretic multi-method research, principles and practices are being formulated as to which cases to choose for which analytic purpose. My sense is that being transparent about why specific cases have been chosen (and not other, perhaps more useful cases), is beneficial and does not impose any costs on the researcher.
All in all, it seems to me that QCA does not raise any concerns about DA-RT which have not already been voiced. What might be special is that because QCA is both an approach and a data analysis technique, the list of requirements to be fulfilled is quite long and varied. Fulfilling those DA-RT requirements seems pretty straightforward for QCA as a technique (truth table analysis) but very difficult, if at all possible, for QCA as an approach (phases I and III).
PS: I tried to link the text to its subject line but couldn’t