Access to the broader evidentiary record in Congruence Analysis
Posted: Tue Apr 12, 2016 6:15 am
The goal of my contribution is to point to the need for reasonable compromises when it comes to providing access to the broader evidentary record in case study research.
In German and English textbooks (e.g. Blatter and Haverland 2014: Case Study Designs. Palgrave), together with colleagues I developed "congruence analysis" as a consistent case study methodology that transgresses the boundary between positivist and interpretative research. The main goal of these textbooks has been to make clear that there are different consistent ways to pursue "within-case analysis" and we should not put all these techniques of data collection/production and data analysis/interpretation under the umbrella of "process tracing". Furthermore, we argued that we should take terminology seriously and use the term "process tracing" only if we want to reveal the temporal unfolding of causal process.
Congruence analysis is based on the assumption that theories are holistic and encompassing worldviews and not just specific hypotheses. The goal of a congruence analysis is to compare a plurality of ex-ante formulated expectations that we deduce from a multiplity of theories with empirical evidence.
A good congruence analysis demands that in the theoretical part of a study we deduce a plurality of different expectations from each theory that we apply. In principle, each deduction has to be justified. If we apply three different theories and five expectations from each theory, we have to justify 15 deductions. In the empirical part, for each case we have to present all evidence that is in line or that contradicts the expectations derived from a theory. We have to do this systematically for all three theories for each case. And of course, we have to justify why we think that an empirical information is "in line" or "contradicts" the expectations.
Such an approach should ensure that we treat various theories fair and that we do not introduce some theories/explanatory approaches just to shoot them down with one or two observations and than to use the rest of the paper in order to show how consistent the case is with our prefered theory (as is quite common).
The point is - and now I get closer to the core issue of this contribution - that a good systematic congruence analysis needs a lot of space to lay out all these steps and to provide all the arguments in order to justify one's evluation/interpretation, especially if one analyses more than one case - space that most journals are not willing to provide.
In a recent article that has been the opener of a new online policy journal (European Policy Analyis), we solved the problem as follows: In the published article, we presented each of the four cases only in the light of the most congruent theory. But we added an online supplement in which we documented the full results of our analysis, which involved the comparison of each of the four cases with the expectations from each of the three theories. That document is 75 pages long, in German language and includes also a long battery of primary sources and a list of interview partners. In other words, such a supplement helps a lot in respect to enhance production transparency and analytic transparency, but it is far away from what some adherents of transparancy demand. I would like to stress that we do not provid access to the transcripts of the interviews. Actually, we often did not produce fullfledged transcripts since we are interested only in some parts of the information that we gather through interviews.
I think that we have to resist demands to provide access to every source and each piece of information that we gather in a case study research project. It would simply be such a burden that case studies could not be produced anymore.
To sum up: I think that we have to resist too far reaching demands for providing access to primary data. Nevertheless, I think that qualitative scholars and case study researchers should provide more insights in the intermediate steps in their endeavours to connect empirical observations and theoretical concepts/expectations.
Joachim Blatter
Universität Luzern
In German and English textbooks (e.g. Blatter and Haverland 2014: Case Study Designs. Palgrave), together with colleagues I developed "congruence analysis" as a consistent case study methodology that transgresses the boundary between positivist and interpretative research. The main goal of these textbooks has been to make clear that there are different consistent ways to pursue "within-case analysis" and we should not put all these techniques of data collection/production and data analysis/interpretation under the umbrella of "process tracing". Furthermore, we argued that we should take terminology seriously and use the term "process tracing" only if we want to reveal the temporal unfolding of causal process.
Congruence analysis is based on the assumption that theories are holistic and encompassing worldviews and not just specific hypotheses. The goal of a congruence analysis is to compare a plurality of ex-ante formulated expectations that we deduce from a multiplity of theories with empirical evidence.
A good congruence analysis demands that in the theoretical part of a study we deduce a plurality of different expectations from each theory that we apply. In principle, each deduction has to be justified. If we apply three different theories and five expectations from each theory, we have to justify 15 deductions. In the empirical part, for each case we have to present all evidence that is in line or that contradicts the expectations derived from a theory. We have to do this systematically for all three theories for each case. And of course, we have to justify why we think that an empirical information is "in line" or "contradicts" the expectations.
Such an approach should ensure that we treat various theories fair and that we do not introduce some theories/explanatory approaches just to shoot them down with one or two observations and than to use the rest of the paper in order to show how consistent the case is with our prefered theory (as is quite common).
The point is - and now I get closer to the core issue of this contribution - that a good systematic congruence analysis needs a lot of space to lay out all these steps and to provide all the arguments in order to justify one's evluation/interpretation, especially if one analyses more than one case - space that most journals are not willing to provide.
In a recent article that has been the opener of a new online policy journal (European Policy Analyis), we solved the problem as follows: In the published article, we presented each of the four cases only in the light of the most congruent theory. But we added an online supplement in which we documented the full results of our analysis, which involved the comparison of each of the four cases with the expectations from each of the three theories. That document is 75 pages long, in German language and includes also a long battery of primary sources and a list of interview partners. In other words, such a supplement helps a lot in respect to enhance production transparency and analytic transparency, but it is far away from what some adherents of transparancy demand. I would like to stress that we do not provid access to the transcripts of the interviews. Actually, we often did not produce fullfledged transcripts since we are interested only in some parts of the information that we gather through interviews.
I think that we have to resist demands to provide access to every source and each piece of information that we gather in a case study research project. It would simply be such a burden that case studies could not be produced anymore.
To sum up: I think that we have to resist too far reaching demands for providing access to primary data. Nevertheless, I think that qualitative scholars and case study researchers should provide more insights in the intermediate steps in their endeavours to connect empirical observations and theoretical concepts/expectations.
Joachim Blatter
Universität Luzern