Qualitative Transparency Deliberations

[Guest]

The concerns bubbling up in these discussions suggest parallels with the quality initiatives and process improvement initiatives launched in the 1980s. These initiatives aimed to improve the outcomes of software development efforts (mostly at the Department of Defense) that perennially fell behind schedule, over budget, and far short of meeting requirements. Their stated goal was to improving quality -- a goal that everyone agreed was important, as well as a magic word that could be interpreted in different ways by different actors. Various process improvement guidelines were developed, for example the Software Engineering Institute's Capability Maturity Model. These were subsequently adopted by government procurement offices as standards for judging and selecting contractors. At the same time, a brand new set of actors were empowered to advise, assess, and certify existing software creators... for a fee, in this case. But more often than not, contractors complied with these standards at a superficial level. Who could blame them? Their incentive structure was grounded in the basic profit motive: increase revenue (i.e. comply with the standards in order to win contracts) and reduce costs (compliance is costly, so do the absolute minimum). It's difficult to know how many of these approached compliance as a short-term profit game, following the letter of the guidelines even when this undermined its intent and spirit. The pursuit of efficiency and standardization, succeeded to some extent, in making software development more predictable, and this was a positive outcome for mid-level managers tasked with estimating and delivering projects. However, it also squeezed out creativity and innovation in companies who adopted it unreflectively. The trouble was, this unreflective group included nearly everyone other than the handful of organizations who initially designed and piloted these efforts. By design, the innovators developed guidelines for wider community use that were intended to be simple for others to follow. What happened here? In a sound bite: doing the thing right overshadowed doing the right thing.

I don’t mean to imply that this is happening here, but rather that it could, if we're not vigilant to guard against it. I see both worrisome and hopeful signs. With the envisioned output of this effort is a set of Community Transparency Statements to guide publishers, editors, researchers, and students, then it seems the focus of this effort is on doing the thing right -- in this case, the thing is transparency, and the struggle is about how to do it. That’s good, but more is needed. It seems many of those who have raised concerns are asking something different: whether to do it and why. In other words, they're asking whether we're doing the right thing. In this case, the right thing is high quality research. Both questions are important. The second one needs to be asked at least as often as the first.