Tim van Gelder, 31 December 2007
Summary and Critique by Jillian J
Van Gelder begins by describing the ACH method—whereby an analyst can “determine which of a range of hypotheses is most likely to be true, given the available evidence” and identifies the hypothesis-testing structure and external representation aid as some of the method’s strengths. He then lodges five complaints about the method: too many judgements, no e is an island, flat structure of hypotheses, subordinate deliberation, and decontextualisation and discombobulations.
Van Gelder writes that analysts must make a judgement of consistency for each piece of evidence they enter into the matrix, further stating that the number of judgements can quickly become cumbersome. On top of that workload, the relationship between a given piece of evidence (e) and the given hypothesis (h) may be irrelevant or inconclusive.
Next he cites the matrix structure as a problem that makes ACH treat “an item of evidence as consistent of inconsistent on its own with each of the hypothesis”. He writes that while the analyst may deem e consistent with h, this judgment is only valid in the context of other relevant information (or auxiliary hypotheses). If that other information said the opposite, then that same e would be inconsistent with h—in essence, a multi-premise structure. He acknowledges the short term utility of organizing h’s and e’s, but writes that further along the process, if the analyst finds information that challenges an initial assumption such that e1 is inconsistent with h only until combined with additional information (a), she/he has a problem. ACH doesn’t allow for that type of nuance.
The flat structure of hypotheses presents an issue because hypotheses can be, and often are complex. Van Gelder posits that ACH doesn’t efficiently address the multiple facets of complex hypotheses.
His fourth qualm is that ACH doesn’t have a way of weighting the salience of a given e. ACH allows the analyst to judge the magnitude of consistency (very consistent, consistent, neutral, not applicable, inconsistent, or very inconsistent), but doesn’t let the analyst delineate how seriously she/he takes the e itself.
The final issue is that while ACH tries to strip away excess details, the result is an e without context. This leaves the analyst uncertain of the relationship between e and h which leads to a muddied analysis.
I would add that ACH can also perpetuate cognitive biases. When I search for evidence, sometimes the result is a(n) “(in)consistency heavy” matrix. Then I might actively seek out disconfirming evidence to make the inconsistency to consistency ratio a little closer. While it’s useful to search for disconfirming evidence, I have to stop somewhere. I run the risk of choosing a stopping point that fits my bias.
I also think there’s a quick fix for van Gelder’s fourth issue about weighting salience. ACH allows the analyst to control the order in which the evidence appears on the matrix. It would be easy for the analyst to rank the evidence according to importance.
I found the flat structure critique only moderately valid. ACH allows the analyst to create virtually unlimited matrices. It may be tedious, but it is possible to break down a complex hypothesis into multiple matrices and apply evidence to each facet.
Overall, I agree with van Gelder. ACH has problems and its utility is limited. The issues he identified resonated with the frustrations I've had while using the method. But I maintain that challenging our analyses is as important as coming up with our analyses in the first place. Therefore, even with its flaws, analysts should apply the principles of ACH to their analyses, if not the ACH method itself.