A cornucopia of evidence

"Space is big" said Douglas Adams, ruminating on the scale of the universe.

This bigness is something we need to face whenever we consider the application of logic to evidence (or more classically Logos). The results we derive depend not only on the validity of our analysis, but on the choice of data. Not just the direct validity of the data chosen, but the completeness of our selected sub-set of data relative to the totality of the relevant data.

But the universe of data itself is big in the Douglas Adams sense. In most problems complicated enough to be interesting - especially those relevant at a human scale - there is almost no end to the facts and figures that might have some bearing on the issue.

In contrast to the scale of the relevant data is the scarcity of our attention span. The amount of information that we can consider at a time is small. This is true for blog posts, journal papers, newspaper articles, and conversations. Less so for a book-length piece of work, but even these can synthesize only a tiny fraction of the available evidence.

So the limited choice we can make of the large amount of evidence available is critical to the validity of our work.

Of course the above points not only applies to arguments that we make for other's consumption, but to a certain extent for our own. While the scope of data we can consider when forming an opinion is larger than we can practically present in a cogent argument to someone else it is still limited. Before we can attempt to draw conclusions we need to be sure we've considered the wider context of the data.

A consequence is that whenever we read an argument we are placing our trust in the person making it. Because while we might be able to use our intelligence to follow the argument unless we are an expert at the subject matter we almost certainly don't know the evidence. So we can't see the lies of omission. I suspect we know this at some level, explaining the success of ad-hominem attacks. If our basic trust in the source of an argument is undermined then we stop listening as the logic of the argument becomes irrelevant.

In medicine the effect of the cornucopia of evidence is well known. New drugs, treatments and etiological risk factors usually undergo a plethora of clinical trials with varying quality and statistical power. Even the gold-standard Randomised Controlled Trials often show a pretty mixed pattern of results. Some show no effect, some a negative and some a positive. That's the point of the meta-analysis, it encompasses the cornucopia. Or at least enough of it to reveal the shape of the rest.

Tabloid journalism on the other hand goes out of its way to abuse this concept. Fact 1... fact 2... - inductive leap - verfication! From an handful of emotive crimes to a broken society. The most serious flaw in this process isn't the emphasis on pathos (or ethos for the Mail) over logos, or the use of induction, but the cherry picking of data in the first place. So there's the flippant answer at least - consider what a red-top would do, and don't do that.

But really all this is begging the question... how do we choose our data? I plan to explore that next time.