Criminal by Tom Gash is a great book about why people commit crime. But it also has one of the most accessible and straightforward analyses of the weaknesses of econometric modelling that I’ve seen.
Picking on two famous analysis of the causes of crime – Steven Levitt’s (of Freakonomics fame) thesis that increases in abortion in the 1970s led to reduced crime in the 1990s, and John Lott’s claim (set out in More Guns, Less Crime) that part of the crime decline was due to new laws in certain US states allowing for people to carry firearms – Gash states:
Despite vast differences in their conclusions, these studies have at least two things in common: their ability to capture headlines and their method of analysis. This method – generally known as econometrics – sounds intimidating. It results in papers filled with equations, data tables and the results of various ‘tests’ run through powerful computer programs which can make even the mathematically minded feel insecure.
Of course, it’s tempting to defer to the wisdom of those clever enough to wield this startling array of tools…but the basics of econometrics are reasonably easy to grasp. Essentially, econometric studies look at a number of different trends or events and try to understand the relationships between them. To create models explaining shifts in crime, our dispassionate scientists must therefore first identify the main factors that might influence crime. Then they must find the data that measures these factors. Then they must discover how all these factors interrelate… Results will tell researchers whether there is a relationship between crime and the factors identified, how ‘confident’ researchers can be of the relationship, and the likely magnitude of the relationship’.
While Gash is talking about ‘crime’ in the above, you can subject pretty much any phenomenon that has been subjected to econometric research – from other social trends, to the economy, to political dynamics, etc. Gash further explains:
This is the process that allows authors like Levitt…to say ‘legalised abortion appears to account for as much as 50 percent of the recent drop in crime’. Such methods and the arguments that rest on them appear superficially plausible. But a deeper examination soon reveals that what looks like dispassionate science is in fact messy art…econometric models are typically underpinned by myriad assumptions.
The choices of which variables to include in a model – and, by implication, which to exclude – and how these variables are measured play a crucial role in determining the findings. On top of this, findings can be skewed by other data selection issues like the geography (some countries / cities and not others) or the timescales over which data is examined. Gash explains that:
…[the variables used]…vary vastly between different models. Levitt’s include abortion laws; most don’t. John Lott’s include gun laws; most don’t. Levitt, Lott and Marvell [Thomas Marvell, who has argued that crime in the US would be many times higher without the historic increase in the prison population] do not regularly look at factors such as drug consumption, social values, or marriage – which other studies do examine.
Almost all studies include variables such as police numbers – but virtually none bother to examine the number of people working in private security companies, even though the US, for example, has significantly more security guards than police officers…There are choices as to whether to use police force figures for recorded crime (which are more plentiful) or data from victim surveys (which are more meaningful)…As importantly, I have never seen an econometric model that includes ‘the number of well-positioned street lights’ or ‘good transport out of urban centres at night’ as variables – even though we know from small-scale studies that these can have an impact on crime rates.
The process of deciding which data should be used also requires a high degree of precision…Many people think that inequality matters in assessing crime rates but… which types of inequality… differences between incomes, differences in wealth…or how unequal people feel their society is [?]…
Choices must be made – and they are choices that are constrained by the limits of our ability to measure complex phenomena in our complex world.
Even after data has been collected, there are many judgements to be made…One major problem is that having too many variables in a model creates mathematical complexity and confusion – so modellers go through a process to narrow down to the variables that appear to ‘matter’. As factors that show an apparently ‘weak’ relationship… are excluded, the mathematical relationships between… those factors that remain grow stronger.
In other words, the myriad small choices that econometricians make – due to their own research interests, the availability (or lack thereof) of suitable data, and how they decide to ‘tidy up’ the model towards the end of the research project (i.e. removing ‘messy’ variables that create unnecessary complexity) – all have a big influence on the findings of the work. Tom Gash puts it like this:
In the process of creating a model, it’s often surprisingly easy to persuade yourself that this vast array of omissions and assumptions aren’t that important and that the conclusions you come to are still highly meaningful. As an old economics joke puts it: ‘econometricians, like artists, tend to fall in love with their models’.
These are not trivial issues. Some of the most significant and highly publicised works of the last few years have been subsequently taken to task over their use of figures and modelling – including Thomas Piketty’s Capital in the Twenty-First Century, and Kate Pickett and Richard Wilkinson’s The Spirit Level. Equally, the complexity of these models means it can take some time to recognise and correct errors (Gash notes that Levitt had to downgrade his estimate of the impact of abortion on crime when a coding error was discovered).
All of this leaves the casual reader in a bit of a pickle. Without enough time or expertise to properly vet research, what should we make of studies that use these methods?
Gash’s view is:
The results of such analysis remain opinions and not pure facts…we simply cannot rely on these methods alone to draw robust conclusions, and we must certainly not accept their conclusions at face value.
In fairness to Gash, he provides a nuanced take on the above, stating that in smaller scale studies these methods are more suitable and produce better results. He also notes that his own conjectures about the causes of crime ‘are no more provable than the theories proposed by our bold econometricians’.
At a time when these kinds of studies are increasingly supporting policy conclusions, recognising their (serious) flaws is important, as is having the humility to recognise the weakness in one’s own ideas.
