Where does the burden of proof lie?
Antonio Fatas discusses inherent bias in economics given our reference point – an important issue, and one that economists need to think on (Note: James wrote on this as well – we did our posts separately, so are focusing on different points). Specifically:
This subtle (or not so subtle) bias in economic analysis is my biggest source of frustration with my profession. Not being able to predict crisis, the stock market or exchange rates does not bother me, it is just a reflection of the limits of our knowledge and I can live with it. But using the same naive predictions of models that refer to a fictitious world as the reference and only moving away from them when someone produces an unquestionable piece of empirical evidence is in my mind the true cost of our profession to society.
His post raises good points, but I suspect that what he is laying out runs into the same problem many of us run into when using models to think about policy and the impact of policy – we are not being clear on where the “burden of proof” lies or what we think about assumptions.
Now I like his writing, and this is a good post, but I have a bit of a different view on what economists do with reference to this. Perhaps New Zealand economists are a bit different? Essentially, the question of burden of proof is usually treated as a central part of how we frame and discuss policy questions in New Zealand, so it becomes part of the way we discuss models.
Let me give an example. In New Zealand our central bank has pushed towards macroprudential policy, and more broadly we have pushed towards an idea of having financial institutions accept “compulsory insurance”. This comes from a view that the “burden of proof” has shifted from showing the actions of financial institutions will do harm to regulate, to a situation where we have to justify why certain actions in the current institutional structure will not harm.
So why the change? Well the “benchmark model” of the appropriate institutional factors in this area changed. It is as simple as that.
How does a benchmark model/models change? This makes more sense when we move away from talking about “increasing realism” and just think about the appropriate scope of assumptions in the economic method.
On assumptions
To motivate his post, Fatas links to a post by Greg Mankiw where he discusses an “implausibly strong assumption”. However, Mankiw was doing something very specific with his view on the model. He was trying to identify core assumptions, and then asking if they were appropriate. Economists, and scientists generally, make plenty of unrealistic assumptions – but the core assumptions of our model (the fundamental causal factors behind the result) need to be realistic. Not all assumptions are created equal, and discussing and comparing core assumptions is part of what we do.
Models are conditional, and one of the elements they are conditional on are that our “set of core assumptions” is both realistic and appropriate (namely that assumptions we identify as peripheral are in fact peripheral). So we can critique each others models on that basis, and use a variety of models that enlighten the way some assumptions translate into conclusions. This ain’t pretty, but it is the best we can do. We can view this as a way of “updating our beliefs“, something we can only do in a rational manner if our argument (the core assumptions and their link to our conclusion) is transparent.
In that environment what Mankiw did (calling out that a core assumption was entirely unreasonable. Note: Not saying I agree it was unreasonable 🙂 ) was appropriate.
Now do not get me wrong, if we had the stock of data and knowledge to make models massively more realistic this is grand – but we don’t. As data availability improves, and our ability to isolate causes rises, this will change … but not today! With this more limited set of knowledge, clarity about core assumptions, and a focus on them specifically when making policy related claims, is essential. Given that conclusion I quoted earlier seems like a non-sequitur to me:
But using the same naive predictions of models that refer to a fictitious world as the reference and only moving away from them when someone produces an unquestionable piece of empirical evidence is in my mind the true cost of our profession to society.
The worlds we build with theory are fictitious worlds, and always will be. What is important to understand is what assumptions matter and their relation to reality. In this context what Mankiw did was completely appropriate – if we are going to look at policy, we don’t need to defend the realism of all assumptions, but we do need to be able to justify our core assumptions.
That discipline is not a “cost to society” – it is a benefit. Economists just need to be sure they remain mindful of these lessons, as without care about expressing and testing core assumptions the discipline becomes little more than folk economics 🙂
It was completely coincidental that we wrote about the same thing! I generally agree with your view. My problem with it is that you are sidestepping the question of anchoring effects and the burden of proof. The reference model was never empirically tested, nor intended to be descriptive, yet that is how it is used. It has become an anchor point that requires an extremely high burden of proof before we shift away from it even though many of the crucial assumptions are not satisfied. For example, we know that measured preferences are not transitive and people tend not to maximise. We know people are loss averse and care about changes in utility, not levels. However, none of these innovations have significantly changed the way most economists think about welfare economics and policy questions despite their obvious importance. I think that is starting to change, but very slowly.
Indeed very good question – and the adds to what Fatas was getting at IMO as well.
However, I am not convinced I really sidestepped the issue – more that I accepted that core areas of economics need to be tested, either explicitly or implicitly, when we decide whether they are useful for policy. The whole idea that we should base policy on “many models” and with a keen acceptance of their limitations was both the core of what we were taught at uni, and how the public service is told to look at models over here – so if we are really going to go down the policy line this is consistent with action and I don’t see the problem.
Fatas seemed frustrated at Mankiw calling out a assumption in a model, when all models involve unrealistic assumptions. But this was an assumption on top of assumptions, it was a core assumption that drove the result of the model, by rejecting that assumption he was rejecting the result in that frame in of itself. We can hardly expect him to say “given other assumptions this may not be the causal assumption of the result, and given that I’d accept it – this is both self-evident and confusing.
While we desire to understand the “objective reality” that exists, we have a shortage of relevant observational data … both to test and to differentiate between explanations. In that environment, we rely on a stock of “accepted premises” between economists – and how these are accepted in the discipline depends upon rhetoric and may well be path dependent.
But this is unavoidable, and simply tells us to be more cautious with our results. Given this I don’t actually know what Fatas is complaining about – the discipline is doing this, and if (when it comes to setting policy) there are core assumptions that are believed to be unrealistic (and we can tell as the economic method makes them transparent) then policy makers can adjust what they are doing with regards to that.
I’ll be honest, I don’t see a “counterfactual” in Fatas’s post – no matter what we do we are using “fictitious worlds”. If we use a whole series of “other assumptions” it is still a fictitious world, and its acceptability will still largely be driven by rhetorical convention within the discipline. As a result, I don’t feel the relevance of his critique.
Now the issues you have raised – how we think about normative economics is an area that really does require significantly more practical work, including the question of what we are “targeting” with policy. I think that there is a lot of muddling trying to translate economic ideas into policies – and for a larger part than we’ll admit economists are to blame (eg if you don’t want responsibility for their policy relevance, why end each paper discussing how it is relevant for policy?). I also got a sniff of this in the Fatas post, but this isn’t a critique of what Mankiw did, or how we treat assumptions, it is much deeper – it is about having the discipline consider normative economics instead of just throwing it in the too hard basket (or to steer away from noting policy relevance in that way all together).
“we should base policy on “many models” and with a keen acceptance of their limitations…[that is] how the public service is told to look at models”
“if there are core assumptions that are believed to be unrealistic then policy makers can adjust what they are doing with regards to that.”
These are both convenient fictions. CBA is very poorly implemented in the public service worldwide and there is no evidence that practice is improving. Civil servants rely on the expert judgment of economists and others. They do not usually have the expertise to ‘adjust’ for assumptions that they are not aware of and that they do not understand the consequences of. If our advice requires that then it is not useful. Many people are working to improve the core model but we cannot become complacent about applying it in the meantime. As you often say, more humility is required and more openness to discarding ideas that are falsified, even if no obvious replacement is to hand.
I’m not sure I would call it a fiction – instead the frame I’m using to describe the process separates the “descriptive elements of economic science” from the actual action of “normative economics”. Too often, economists will confuse the two – which leads them to give too little weight to issues of moral philosophy even given acceptance of the limitations of the description in the first place.
It is important to separate these out, as if we chuck them together and start criticising “economics and assumptions” with regards to policy the argument is hard for me to understand – is it the arguments that we are providing that is poor (relative to what is attainable) or is it a fundamental disagreement about appropriate conclusions/targets of policy?
At worst, I have to separate the two as I get too confused if I try to look at both at the same time 🙂
“As you often say, more humility is required and more openness to
discarding ideas that are falsified, even if no obvious replacement is
to hand.”
Here is the thing, even when something is falsified (which in itself is a high burden of proof) what does it mean to discard it when we are talking about active policy? There is a conservative towards the status quo bias in economics because of this – as “falsification” in normative economics implies not accepting a justification for policy change!
Seeing economists cautious of changing policy is indicative of the fact we are nervous about the realism of assumptions – not an indication that some specific “benchmark model” is biasing the entire discipline towards certain actions.
In fact, it is the economists who are determined to rush through policies and change because “there model says so” that are more likely to be the ones suffering from a bias – so economics bloggers in other words 🙂
Of course we should be hesitant to reject a descriptive model that has performed well on the basis of limited evidence. I am not so optimistic about current models in economics and it is far too easy to fall prey to a status quo bias and keep rehearsing arguments.
For example, the weight of evidence against standard choice models has been strong since Kahneman and Tversky. A common objection is that there is no complete replacement for choice theory that would support the current breadth and depth of analysis in economics. That is true but presents an exceptionally high bar for discarding a theory that has little use in descriptive analysis.
Indeed – I certainly don’t disagree with you on any of that.
But I also can’t get past the point that all economists are doing, when they build a model, is putting together a “set of assumptions”. Is Fatas saying that he thinks the shared set of assumptions economists accept are fundamentally flawed and need to be changed? Or is he saying (this is the one I read over most of the post) that we need to be more cautious about policy due to the difficulty building a compelling argument.
If it is this second one I agree, defo! But, I feel that his conclusion is a bit strong – he goes from warning about certainty to criticising the use of fictitious worlds, that is a non-sequitur. That was my real point here.
Note: By including these other assumptions you mention, you are discussing trying a different fictitious world. I agree. Hence why I was uncomfortable with the why Fatas was treating them, and the associated assumptions, when discussing policy 🙂