Black box modelling

Posted on by

Nick Rowe is concerned that agent-based modelling (ABM) is a black box that provides no intuition and doesn’t really add to our knowledge:

Agent-based models, or any computer simulations, strike me as being a bit like [a] black box. A paper written by a very reliable economist where all the middle pages are missing and we’ve only got the assumptions and conclusions. I can see why computer simulations could be useful. If that’s the only way to figure out if a bridge will fall down, then please go ahead and run them. But if we put agents in one end of the computer, and recessions get printed out the other end, and that’s all we know, does that mean we understand recessions?

My question is how a model where you set the rules can ever be a black box? Shouldn’t the results always be understandable by reference to the initial conditions and ‘rules of the game’?

I work a lot with computable general equilibrium models, which are often referred to as black boxes. It’s true that they produce voluminous results that aren’t always easy to understand. But the reason they’re difficult to understand is usually due to the volume of inputs and outputs, not the overwhelming complexity of the model. Consequently, you can usually provide intuition to the results if you go back and examine the initial conditions and ‘rules’ of the simulation. Maybe ABMs are different because of the complexity of the rules, but that’s not the sense I get from talking to modellers.

My hypothesis is that people who point at models and call them black boxes usually just can’t be bothered expending the effort to understand them. As Rowe says, any mechanism that you don’t understand can appear to be a black box. Is there really anything special and different about ABMs?

  • Nathaniel

    Interesting hypothesis – I would call a black box a mechanism that cannot be fully observed and therefore the outputs of which cannot be independently verified. I would therefore call a model a black box if, regardless of how much effort one expended, the path from the model to the results cannot be seen. Essentially the results have to be taken on faith. This is fine if the consumer of model results are willing to do so – in which case BOTE models are useful – but not sufficient for quality assurance.

    • http://www.tvhe.co.nz/ jamesz

      How do you build a mechanism that you can’t observe while still satisfying the Lucas critique?

      • http://tvhe.co.nz/ Matt Nolan

        Black boxes don’t satisfy the Lucas critique with such models. As a result, they are useful for raw “description” and for prediction given “unknown” CP clauses (as we should be able to boil down black box models to a structural form with enough effort) and as long as policy rules don’t change given their results.

        Black boxes neither explain, nor allow for policy relevant analysis – I see them as akin to reduced form time series models where the structural coefficients are unable to be solved. Useful for framing what has happened, and creating forecasts conditional on things we don’t know.

        • http://www.tvhe.co.nz/ jamesz

          I agree. What I don’t really get is how you can build a model in which you specify the structure and the input data but are unable to provide intuition about the results.

          • http://tvhe.co.nz/ Matt Nolan

            Black boxes are processes where we have a set of inputs, and we just put those inputs in and outputs magically come out. The way I see it there are two places where this turns up with economists

            http://en.wikipedia.org/wiki/Reduced_form

            Black boxes in this form also try to see themselves as “theory free”, we are discussing a historical relationship between variables without including structural terms that will show up in the error, or within the parameter estimates of our variables. The process of using time series analysis is well understood – but our conception of the model where choice is being made is not.

            The second place is to do with our conception of choice directly – we assume that inputs go into a person and outputs come out, without fully describing the mental process that we undergo. Rational choice, in this way, is more of a black box concept than a true descriptive concept – and it is our determination to move away from this black box conception that helps to make behavioural and experimental economics more popular.

            • http://www.tvhe.co.nz/ jamesz

              1) Sure, but not really applicable to ABM
              2) Yes, hopefully neuroeconomics can help here. Behavioural stuff just seems to add black-box heuristics.

              • http://tvhe.co.nz/ Matt Nolan

                1) I assumed that the comment thread here was about black box modeling more generally rather than ABM – as it started off with Nathaniel discussing what a black box was and you asking about the Lucas Critique, which is where my comments came from. I decided to trust what you said in the post about ABM having explained paths that make it not black box :)

                2) Indeed.

  • http://tvhe.co.nz/ Matt Nolan