jamesz

The problems of measurement in education

/in , /by

The Dim Post has a great guest post up that demonstrates what researcher drily refer to as ‘ability bias’. I’m not going to reproduce it all here but it’s worth reading if you’re not familiar with the problem.

The bit that I find really depressing about the article is this:

So, after all this has been done, Ministry Faktdrones come in and look at one piece of paper that has all of our grades on it. They don’t look at the students as humans, don’t look at any of our processes, and often don’t even look at previous results which would let them know things like value-added results. They look at one sheet of results and say “Look here – the class with 30 all got Achieved, and the class with 15 all got Achieved too. That means, statistically, class size doesn’t make a difference. Let’s cram forty of the little firestarters in there next year!”

Something I constantly and tiresomely have to harp on about when I’m talking to (at?) people about this and other education issues, is that the students are not statistics, and sometimes can’t be pigeonholed to fit a statistically clean model.

I’ve worked with people in the Ministry of Education’s research division a fair bit and they’re all hugely knowledgeable and competent people. They’re well aware of these problems with the statistics and highlight them in their research papers. They even work with the people who gather the data to try to improve the way it’s collected and overcome some of the measurement problems. So why do teachers still perceive Ministry researchers as ‘Faktdrones’? Either the research is being poorly interpreted by the policy teams or the outcomes of the research are being poorly communicated to the teachers. Possibly both.

Whatever the problem is, the author’s response seems to be to reject the use of statistics to inform policy because “students are not statistics”. Unfortunately for the author there is unlikely to be a reduction in the reliance on statistics to determine policy. From what I can tell the trend is actually for more statistics. Given that, the solution to problems such as ability bias is to collect better data. Sadly, the experience of teachers such as the author of this piece is likely to deter them from helping researchers interested in collecting more detailed data about their students.

In policy circles people often say that some data, however rough, is better than no data at all. Unfortunately, a little bit of data coupled with a little bit of knowledge can be a very dangerous thing. If it causes practitioners in the profession to be alienated by policy makers then it may actually be detrimental. It’s sometimes hard to remember that the ultimate goal of research is to help the people being studied, not simply to study them and experiment for its own sake. At times that may require a bit of caution and restraint.

jamesz

Why I’m not worried about struggling students

/8 Comments/in , /by

The recent budget reduced the number of students who are eligible for the student allowance, particularly for postgraduate students. There have since been almost daily articles in the newspapers lamenting the students’ plight. Today there is an article in which a student describes how she will have to take on extra debt to finance her postgraduate education. Similarly, my Facebook is full of people wondering why the government refuses to ‘invest in our future’. The reason is fairly straightforward: when the government pays for tertiary education it transfers money from poor people to wealthy people. Tertiary students, particularly those in postgraduate education, are some of the wealthiest people in our society and don’t deserve to get a free lunch on top of it.

You might wonder how I can justify that statement when most students have very low incomes. Well, they are not rich because of their present income, but because of the income they will earn in future. Three years after completing their degree, a bachelor’s graduate will earn 51% more than someone with only secondary qualifications. Someone with a master’s degree will earn 74% more and a doctoral graduate 120% more. Yet it was the secondary graduate, working full time while the student was studying, that helped fund about 75% of their education. How can that possibly be fair?
Read more

jamesz

Politics and cost-benefit analysis

/in , , /by

Brian Rudman in the Herald, discusses the cost-benefit analyses of major infrastructure investment that are required by Treasury guidelines:

I’ve come to the conclusion that the main beneficiaries of the big events politicians subsidise at our expense are the cost-benefit analysts hired to justify the expenditure in the first place.
…
The cynical take on the process is that the whole cost-benefit palaver was introduced by politicians to put a veneer of neutrality on the decision-making on pet projects.
…
That doesn’t worry me that much [but if] politicians … are going to support certain events and projects, despite the evidence of the cost-benefit analyses, why is so much public money wasted commissioning these expensive reports?

I’m a bit torn on the subject of CBAs of major projects. On the one hand, I think they should be extremely useful for informing political decisions. On the other hand, there’s plenty of evidence that politicians don’t pay any attention to them.
Read more

jamesz

Why can’t things just ‘be’?

/in /by

While browsing Andrew Gelman’s recent posts I also came across this gem, which reinforces my priors in every possible way. So of course I loved it! He writes:

More and more I feel like economic reporting is based on crude principles of adding up “good news” and “bad news.” Sometimes this makes sense: by almost any measure, an unemployment rate of 10% is bad news compared to an unemployment rate of 5%. Other times, though, the good/bad news framework seems so tangled.
…
Who’s supposed to be “concerned” [about the drop in the price of Facebook shares]? … I just don’t get it. Why should I care? If the shares underperform the market, people can buy a piece of Facebook for less. That’s fine too, no?

I think Gelman has put his finger right on the problem: news needs to be normatively charged to be worthwhile. A drop in share prices is good for some people and bad for others, but it probably doesn’t involve a loss of social welfare. Yet, to be a good news story, it needs to have a normative dimension with a hero and a baddie. I’m not blaming the reporters for that; it’s just human nature to want it. Unfortunately, it makes the job of reporting on economics particularly difficult.

In fact it explains why economists are constantly accused of being unable to reach a conclusion. It’s not because they don’t understand what’s going on: it’s because they attempt to describe the costs and the benefits of policies, which usually accrue to different groups. Few policies are unambiguously good or bad, so almost any economist’s commentary provides ammunition for both sides of the normative debate.

jamesz

Unbelievably exciting results!

/in , , /by

Chris Dillow, via Eric:

…even intelligent and numerate people are quick to misperceive randomness and to pay for an expertise that doesn’t exist; the subjects included students of sciences, engineering and accounting.

Which reminded me of Andrew Gelman’s recent post about results in headline academic journals such as Science and Nature. He quotes Sanjay Srivastava saying:

As long as a journal pursues a strategy of publishing “wow” studies, it will inevitably contain more unreplicable findings and unsupportable conclusions than equally rigorous but more ‘boring’ journals. Groundbreaking will always be higher-risk. And definitive will be the territory of journals that publish meta-analyses and reviews.

We may all know that the implausible results that often lead headline journals are likely to be wrong, but it doesn’t seem to stop us citing them constantly!

Matt Nolan

Points for the Budget

/in , , , /by

Over at Dim Post, there is the general question of whether a “zero-budget” makes sense.

Commenting on the post, I remembered some of the key points to keep in mind when looking at actual policy – rather than the tiresome and incoherent ramblings we will get from politicians of all stripes and colours when the Budget is released.

What you are saying is valid – but when we view the budget in economics terms, instead of the political terms it is being sold in, there are a few things to keep in mind:

  1. The goal should be to have a balanced budget, once the economy is back at “potential”. So the surplus in 14/15 has to come with a forecast that has unemployment at or below 5%. If unemployment ends up higher than this, the government will continue to run a deficit by default – and so even when the forecasts are wrong this aim squares out.
  2. Government has been borrowing heavily, at low interest rates, to fund infrastructure. If it wasn’t for government building there would have been virtually no non-residential building in recent years. Infrastructure spending (in real terms) is at record highs. The government is actually being very Keynesian – they just don’t want to say it because their supporters might get grumpy.
  3. If the government keeps to the idea of getting the budget back in balance by the time the economy is back at potential, and makes that clear, it makes it easier for the RBNZ to cut the OCR or at least keep it at its current level, which will also help to boost activity/cut unemployment.

You are right it is an incredible period of uncertainty – but the government can’t do too much about that directly except limit uncertainty from them by ensuring their plans are clear. We can debate what the appropriate long term “size” of government should be – but I think both parties are being responsible by making sure that we have the budget back in balance by the time the economy is back at potential.

The key way to judge the Budget in “macro” terms will be the consistency of the UR forecast to the surplus.  In “micro” terms there will also be a lot of marginal issues we can address – hopefully there will be more talk about superannuation.

Either way, I’m not forward to listening to all the blah blah blah about politics, but I will be sifting through the Budget to get an understanding of ways government policy will influence the outlook for the general economy during the coming years.