Blog: Making the most of evaluation to ensure health policy reform is evidence based.
Evidence based medicine is here to stay and has revolutionised the way clinicians around the world manage patients. At a recent surgical convention in the US, Atul Gawande made the observation about evidence based medicine on twitter that “the fury of clinical debates is inverse to the strength of evidence”.
The same may be said for health policy, which, in comparison to medicine, suffers from a chronic lack of high quality evidence and evaluation. The results can be clearly seen in the debates surrounding current proposed reforms to English health services in which anecdote and political point scoring rather than a grounded discussion of sound evidence frequently dominates.
Papers published on The Lancet UK Policy Matters website this week raise concerns about the quality of past evaluations commissioned directly or indirectly by the Department of Health. The ‘Making The Most of Evaluation’ research group was commissioned to review the evaluation processes used for a range of policy initiatives that arose from the previous Labour government’s health white paper, ‘Our health, our care, our say’.
The aim of the programme was to provide an overview and synthesis of the evidence about the extent to which the initiatives were achieving the goals of the white paper. Using guidelines provided by the Clinical Appraisal Skills Programme (CASP), Making The Most of Evaluation systematically reviewed each evaluation (see Panel for a full list) and produced structured summaries of the evidence available for each initiative. They also interviewed those who commissioned and conducted the evaluations in order to make recommendations about how the use of evaluation in the NHS could be improved1,2.
Certain evaluations, such as the one on Individual Budgets, were found to have been conducted in an extensive and rigorous manner and provided useful information, highlighting important areas for concern for policy makers. However, it is not clear whether the issues highlighted by this evaluation process have been fully considered by the coalition government’s White Paper “Equity and excellence: liberating the NHS” which plans to build upon personal health budgets in a pilot programme in 2012 to inform a wider roll-out.
The ‘Making The Most of Evaluation’ team also reviewed the Social Enterprise Pathfinder Programme, an initiative which has striking similarities with the coalition government’s Big Society agenda. They found the original evaluation to have some unclear objectives, raised concerns about the methods, questioned whether there was sufficient data presented to support the conclusions, and highlighted the absence of a definition of success. The research group also raised concerns about the objectivity of those conducting the evaluation, given that another arm of the same organisation had provided consultancy services to PCTs to develop social enterprises.
The reality of policy making does not need to be, and should not be, like this. Evidence based health policy is not an ideological panacea that is impractical in the real world. Evidence can be used to ensure that policy and programmes achieve the stated aims and objectives of an elected government, both through better evaluation to provide strong evidence from past policy initiatives, and by pro-actively testing proposals, using established and transparent methodologies, such as randomised controlled trials. Although it is tempting for politicians to want to promote the benefits of their policies, a serious attempt to ensure that an initiative is genuinely cost-effective before it is widely implemented can prevent costly mistakes and later political embarrassment. Policy aims, objectives and processes are clearly influenced by political ideology, but better evaluation is needed to enable the electorate to scrutinise performance and hold governments to account.
Such evaluation will allow a systematic approach to resource allocation decisions, although it is recognised that decisions about the prioritisation of evaluations also need to be addressed in regard to financial restraint3. In order to ensure better democratic accountability, there also needs to be better information provision to the public, professionals and policy makers about the evidence of how policies affect health4.
The Lancet UK Policy Matters aims to do just that. We are developing a library of policy summaries that look at the evidence about how existing and planned policy changes will affect health. At the same time we are gathering new evidence about the health impacts of policies as they happen. The series of submissions by the Making The Most of Evaluation group are excellent examples. Through publication of reports like this we aim to create better evidence for better health policy to reduce the “fury” of health policy reform debates and to invert the strength of evidence.
Panel: Initiatives promoted by ‘Our health, our care, our say’
Care Closer to Home demonstration sites
New types of social care workers
Self-referral to physiotherapy
Blog by: Lancet UK Policy Matters Team & Professor Chris Salisbury
1 Salisbury C, Stewart K, Purdy S, et al. Making the most of evaluation: a mixed methods study in the English NHS. J Health Serv Res Policy 2011; 16: 218–25.
2 Salisbury C, Stewart K, Purdy S, et al. Lessons from evaluation of the NHS white paper Our Health, Our Care, Our Say. Br J Gen Pract 2011; 61: 766–71.
3 Ogilvie D, Cummins S, Petticrew M, White M, Jones A, Wheeler K. Assessing the Evaluability of Complex Public Health Interventions: Five Questions for Researchers, Funders, and Policymakers. Milbank Quarterly 2011; 89: 206–25.
4 Gray JAM. Evidence based policy making. BMJ 2004; 329: 988–9.
Leave a Reply