Framework for evaluation of policies, programs and services

Development of framework for evaluation of policies, programs and services that aim to improve Aboriginal and Torres Strait Islander health and wellbeing

Project aim and objectives  

This project aimed to develop a framework to guide future evaluation activities aimed at Aboriginal and Torres Strait Islander policy, programs and services oriented towards health equity. Its objectives were to:   

  • Determine the key principles that underpin evaluation of any policy, program or service that aims to improve Aboriginal and Torres Strait Islander health and wellbeing.  
  • Understand the key elements in a systems-based framework to guide evaluations of any policy, programs or services that aim to improve Aboriginal and Torres Strait Islander health and wellbeing.  
  • Identify the key elements that support and advance Aboriginal and Torres Strait Islander community level engagement in relation to policy, programs and services evaluation to inform local decision-making processes.  
  • Determine the indicators or evaluation questions, for which data could be collected relevant to each identified element at the different levels.  
  • Identify the key elements of good practice evaluation at a policy or program level.  
  • Identify the elements of previous program evaluations that have demonstrated good practice.  

Project team 

Project leader: Professor Margaret Kelaher, Centre for Health Policy, Melbourne School of Population and Global Health, University of Melbourne  
Project partner: Deakin University  
Administering organisation: The University of Melbourne   
Project timeline: 01/07/2018—31/12/2018  

Methodology  

  • Established project reference group  
  • Project reference group identified organisations to be consulted and/or key informants to be interviewed  
  • Literature and document review completed  
  • Facilitated workshop for practice in evaluation of Aboriginal and Torres Strait Islander health initiatives  
  • Administered survey  
  • Identified case studies  
  • Conducted final workshop  
  • Finalisation of framework and recommendations  

Project findings  

The key principles underpinning many government programs to improve Aboriginal and Torres Strait Islander health and wellbeing included: shared responsibility, cultural competence, engagement, equity, partnership, capacity building, accountability, evidence-based and a holistic concept of health.   
The key elements in a systems-based framework to guide evaluations of policy, programs or services were identified as:  

  • translation from evaluation into action, including providing feedback to communities about evaluation findings  
  • Aboriginal and Torres Strait Islander leadership and partnership in the evaluation process  
  • building the evidence base, including better access to evaluation information and findings  
  • evaluation and ethics, specifically that current ethical guidelines are tailored to investigator-driven research rather than involving various parties.

Key elements that support and advance Aboriginal and Torres Strait Islander community level engagement included:   

  • making evaluation relevant to communities  
  • sharing the benefits  
  • orientating ethics to the benefit the community  
  • enacting and evaluating principles for working with Aboriginal and Torres Strait Islander people  
  • promoting data sovereignty, including both leadership and ownership of evaluation reports and data  
  • Aboriginal and Torres Strait Islander leadership in evaluation  
  • developing capacity for commissioning Aboriginal and Torres Strait Islander evaluation.  

Key indicators or evaluation questions, for which data could be collected, included:   

  • partnerships with Aboriginal and Torres Strait Islander organisations and communities  
  • shared responsibility  
  • engagement with Aboriginal and Torres Strait Islander people and communities  
  • capacity building of Aboriginal and Torres Strait Islander communities  
  • equity  
  • accountability  
  • evidence based  
  • holistic concept of health  
  • cultural competence  
  • data governance and intellectual property  
  • capitalising on Indigenous strengths.  

The study identified many aspects of good practice evaluation involving evaluators, commissioners and program implementers. Some of these included:  

Development stage:

  • engaging communities and stakeholders and establishing processes to ensure accountability  
  • establishing program governance, engaging with partnerships and enabling Aboriginal and Torres Strait Islander leadership.  

Implementation stage  

  • developing long-term partnerships between program implementers/ACCOs and community to build capacity around program implementation  
  • using training and employment to build community capacity to support program implementation.  

Evaluation stage  

  • establishing evaluation governance, engaging with program governance and enabling Aboriginal and Torres Strait Islander leadership  
  • building Aboriginal and Torres Strait Islander evaluation capacity through training and employment over the long term.  

Knowledge translation  

  • ensuring evaluation findings/data are available to participating communities and building capacity around the use of evaluation findings/data 
  • developing policy/program response to evaluation reports and ensuring responses are publicly available in perpetuity.   

Project outcomes  

Knowledge  

  • The project identified areas where evaluations could be improved, such as:  

  1. increasing transparency and accountability  

  2. incorporating principles for working with Aboriginal and Torres Strait Islander people into programs  

  3. using ethical frameworks that recognise the responsibilities of all parties in evaluation and supporting Aboriginal and Torres Strait Islander leadership and ownership at all phases of the program planning and evaluation cycle.   

  • The project found, overall, that there was recognition of limitations of current practice from a range of perspectives.   
  • There was also a recognition that systemic change is required to fully implement the changes required to improve the benefit of evaluation for Aboriginal and Torres Strait Islander communities.  

Awareness  
  • The results of the study were presented at the Lowitja Institute forum (“Community priorities into policy”) in Canberra, the National Aboriginal Community Controlled Health Organisation (NACCHO) conference in Brisbane and the Global Evidence and Implementation Science conference in Melbourne.  
  • A users guide for the framework and course materials have been developed.  

Behaviour  
  • The tools developed will assist commissioners of evaluation to improve their practice.  

Skills  
  • The project governance and research team includes leading Aboriginal researchers, policy makers and community members. Aboriginal and Torres Strait Islander people have been involved in all phases of the project  
  • Opportunities for capacity building and shared learning are incorporated throughout the project design. At this stage capacity building activities have been initiated through the project reference group and development of the research team. This was expanded in ongoing workshop phases and the development of resources to support ongoing evaluation practice.  
Related resources:

Acknowledgement of Country

We acknowledge the traditional owners of the land across Australia and pay our respects to their Elders past, present and future. Aboriginal and Torres Strait Islander people should be aware that this website may contain images, voices and names of deceased persons.

pattern