Outcomes

Outcomes are intended or unintended changes that occur as a result of an intervention.

Understanding Outcomes

Outcomes are not activities or decisions - they are the changes that arise from activities or decisions. They include changes in knowledge, behaviour, decision-making, policies or systems.

Thinking about the difference that an intervention will make to individuals, groups, organisations, systems and populations can help to determine outcomes. Outcomes may be short term, for example, an individual may learn or develop a new skill, or long term, such as a change in organisational culture. Short term outcomes can contribute to achieving outcomes in the long term. For example, short term improvements in the learning environment for children may lead to longer term improvements in their learning outcomes. Research indicates that it takes anything from 2-4 years to 7-10 years to achieve full implementation, depending on the size and complexity of what is being implemented.

Implementation outcomes are changes relating to the implementation process.

Implementation outcomes are pre-conditions for achieving other outcomes, for example, for children, families and other people using services.

Defining and measuring implementation outcomes is important for the following reasons:

  • They help to provide vision and direction for implementation
  • They are indicators of implementation success
  • They highlight implementation processes
  • They help to ensure all stakeholders are on the same page
  • They are key parts of a logic model or theory of change
  • They can indicate progress in working towards better outcomes for services and people who use them

Measuring implementation outcomes

Different terms are used to describe implementation outcomes. They can be measured in terms of attitudes, opinions, intentions or behaviours. This table outlines different ways of measuring implementation outcomes developed by Proctor et al. (2011)

Acceptability

The perception among stakeholders that an intervention is agreeable, or satisfactory. If a change is not acceptable to stakeholders, it will be challenging to implement. Stakeholder engagement can support acceptability.

Adoption

The initial decision to employ the intervention.

Appropriateness

The perceived fit, relevance or compatibility of the intervention within a given setting. Stakeholders may feel that a new innovation is not consistent with the organisations mission, staff skills, roles or job expectations. Understanding this can help inform implementation efforts.

Feasibility

The extent to which an intervention can be carried out within a given setting. A variety of factors can influence feasibility, including resource requirements.

Fidelity

The degree to which the intervention is delivered as described. Quality, protocols, quantity are all aspects of fidelity.

Cost

The cost of implementing the intervention, and any costs required to support its implementation. Costs will vary in different settings, and depend on size and complexity.

Penetration

The integration of the new practices into a service setting and related systems. One way of calculating penetration is to multiply the number of organisations delivering an intervention by the number of individuals trained in, or expected to deliver the intervention.

Sustainability

The extent to which the intervention is institutionalised within ongoing operations. This means that it has become integrated into the routine way of working.

The A—Z of Implementation

There are a range of terms used when writing or talking about implementation and in implementation science. This glossary provides a short definition for each of these terms.

Adaptable components

Elements of an intervention which may be tailored to local settings during implementation without undermining the integrity of the intervention itself.

Barriers

Factors which hinder the implementation process and reduce the probability of successful implementation.

Capacity

The ability or power to do, understand or absorb something. This can apply to an individual, a team, an organisation or a whole system.

Coaching

A formal, typically short-term, arrangement between a coach and an individual focused on developing work-related skills or behaviours.

Community

A group of people living in an area or having characteristics in common (e.g. city, neighbourhood, organisation, service, business, professional association); the larger socio-political-cultural context in which an intervention is intended to operate.

Consultation

The action or process of formally discussing something with stakeholders, generally asking stakeholders a relevant question and receiving answers to that question. While the views of stakeholders may then be used to influence decisions, there is no commitment or requirement to do so.

Context

The set of circumstances or unique factors in which implementation takes place. This can refer to both the wider, systemic context, as well as the specific setting in which a specific intervention will be implemented.

Continuous Improvement Cycles

Ongoing use of emerging data and evidence on outcomes and implementation, and using that information to learn from experience, inform future implementation and improve outcomes. Progress is, therefore, achieved in an incremental manner over time.

Core Components

Indispensable elements of an intervention or implementation plan, which cannot be changed without undermining it. All core components should be delivered with fidelity.

Data-Based Decision Making

Using processes for collecting and analysing different types of data to guide decisions with the aim of improving outcomes on an ongoing basis.

Diffusion

A process by which an intervention is communicated through certain channels over time. The spread of ideas through diffusion is generally a passive process, following an unpredictable, unprogrammed, emergent and self-organising path, e.g. word of mouth.

Dissemination

An active, negotiated and influenced means of spreading an intervention or information about an intervention to relevant target groups.

Enablers

Factors which increase the probability of successful implementation.

Evaluation

A planned investigation of a project, programme, or policy used to answer specific questions. It can be related to design, implementation, results, and outcomes (cause and effect) of an intervention.

Evidence-Based Interventions

Practices, programmes, policies, strategies or other activities that have been empirically shown through scientific research and evaluation processes to improve outcomes to some degree.

Fidelity

Delivering an evidence-based intervention exactly as intended by those who developed it.

Framework

A structure, overview, outline, or system consisting of various descriptive categories and the presumed relationships between them.

Implementation

The carrying out of planned, intentional activities that aim to turn evidence and ideas into policies and practices that work for people in the real world. It is about putting a plan into action; the ‘how’ as well as the ‘what’.

Implementation Plan

A plan outlining the key activities, responsibilities, timelines and other important information required to achieve the desired outcomes from implementing an intervention.

Implementation Readiness

The extent to which organisations and individuals are both ‘willing’ to, and ‘capable’ of, implementing any specific intervention.

Implementation Science

The scientific study of how interventions are incorporated into service settings. It seeks to identify activities, contexts and other factors that increase the likelihood of successful implementation and lead to improved outcomes for people.

Implementation Strategy

Any specific method or action aimed at overcoming barriers, increasing the pace and effectiveness of implementation, and sustaining interventions over time.

Implementation Team

A group or structure that oversees, supports and attends to, moving an intervention through the stages of implementation. They actively use strategies to facilitate implementation.

Inputs

Resources needed to carry out activities and outputs.

Intervention

Any policy, practice, service or programme that is being implemented. It can be a change to something already in existence, or an entirely new intervention.

Leadership

The action of leading a group of people, or the ability to do this. This does not just apply to leading a whole organisation or system – leadership can take multiple forms and can occur at any level of an organisation or system.

Logic Model

An adaptable tool that describes the theory of change underpinning an intervention, a programme or a policy. It allows the user to systematically work through the connections between the components of an intervention or process, usually in graphical format on a single page.

Mentoring

A formal or informal arrangement which typically involves an ongoing relationship of support for significant transitions in knowledge, thinking and skills.

Model

A deliberate simplification of a phenomenon. Models are intended to be descriptive and need not be a completely accurate representation of reality to have value.

Monitoring

The routine and systematic collection of information against a plan. It makes use of existing data and information about inputs, outputs, outcomes, or about outside factors affecting an organisation or project, with a view to ongoing cycles of improvement.

Needs Assessment

A process which clarifies the extent to which needs, as well as enablers and barriers to meeting those needs, are accurately known and prioritised by an organisation or group of people.

Outcomes

Intended or unintended changes that occur as a result of implementing interventions. These changes can occur at the level of individuals, groups, organisations or population, and can occur in the short-, medium- or long-term.

Outputs

Key activities and areas of work that will help to achieve the desired outcomes.

Organisational Culture

The norms, values and beliefs that exist and govern behaviour within an organisation.

Resources

A stock or supply of money, materials, staff, and other assets that can be drawn on by a person or organisation in order to effectively implement an intervention.

Stakeholders

Anyone who is affected by or is involved in the intervention being implemented. This includes staff, public, clients, managers, professional bodies, unions, educators, policy-makers and funders.

Strategies

Methods or actions that aim to overcome barriers, increase the pace and effectiveness of implementation, and sustain interventions over time.

Sustainability

An intervention can be considered to be sustainable when not only have the process and outcomes changed, but the thinking and attitudes behind them are fundamentally altered and the systems surrounding them are transformed as well. In other words, the intervention has become an integrated or mainstream way of working rather than something ‘added on’.

Synthesis

A structured process where relevant information and evidence on a topic is gathered, reviewed, assessed and brought together to support decision making.

Theory

A set of analytical principles or statements designed to structure our observation, understanding and explanation of the world.

Theory of Change

An explicit, step-by-step statement of the expected relationship between the intervention and the outcome, i.e. why providing input X should lead to a change in outcome Z, by way of output Y.

Vested Interests

A special interest in maintaining or controlling an intervention, arrangement or institution, usually for personal gain.