3 Process for developing indicators

3.1 Prioritising areas for indicator development

Indicator development will reflect priorities agreed with NHS England, the Department of Health and Social Care and the devolved administrations in Northern Ireland and Wales. These organisations in turn may establish processes to engage with other stakeholders and respondents.

Initial suggestions are based on publication of:

new national policy, for example, the NHS Long Term Plan
new or updated NICE quality standards or guidance
new or updated reports of current national performance.

The NICE indicator work programme is agreed on at least an annual basis and more frequently if demand requires it. Indicators may also be developed or assured for organisations that commission NICE to undertake specific pieces of indicator related work. A summary of all indicators in development or under review will be available on the NICE website.

3.2 Indicator advisory committee consideration

Areas for indicator development (see section 3.1) are presented to the indicator advisory committee alongside relevant guidance recommendations and current practice data. The indicator advisory committee advises on progression to the development stages by examining current variation in practice, opportunities to improve clinical outcomes, early feasibility assessment and the content of evidence-based guidance.

To utilise externally developed indicators and avoid system wide duplication of effort, the indicator advisory committee may also use the criteria outlined in appendix B to assess the validity of indicators developed by external organisations for inclusion on the NICE menu.

3.3 Indicator development stages

3.3.1 Indicator drafting

Drafting indicator wording and specification is an iterative process undertaken primarily by NICE and the National Collaborating Centre for Indicator Development (NCCID). Advice may be sought from NHS England, the Department of Health and Social Care, experts on the indicator advisory committee, and from experts involved in developing relevant NICE quality standards or guidelines.

3.3.2 Testing

All NICE indicators undergo testing to assess feasibility and acceptability. Testing is undertaken primarily by NICE and the NCCID. The approach to testing indicators is agreed with partner organisations and the indicator advisory committee and should be appropriate to:

the indicator's intended use (for example, as an aid for quality improvement purposes only or inclusion within a national pay-for-performance framework)
the existence of similar indicators that are already being used in practice.

The testing options available include, but are not limited to:

desktop review to assess availability of existing data sources
quantitative data analysis of relevant and available data
qualitative feedback from a workshop of experts, patients and service users
piloting general practice level indicators in around 30 GP practices with quantitative data extraction and qualitative feedback from GPs and practice staff.

Figure 1 Illustrative example of how testing may be used

3.3.3 Consultation

NICE asks for comments from stakeholders and respondents (including patient organisations and professional groups) on potential new indicators during a 4-week public consultation. They are asked to comment on:

risk of unintended consequences
barriers to implementation
impact on equality groups.

Stakeholders and respondents may also be asked specifically about any important areas for consideration that have been identified.

NICE informs stakeholders and respondents in advance about the public consultation by email and on the NICE website. Once it begins stakeholders and respondents can see the proposed indicators on the NICE website and submit comments. Comments received after the deadline for submission will not be considered. Responses to comments will be made available on the NICE website alongside committee minutes.

3.3.4 Cost-effectiveness analysis

Indicator development may include a consideration of cost effectiveness when indicators are intended for inclusion within a pay-for-performance framework.

3.4 Indicator advisory committee review

3.4.1 Testing and consultation review

The indicator advisory committee considers the results of indicator development (including the testing results, equality analysis and any cost-effectiveness analysis) alongside comments submitted during the public consultation. The indicator advisory committee advises on progression to the NICE indicator menu using criteria to assess the validity of indicators as outlined in appendix B. Committee considerations against each of these criteria will be published; however, they will not be used to assign a grading or score.

3.4.2 Resource impact analysis

A high-level assessment of resource impact is undertaken for indicators published as suitable for use in a pay-for-performance framework. This considers the likely resource impact of additional activity resulting from the implementation of new indicators. For more information, see resource impact on the NICE website.

3.4.3 Threshold setting

NHS England and the devolved administrations in Northern Ireland and Wales may ask the indicator advisory committee to propose achievement thresholds if indicators are intended for inclusion within a pay-for-performance framework.

Setting of thresholds is usually based on factors such as the baseline level of achievement, maximum expected levels of achievement and current levels of variation.

3.5 NICE Guidance Executive

Indicators recommended for progression to the NICE menu are presented to the NICE Guidance Executive for final approval before publication. When considering indicators for publication, the NICE Guidance Executive assesses whether:

the agreed process and methods have been followed
the indicators are consistent with NICE quality standards, NICE guidance or other sources of high-quality evidence
the indicators promote equality and avoid unlawful discrimination.

If a major issue is identified, the NICE indicator team will work to resolve it with assistance from the indicator advisory committee chair and members. The NICE Guidance Executive does not generally comment at other stages of indicator development.

3.6 Publication

Indicators that progress to the NICE indicator menu are available on the website and may be accompanied by:

a summary that includes:
- the outcomes that the indicator aims to improve
- an indication of the intended use (for example, as an aid for quality improvement purposes only or for inclusion within a pay-for-performance framework)
- an indication of the system or provider level at which the indicator is intended to be used
- an expected population size and if a minimum number of patients is required, related to its intended use
- a methodology for constructing the indicator including available data sources, inclusions, exclusions and personalised care adjustments
- an expected review date.
an assessment of validity using the criteria outlined in appendix B
cost-effectiveness analysis (if relevant)
resource impact analysis
equality and health inequality impact assessment.

3.7 Reviewing NICE indicators

NICE will review published indicators using the assessment criteria in appendix B when they reach their review date or the underlying guidance is updated. The indicator advisory committee will advise on actions required to ensure the indicators continue to be valid or propose retirement from the NICE menu (including the need for consultation on indicators to be retired). Development of new indicators based on updated recommendations will follow section 3.3.

An indicator may also be reviewed when an unanticipated risk or consequence is identified or a patient safety alert is published. Corrections or changes to a published indicator will be made if an error:

puts people using services at risk, or negatively impacts on their care or
damages NICE's reputation or
significantly affects the meaning.

If an error in a published indicator is identified, we will follow NICE's internal policy for dealing with errors. The individual or organisation who reported the error will be contacted in writing, and we will explain our rationale for the decisions and actions taken. Routine maintenance changes may also be made after publication of a NICE indicator. These include minor changes such as updating or fixing broken links or updating standard text in line with agreed template changes.

NICE process and methods