NICE process and methods

5 Production

5 Production

5.1 Equality and diversity considerations

'Evidence summaries: unlicensed and off-label medicines' (ESUOMs) are developed in accordance with NICE equality scheme.

5.2 Process and timescales

ESUOMs are not formal NICE guidance and therefore are not subject to the same intensity of process as other NICE products.

Table 1 below shows the key steps in the development of ESUOMs.

Table 1 Key steps for developing an ESUOM with timelines

Key step

Completed by

Topic selected and referred to supplier

Week 0

Scope topic

Week 0

Contact manufacturer and regulator for data

Week 0

Literature search

Week 1

Searching for evidence

Sifting and selecting the evidence

Appraising and categorising the evidence

Week 2

Week 2

Week 2

Authoring the ESUOM

Produce initial draft of ESUOM

Preliminary technical check of initial draft

Week 5

Week 6

Review of draft ESUOM

Initial draft sent to manufacturers, regulators, specialist commentators, NICE Clinical Guidelines team and NICE Technology Appraisals team for review

Week 6

Initial draft of information for the public sent to NICE Public Involvement Programme

Week 6

Review comments received and produce revised draft

Week 9

Quality assurance of the ESUOM

Technical check of content by NICE senior adviser

Week 10

Editorial check of content by NICE publishing team/ NICE senior adviser

Week 12

Internal sign off by Medicines and Prescribing Centre Programme Director, Associate Director or Consultant Clinical Adviser

Week 13

Manufacturer invited to check for any remaining factual errors

Week 13

Sign off and publication of the ESUOM

Guidance Executive sign off

Week 14

Manufacturer informed of date that ESUOM will be published

Week 15

Publication on NICE website

Week 16

5.3 Scoping of individual topics

The Medicines and Prescribing Centre scope each ESUOM topic, supported by the NICE Guidelines Information Services team. The scoping confirms the following:

  • Key contacts at the pharmaceutical manufacturer

  • Key contacts at the Medicines and Healthcare products Regulatory Agency (MHRA) or European Medicines Agency (EMA), to obtain evidence held on file that is not confidential

  • Specialist commentators (through the NICE Medicines and Prescribing Associates and other existing NICE networks, see section 3.8)

  • Commentators for the information for the public, in liaison with the NICE Public Involvement Programme (PIP, see section 3.2)

  • Terms for a literature search to identify published clinical trial data that reflect the indication for the unlicensed or off-label medicine

  • Arrangements for identifying:

    • regulatory status

    • relevant published trials, or other data on the unlicensed or off-label medicine

    • evidence of clinical effectiveness for the unlicensed or off-label medicine in the condition under consideration

    • safety issues, encompassing key adverse drug reactions, precautions and contraindications providing an indication of frequency of the adverse effects if possible

    • incidence and prevalence of the condition, what treatment alternatives exist and an estimate of current drug usage

    • cost of the medicine and the cost of alternative treatment options.

5.4 Contacting the manufacturer and medicines regulator

NICE asks the manufacturer[5] to support the production of the ESUOM by providing any of the following data it holds (within 10 working days):

  • Key published clinical trials relating to the indication being reviewed in the ESUOM and information regarding ongoing or recently completed studies.

  • The licence status within the European Union or the UK, including whether or not the manufacturer expects to hold a UK marketing authorisation for this medicine in this indication within the next 2 years or is aware that a medicine licensed in the UK for this indication is likely to become available from another manufacturer within 2 years.

  • The presentation of the medicine, including brief details of form, strength and pack size (if relevant).

  • The usual dose, if known, or best estimate from the available data.

NICE also contacts the MHRA (or EMA, as appropriate) to ask them for any evidence they hold on file that is not confidential.

Both the manufacturer and the MHRA (or EMA) are sent the timelines for the production of the ESUOM, including the deadlines for receipt of data and the expected dates for comment on a draft of the ESUOM.

5.5 Literature search

5.5.1 Searching for evidence

A literature search is conducted by the NICE Guidelines Information Services team according to the agreed scope and strategy. The search strategy is documented and included in an appendix to the published ESUOM. Quality assurance for the search process is also documented.

The literature search is intended to locate the best (highest quality) available published evidence relating to the efficacy, safety and cost effectiveness of the medicine (recognising that evidence relating to cost effectiveness of unlicensed or off-label medicines is scarce). In addition, explicit reference is made to information in the summary of product characteristics (if one exists) relating to precautions, warnings and undesirable effects and also to published advice from the Medicines and Healthcare products Regulatory Agency (MHRA) (or the European Medicines Agency [EMA] or Food and Drug Administration [FDA] if no there is no relevant MHRA advice). Cost information is obtained from the current Drug Tariff, or if the product is not listed there, the current edition of MIMS, or if the product is not listed there, the manufacturer.

5.5.2 Sifting and selecting the evidence

The final set of search results are sifted using the title and abstract of each article, first applying exclusion and then inclusion criteria. These include the basic criteria as set out below.

First sift

This process removes evidence based on the following exclusion criteria:

  • articles of poor relevance against search terms

  • publication types that are out of scope:

    • non-English language studies

    • conference abstracts[6]

    • review protocols (for example, Cochrane review protocols)

    • articles if neither the abstract nor full text is freely available online.

Second sift

This sift of evidence includes relevant primary research that addresses the use of the medicine within the defined indication under review. If robust randomised controlled trials or systematic reviews are available, they form the basis of the review. However, given the nature of the work, the best available evidence on which to produce the evidence summary may include evidence other than randomised controlled trials. Evidence that has been excluded from a previous NICE clinical guideline is not normally included.

The reasons for inclusion and non-inclusion are recorded based on the second sift, as well as a 'long list' of those studies that are excluded from the first sift.

5.5.3 Appraising and categorising the prioritised evidence

The evidence for critical appraisal is prioritised and the reasons for non-inclusion of evidence are recorded. The full text of the prioritised evidence is appraised using an assessment form suitable for the type of evidence.

5.6 Authoring of the ESUOM

The ESUOM is drafted using a standard template, which includes sections relating to the following:

  • Key points from the evidence, including:

    • a brief summary of the key points

    • an executive summary of the main sections of the document as detailed below

  • Introduction and current guidance

    • general information about the disease or condition, incidence or prevalence and overall survival rate

    • alternative treatment options with links to relevant guidance/evidence

  • Product overview

    • information about the drug's action, the product's regulatory status, the course of treatment and cost

  • Evidence review

    • a review of the available evidence, with relative strengths and weaknesses of evidence and the evidence selection process

  • Context and estimated impact for the NHS

  • Relevance to NICE guidance programmes

  • References

  • Information for the public.

5.7 Review of the draft ESUOM

The draft ESUOM is sent to the identified external specialist reviewers, the manufacturers, the NICE Clinical Guidelines team, the NICE Technology Appraisals team and the drug regulators for review. The draft information for the public is also sent for comment to the NICE PIP team. Any comments received are recorded and considered within the production of the revised draft. Actions and feedback to commentators are also recorded.

5.8 Quality assurance of the ESUOM

Quality assurance of the ESUOM involves a detailed check of all content, to ensure all sections of the document contain statements and conclusions that are fair and balanced. They must accurately reflect the evidence reviewed and be substantiated by an explicit and appropriate source of evidence. A further check for clarity, grammar, spelling and style is carried out, in conjunction with the NICE publishing team. All drafts and any changes to drafts are recorded for audit purposes.

Once sign-off is received from the Medicines and Prescribing Centre Programme Director, Associate Director or Consultant Clinical Adviser, the manufacturer is given the opportunity to review the near-final draft to check for any factual errors (1 working day) and any necessary corrections are made by the Medicines and Prescribing Centre. NICE Guidance Executive reviews the ESUOM, and if appropriate, approves the ESUOM for publication, ensuring that due process has been followed in its development. The manufacturer[7] is informed of the scheduled publication date, and may request an embargoed copy of the ESUOM to be sent to them 24 hours before publication.

5.9 Publication of the ESUOM

The final ESUOM is uploaded and made available online through the Medicines and Prescribing Centre page of the NICE website.

The NICE Communications team develops a communications plan for the ESUOM, together with the Associate Director within the Medicines and Prescribing Centre, and is responsible for disseminating the ESUOM once it has been published.


[5] For some unlicensed or off-label medicines, more than 1 manufacturer may be involved in the development of the drug. On these occasions, all relevant manufacturers based in the UK are contacted for information.

[6] Studies that have been reported only as conference abstracts or otherwise not reported in full are excluded because they cannot be critically appraised. However, the ESUOM may indicate if key clinical trials are ongoing or have been completed but not yet published in full.

[7] For some unlicensed or off-label medicines, more than 1 manufacturer may be involved in the development of the drug. On these occasions, all relevant manufacturers are informed.