NICE process and methods

3 The appraisal process

3 The appraisal process

Although there are many similarities between the standard technology appraisal and fast track appraisal processes, they differ in process steps and timelines between the start of the appraisal and the first appraisal committee meeting. Differences between the processes are described in section 3.2.8 and sections 3.3.23 to 3.3.27.

3.1 General points

3.1.1 NICE sends the name and contact details of the project manager assigned to an individual appraisal to all consultees and commentators. Consultees and commentators should send all correspondence, including consultation responses about an individual appraisal, to the project manager.

3.1.2 NICE sends correspondence for an appraisal electronically (or in other formats on request) to key contacts identified by each consultee and commentator organisation. It is therefore essential that consultees and commentators notify the project manager of any change in contact details, or in organisation or company name, during the appraisal process.

Process timelines

3.1.3 It is not possible to set absolute timelines for all stages of the appraisal process. The length of time needed for each stage can vary depending on the nature of the particular appraisal. The timelines set out in table 3, table 4 and table 5 indicate the minimum number of weeks for each stage of the appraisal process. Additional time may be given to particular stages if they coincide with public holidays.

3.1.4 Throughout an appraisal, up-to-date information about timelines and progress is available on the NICE website. Further information is available from the project manager.

3.1.5 If possible, NICE informs consultees and commentators about timeline changes during an appraisal and the reasons for these changes. Sometimes, however, if the reasons are commercially sensitive, NICE cannot disclose the details. NICE works with the company to release as much information as possible to consultees and commentators, and on the NICE website.

Information handling – general considerations

3.1.6 NICE adheres to the principles and requirements of data protection legislation, including the General Data Protection Regulation and the Freedom of Information Act when dealing with information received during an appraisal.

3.1.7 Organisations who want to be involved in an appraisal must sign a confidentiality agreement first (formally known as the confidentiality acknowledgement and undertaking) to be considered a participating consultee or commentator. After this, NICE can release appraisal documents to them.

3.1.8 NICE is required to meet the requirements of copyright legislation. If a company cites journal articles in its submission, it must include the full articles in its submission and have copyright clearance to do so.

3.1.9 If NICE requires journal articles for its own use within the process, NICE will obtain the article, paying a copyright fee when necessary.

3.1.10 NICE requires the medical director of the company to sign a statement confirming that all clinical trial data necessary to address the remit and scope of the technology appraisal as issued by the Department of Health and Social Care and NICE, within the company's or any of its associated companies'[1] possession, custody, or control in the UK or elsewhere in the world, have been disclosed to NICE or its authorised agents.

3.1.11 NICE requires companies to consent to NICE being provided directly by European Economic Area regulatory authorities all clinical trial data necessary to address the remit and scope of the technology appraisal as issued by the Department of Health and Social Care and NICE. This includes all data that have been submitted to the regulatory authorities by the company or any of its associated companies and that were relevant to the granting of a marketing authorisation, and for NICE to use those data in carrying out the technology appraisal. NICE will only ask regulatory authorities directly after having first approached the company for the information and the company is unable or unwilling to provide the information in a timely manner.

3.1.12 Care should be taken when submitting information about individual people. Personal and sensitive information, for example, the name of a person's clinician, should be removed from submissions.

3.1.13 NICE encourages consultees to make their individual submissions accessible – for example, by putting them on their own websites after they have sent their submission to NICE.

3.1.14 NICE may comment publicly on the content of an appraisal during the process and when draft or final guidance has been produced. The following circumstances may also apply:

  • NICE reserves the right to comment publicly if there has been an unauthorised disclosure from a confidential NICE document before it has been published on the NICE website. NICE's chief executive will take this decision. NICE will inform consultees and commentators of this decision as soon as possible.

  • NICE reserves the right to issue a correction if a public comment is made on an appraisal consultation document (ACD) or final appraisal document (FAD) that could mislead or misinform.

3.1.15 Consultees and commentators, including any other party that has signed a confidentiality agreement for the appraisal, are responsible for treating appraisal documents that are not in the public domain as confidential until NICE makes those documents, or the data within them, public. NICE considers individuals in a consultee or commentator organisation who see appraisal documents to be bound by the terms of the confidentiality agreement signed by the consultee or commentator organisation.

3.1.16 Any organisation or individual not directly employed by the consultee or commentator organisation is a third party. Consultees and commentators may release appraisal documents to third parties when:

  • it is necessary to enable the consultee or commentator to contribute to the appraisal and

  • the third party has seen and agreed to be bound by the terms of the NICE confidentiality agreement.

3.1.17 Consultees and commentators may discuss confidential appraisal documents with other consultees and commentators but, before doing so, they must be satisfied that the other consultees and commentators have signed and returned their confidentiality agreement to NICE.

3.1.18 In the technical report, committee papers (see section 3.5.3), ACD and FAD, NICE reserves the right to use any material submitted during the appraisal process that is not marked as confidential by the consultee, or which ceases to be so under section 3.1.16. All confidential information should be clearly signposted and marked as such in the committee papers.

3.1.19 If changes are made to the expected marketing authorisation or CE mark during the regulatory approval process, NICE will discuss the implications with the evidence review group (ERG) and the company and agree how to incorporate the changes into the submission, the ERG report and the technical report.

3.1.20 NICE will not make public any final guidance documents on a technology until UK regulatory approval has been granted and the technology's price is known. NICE may share documents with participating consultees and commentators who have signed and returned a confidentiality agreement to NICE.

Information handling – confidential information

3.1.21 To ensure that the appraisal process is as transparent as possible, NICE considers it essential that evidence on which the appraisal committee's decisions are based is made available to stakeholders and is publicly available. In some circumstances, unpublished evidence is accepted under agreement of confidentiality. Such evidence includes commercial-in-confidence information (for example, the findings of a research project considered confidential because public disclosure could have a significant impact on the commercial interests of a particular company) and academic-in-confidence information (because public disclosure would seriously jeopardise the ability of the data owner to publish the information in a scientific paper).

3.1.22 NICE has the following principles for handling confidential information:

  • Information marked as confidential should be kept to an absolute minimum. Data that are likely to be fundamental to the appraisal committee's decision-making cannot be marked as confidential (for example, the list price of a technology after launch and incremental cost-effectiveness ratio [ICER] estimates).

  • Reasons for confidentiality must be stated clearly, including the date of expected release into the public domain by the data owner, with specific consideration to be given to release of data by regulators as part of granting of the marketing authorisation for a medicinal product.

  • When a NICE document quoting evidence from a clinical trial is released before the results are published in a journal, or released through the European Medicines Agency's transparency policy, as a minimum a structured abstract should be made available for public disclosure. This abstract should follow a recognised format for a full trial report, such as that provided by the CONSORT statement. An equivalent approach is needed for all data and studies that underpin, and are included in, economic analyses and models.

  • Evidence designated as academic in confidence (but not 'commercial in confidence') can be presented at appraisal committee meetings with members of the public and press present.

  • Executable economic models used by companies in their submission will be made available (on request) to consultees and commentators who have signed a confidentiality agreement.

  • If NICE wishes to publish or publicly share data regarded by the data owner as academic or commercial in confidence, both NICE and the data owner will negotiate to find a mutually acceptable solution, recognising the need for NICE to support its recommendations with evidence and the data owner's right to confidentiality. However, the data owner retains the right to make a final decision about the release of confidential information into the public domain.

  • Details of a patient access scheme or commercial access agreement, once referred to NICE for consideration in a technology appraisal, are not confidential except when NHS England has agreed that a simple scheme discount is confidential. In this case the discount and any data that could lead to back-calculation of the discount will not be shared with consultees and commentators or released into the public domain.

  • When the details of the patient access scheme or commercial access agreement are not published in final NICE guidance, the NHS must have access to the details, so that providers and commissioners are able to properly account for the patient access scheme.

  • NICE will not share confidential details of a simple discount in a patient access scheme for a comparator technology with the company for a new technology being appraised. For each technology with a comparator that has a confidential patient access scheme, the company must include a 'discount' field in its economic model. This should allow the user to input any value between 0 and 100%, which is then applied as a discount to the list price of the technology. By providing this feature in its model, the company will be responsible for the initial programming, which the ERG will check. All parties should then be confident that the discount is programmed correctly. The ERG will be authorised to know the exact level of discount for all patient access schemes in the appraisal.

  • The ERG will use the list price of the comparator in its main report when reproducing the company's analyses and for any exploratory analyses. To allow the committee to explore the impact of using the actual cost of the comparator in the analyses, the ERG will also create a confidential appendix to its report, which will reproduce all analyses from the main ERG report using the exact level of discount for the comparator. Although the results of these analyses are classed as commercial in confidence, NICE will have to publish an ICER range that informs the recommendation(s), after taking into account the exact level of the discount provided in the commercial arrangement for the comparator.

  • If NICE is challenged that confidential information it has received should be publicly released in the interests of fairness during an appraisal, at appeal, through judicial review or otherwise, data owners must, on request, promptly reconsider whether it is necessary to maintain confidentiality. If disclosure is not possible, the data owner must be prepared to assert publicly that the information is confidential, and must submit evidence justifying why NICE should maintain that confidentiality. Without such assertion and evidence, NICE is entitled to conclude that the information is no longer confidential.

3.1.23 Appraisal committee members and ERG members, and in the case of a cancer drug appraisal the Cancer Drugs Fund clinical lead, attending the appraisal committee meeting will be provided with all confidential information submitted.

3.1.24 The clinical experts, NHS commissioning experts and patient experts who attend the appraisal committee meeting will be provided with all confidential information submitted, except confidential patient access schemes and commercial access agreements.

3.1.25 In the event that the technical engagement phase (see sections 3.3.14 to 3.3.22) occurs before regulatory approval of a technology, all information marked as confidential will not be released to consultees and commentators even though they have signed a confidentiality agreement.

3.1.26 In the event that the technical engagement phase (see sections 3.3.14 to 3.3.22) happens after regulatory approval of a technology, all information marked as confidential, except confidential patient access schemes and commercial access agreements, will be released to consultees and commentators who have signed a confidentiality agreement.

3.1.27 If a company's evidence submission, or a statement from a non-company consultee contains confidential information, it is the responsibility of the submitting organisation to provide 3 versions:

  • A version for NICE, the appraisal committee and the ERG with all the confidential information marked.

  • A version for experts and consultees and commentators with all the confidential information marked, and with information about the patient access scheme and commercial access agreement redacted.

  • A version for public release after the committee has met, in which all the confidential information is redacted.

3.1.28 A checklist will be provided that must be completed by the consultee at the time of submission, listing all confidential information included in the submission or statement, the reason for its confidentiality, and the date at which it will no longer be considered confidential. If NICE does not receive a completed checklist with a document, none of the information will be considered confidential.

3.1.29 Data owners will be asked to check that confidential information is correctly marked in documents created by others in the technology appraisal process before release; for example, the technical report and ERG report.

3.1.30 NICE releases the documents listed in table 2 to consultees and commentators during the appraisal process. NICE publishes these documents on its website at least 7 calendar days after they have been sent to consultees and commentators. After NICE has published these documents on its website, they are no longer confidential.

Table 2 Documents NICE publishes during the appraisal process

Document (confidential information redacted in public documents; see sections 3.1.26 to 3.1.27)

List of consultees and commentators

Final scope and remit for the appraisal

Company's evidence submission(s)

Statements/submissions from non-company consultees and experts

Evidence review group (ERG) report

Clarification questions and responses

Technical report

Comments from consultees, commentators and experts on the technical report, and responses from NICE

If produced, the appraisal consultation document (ACD)

Comments from consultees and commentators and members of the public on the ACD, and responses from NICE

Final appraisal document (FAD)

3.2 Start of the appraisal and evidence submission

See figure 4 and figure 5 for an overview of the process and timelines.

3.2.1 The process consists of 3 distinct phases: start of the appraisal and evidence submission, evidence review (including initial clarification and technical consideration), and appraisal. The first phase can only begin after the scoping phase has been completed and NICE has received formal referral from the Secretary of State for Health and Social Care.

3.2.2 It is the responsibility of the company to inform NICE as soon as possible of any potential regulatory developments or delays. This should be done by contacting the project manager.

3.2.3 Before the start of the appraisal, the company has the opportunity to discuss the decision problem that follows from the draft scope with the NICE team and ERG representatives. The company must submit an outline of how it intends to approach the decision problem when preparing the evidence submission. This outline is to include, but is not limited to, evidence sources to be used, evidence likely to become available during the appraisal and how this might be managed, the planned approach to disease and economic modelling, potential challenges in interpreting the evidence, and the proposed approach to handling of uncertainty. The meeting will also allow companies to discuss potential handling of patient access schemes or commercial access agreements and proposals for access to the fast track appraisal process. The meeting is not an opportunity to discuss or request changes to the scope.

3.2.4 NICE will publish the final remit and final scope (see section 2.4), the name of the ERG and the list of consultees and commentators on its website at the start of an appraisal. Each appraisal is assigned to a project team. The roles of key members of the project team are summarised in table 1.

3.2.5 The appraisal starts when NICE sends consultees and commentators the invitation to participate, together with a list of key dates.

Figure 4 Summary of the appraisal process

Figure 4 Summary of the appraisal process

Figure 5 Summary of the appraisal process when an ACD is produced

Figure 5 Summary of the appraisal process when an ACD is produced

Evidence submission from the company

3.2.6 NICE invites the company to provide an evidence submission using a detailed submission template. The deadline for receipt of the evidence submission is 60 calendar days from invitation. After receiving this NICE sends it to the ERG for review.

3.2.7 The information needed for the evidence submission is derived from the approach NICE uses to evaluate the clinical and cost effectiveness of health technologies. This approach is outlined in NICE's guide to the methods of technology appraisal.

3.2.8 For fast track appraisals the evidence must be submitted in templates; the standard submission template or, if it is a cost-comparison case, in the cost-comparison template.

3.2.9 During the 60‑day submission preparation stage there will be at least 1 opportunity for the company to discuss key issues with NICE and, if needed, the ERG. NICE will ask the company to provide an update on their submission before the meeting. This engagement will also allow companies to discuss potential regulatory developments during the appraisal and the potential inclusion and handling of commercial arrangement proposals. During the 60‑day submission preparation stage companies can request additional engagement with NICE. Engagement will depend on availability of the NICE team at the time of request.

3.2.10 If the company plans to submit an economic model, it should inform NICE which software will be used. NICE accepts fully executable economic models using standard software, that is, Excel, DATA/Treeage, R or WinBUGs. If the company plans to submit a model in a different software package, it should tell NICE in advance. NICE, in association with the ERG, will then investigate whether the requested software is acceptable. When the company submits a fully executable electronic copy of the model, it must give NICE full access to the programming code. Care should be taken to ensure that the submitted versions of the model program and the written content of the evidence submission match.

3.2.11 NICE offers to send the economic model (in its executable form) to consultees and commentators during engagement on the technical report. If the model contains confidential material that the data owner is unwilling to share with consultees and commentators, despite the assurances provided through the signed confidentiality agreements, NICE will ask the company to redact the model if this can be done without severely limiting the model's function. Consultees and commentators must make requests for a copy of the model in writing. NICE provides the model on the basis that the consultee or commentator agrees, in writing, to the following conditions of use:

  • The economic model and its contents are confidential and are protected by intellectual property rights, which are owned by the relevant company. It cannot be used for any purpose other than to inform the recipient's understanding of the committee papers.

  • The economic model cannot be published by consultees or commentators (except by the company who owns the model), in whole or in part, or be used to inform the development of other economic models.

  • The model must not be run for purposes other than to test its reliability.

3.2.12 If the company wishes to include a patient access scheme or commercial access agreement proposal as part of its submission, specific requirements apply (see section 4 for more information).

Submissions from non-company consultees

3.2.13 NICE invites all non-company consultees to make a submission providing information on the potential clinical and cost effectiveness of a treatment using the appropriate templates available on the NICE website. The submission should reflect the experience of patients, healthcare professionals and commissioners of current standard treatment in the NHS in England and the potential impact of treatment on health-related quality of life. Implementation issues, such as staffing and training requirements, should also be included. Consultees have 60 calendar days to provide their submission to NICE. After receiving the evidence submissions, NICE sends them to the ERG and technical team for information.

3.3 Evidence review

Initial clarification and additional analysis

3.3.1 After receiving the company's evidence submission, the NICE technical lead and the ERG assess whether the submission is complete and whether the decision problem is specified appropriately with reference to the final scope.

3.3.2 If the evidence submission is incomplete or the decision problem is not specified appropriately, the technical lead consults with the ERG and sends a letter of clarification and any requests for additional analyses to the company within 21 calendar days of receiving the submission. The company has 14 calendar days from the date of the correspondence to respond. When the company provides additional analyses, it should include full descriptions of the analyses as appendices to the original submission. If necessary NICE will organise a clarification meeting between the NICE team, the company and the ERG to resolve any issues.

3.3.3 If requests for clarification and any additional analyses delay the published timelines, NICE will inform consultees and commentators and publish the reason for the delay on its website.

3.3.4 At the same time as the response to the clarification request the company should review the confidential status of information in its evidence submission before the appraisal committee meeting (see sections 3.1.21 to 3.1.30 for details on submission of confidential information).

3.3.5 The company should not submit additional evidence during the evidence review phase unless NICE requests or agrees to this in advance.

Terminating an appraisal

3.3.6 NICE aims to ensure that the company prepares the best possible evidence submission for the appraisal committee. NICE will not validate the submission but it will help to clarify substantive issues. If, after all reasonable requests for clarification, NICE is not satisfied that the evidence submission is adequate for the appraisal committee to make a decision or if no evidence submission has been received, the centre director or programme director will recommend to NICE's guidance executive that the appraisal should be terminated. NICE will inform the company that an inadequate evidence submission has been received. NICE will subsequently advise the NHS that the appraisal has been terminated and that NICE is unable to make a recommendation about the use in the NHS of the technology because no evidence submission was received from the company. NICE will also provide an explanation to help the NHS make local decisions on making the technology available.

3.3.7 A terminated appraisal can be restarted if the company indicates that it wishes to make a full evidence submission.

Evidence review group report

3.3.8 The ERG prepares a report on the clinical and cost effectiveness of the technology in line with NICE's guide to the methods of technology appraisal. The report is based on a review of the company's evidence submission and advice from the ERG's clinical advisers. The ERG prepares the report in line with the National Institute for Health Research Health Technology Assessment Programme's quality criteria, the scope of work as identified in the service level agreement between the Department of Health and Social Care, the NIHR Evaluation, Trials and Studies Coordinating Centre (NETSCC) and NICE, and will use an agreed report template. The ERG is responsible for the content and quality of the report.

3.3.9 The ERG critically evaluates the evidence submission. If the ERG, as part of exploratory analyses, amends the company's model, NICE will make the analyses available to the company at the technical engagement stage. All other consultees and commentators may request, in writing, the ERG analyses during technical engagement.

3.3.10 Following receipt of the ERG report, NICE will share a copy with the company, for information only. This will allow the company time to prepare for the technical engagement stage of the process.

Technical report

3.3.11 After receiving the ERG report the technical team will create a technical report. NICE may also seek advice from the selected experts at this stage, if additional clarification on the submitted individual expert statement is needed.

3.3.12 The technical report will include:

  • a commentary on the evidence received

  • a commentary on the written statement

  • technical judgements of the evidence by the technical team

  • reflections on the NICE structured decision-making framework.

3.3.13 The technical report will be accompanied by:

  • the company submission (and model when appropriate)

  • the ERG's critique of the company submission

  • statements from stakeholder organisations and clinical and patient experts

  • the overview of the discussions with the company about the technical aspects of the case

  • preliminary scientific judgements of the technical team.

Technical engagement

3.3.14 The technical report is usually sent to consultees and commentators for comment within 30 calendar days of NICE receiving the ERG report. NICE notifies consultees and commentators if a delay is expected.

3.3.15 The technical report is also sent to the clinical experts, NHS commissioning experts, patient experts and, in the case of a cancer drug appraisal the Cancer Drugs Fund clinical lead, for comment.

3.3.16 The purpose of the technical engagement is to seek views on the judgements made by the technical team and to allow the company to consider how it could mitigate the remaining uncertainties in the case for clinical and cost effectiveness in the evidence base.

3.3.17 Consultees, commentators and the experts have 28 calendar days to submit comments on the technical report. Comments must be submitted electronically. Approximately half way through the engagement period, NICE will hold a teleconference meeting with the company. When considered necessary by the technical team, experts will also be invited.

3.3.18 NICE will ask the company to re-confirm the expected timing of marketing authorisation or CE mark in the UK.

3.3.19 If a comment contains confidential information, it is the responsibility of the organisation or person who submitted the comment to provide 2 versions; one with all the confidential information marked and another with the confidential information redacted (to be published on NICE's website), together with a checklist of the confidential information. Detailed instructions on sending NICE confidential information are available from the project manager.

3.3.20 During technical engagement, new evidence and analyses can only be accepted if the technical team agrees that this information is likely to affect the appraisal committee's judgements. The new evidence must be presented in a separate appendix to the comments on the draft technical report. NICE may need to extend timelines and reschedule the subsequent committee meeting to allow the new evidence to be considered. The company must inform NICE, in writing, of its intention to submit new evidence and analyses, as early as possible.

3.3.21 Any ERG review of new evidence will not normally be sent out for additional technical engagement before the committee meeting.

3.3.22 If comments received on the economic model need a company or ERG response, NICE sends those comments to the company or ERG. Their responses will be tabled at the next appraisal committee discussion.

Table 3 Expected timelines for the appraisal process: starting the process, preparing the ERG report and technical engagement*

Calendar days (approximately)

Step 1

NICE invites organisations to participate in the appraisal as consultees or commentators

0

Step 2

NICE invites selected clinical experts, NHS commissioning experts and patient experts to attend the appraisal committee meeting and asks them to submit a written statement

30

Step 3

NICE receives evidence submissions from consultees

60

Step 4

NICE requests clarification on the evidence submission

80

Step 5

Selected clinical experts, NHS commissioning experts and patient experts submit written statements

90

Step 6

NICE receives the ERG report

120

Step 7

The technical team prepare the technical report and send it out for engagement

150

Step 8

NICE compiles the supporting documentation (see section 3.5.3) and sends it to the appraisal committee

195

* Timelines may change in response to individual appraisal requirements.

Fast track appraisal process: evidence review, confirming the process and developing the technical report

3.3.23 When NICE receives a company evidence submission for a fast track appraisal, the NICE team, supported by the ERG, will confirm whether the selection criteria (see section 2.4.31) are met, and that the appraisal can follow the fast track process.

3.3.24 If the selection criteria are not met, the appraisal will follow the standard process. If this is the case and a company has made a case for the fast track process based on cost comparison, the company will be asked to make a submission using the full cost-utility template used for the standard process and the topic will be rescheduled into the work programme at the earliest opportunity.

3.3.25 If an appraisal is not selected for the fast track process, NICE will provide the company with the rationale for this decision. If the company does not agree with this, it must contact NICE within 2 days of receiving the decision stating reasons for its objections. The centre director will then review the routing decision rationale and the company's objections and make a final decision on the appropriate route for the appraisal.

3.3.26 If NICE confirms that an appraisal can follow the fast track process NICE will establish a technical team, who will produce a technical report.

3.3.27 The company will have an opportunity to consider the report before the appraisal committee meets. NICE will not issue the report for technical engagement before the appraisal committee meeting.

3.4 External participation in the appraisal process

Participation of experts

3.4.1 NICE encourages consultees and commentators to nominate clinical experts and patient experts. This is so that the experts can provide their views and experience throughout the appraisal process, help to clarify issues that the technical team has identified, respond to the technical engagement and attend the appraisal committee meeting. NICE asks NHS England and the 2 clinical commissioning groups selected at random to nominate NHS commissioning experts to respond to the technical engagement and attend the appraisal committee meeting.

3.4.2 Experts identified during the scoping process may be invited to take part in the appraisal.

3.4.3 The PIP public involvement adviser gives advice and information to the patient and carer organisations nominating experts and to people interested in becoming patient experts. Patient organisations may nominate both patient and clinical experts.

3.4.4 The nominating organisation and the experts (clinical, patient or NHS commissioning) jointly complete a nomination form. The form includes a section asking the expert to provide a 50‑word summary describing their experience and knowledge of the condition, any experience of the technology, and any previous involvement with NICE. The form also asks for any conflicts of interest as per the NICE declarations of interest policy.

3.4.5 The chair of the appraisal committee, with input from the NICE team and PIP teams, selects experts from the nominations received and from those identified during scoping. The choice of clinical experts and patient experts is based on the nominees' experience of the technology and the condition(s) that the technology is designed to treat. Selection also takes into account the NICE policy on declaring and managing interests for NICE advisory committees. If possible, the clinical experts and patient experts will have complementary rather than similar backgrounds and experiences. NICE uses the following criteria to select clinical experts, NHS commissioning experts and patient experts for appraisal committee meetings:

  • They agree to be bound by the terms and conditions of NICE's confidentiality agreement.

  • They agree to their name and affiliation appearing in the ACD and FAD.

  • They have knowledge or experience of the condition or the technology under appraisal or the way it is used in the NHS.

  • They are willing and able to discuss the condition and the technology at a committee meeting where there may be members of the public and press observing.

  • They are familiar with the purpose and processes of NICE (the PIP public involvement adviser at NICE can give patient experts an overview that enables them to contribute to the technical engagement and discussions at appraisal committee meetings).

  • They are prepared to declare any interests they have in the technology under appraisal at committee meetings.

3.4.6 Additionally, the following criteria are used to select clinical experts:

  • They are in active clinical practice and have specialist expertise in the subject area of the appraisal.

  • Their principal place of work is in the NHS.

  • If they have acted as a clinical expert for the company, or the ERG, they agree to declare this in their personal statement and at appraisal committee meetings.

  • They hold no official office (that is, no paid employment, unpaid directorship or membership of a standing advisory committee) with the technology company or any relevant comparator technology companies. However, there is discretion to invite an expert who holds official office when the work of the committee would be seriously compromised without their testimony.

3.4.7 Usually, 2 clinical experts, 2 patient experts and 2 NHS commissioning experts are selected. NICE asks them to submit a short written personal statement on the technology and the way it should be used in the NHS in England. If the clinical experts and patient experts support the submission made by their nominating organisation they do not need to submit a separate statement. NICE gives the written statements to the appraisal committee and publishes them as part of the committee papers. The experts are expected to engage fully in the technical engagement phase of the process ahead of the appraisal committee meeting. Further advice about the contribution of clinical experts, NHS commissioning experts and patient experts is available from the NICE project manager.

3.4.8 Clinical experts, NHS commissioning experts and patient experts attend appraisal committee meetings as individuals and not as representatives of their nominating organisation. NICE aims to select a cross-section of people from the nominations received for clinical experts and patient experts, taking into account potential conflicts of interest. For example, for patient experts, NICE would select a person with direct personal experience of the condition and, if possible, the technology, and a member of a patient, carer or professional organisation.

3.4.9 For all cancer drug appraisals the clinical lead for the Cancer Drugs Fund, or a nominated deputy, is invited to submit a statement and attend the appraisal committee meeting to:

  • receive, consider and interpret evidence on the clinical and cost effectiveness of health technologies for treating cancer that are being appraised by NICE, particularly when these are potentially eligible for funding from the Cancer Drugs Fund

  • provide the appraisal committee with expert insight into how the Cancer Drugs Fund operates to help its decision-making.

3.4.10 For fast track appraisals all selected experts will not be routinely invited to take part in the appraisal committee meeting. In exceptional circumstances, the technical team may agree to invite clinical, patient or NHS commissioning experts to the meeting to help address specific uncertainties that cannot be resolved in writing.

3.4.11 NICE includes the names and affiliations of the selected clinical experts, NHS commissioning experts, patient experts and the Cancer Drugs Fund clinical lead in the minutes of appraisal committee meetings.

3.4.12 It is important that sufficient expertise feeds into all stages of the technology appraisal. NICE welcomes and values the input from patient experts, NHS commissioning experts and clinical experts. Experts will be able to opt out of attending the appraisal committee meeting if they feel that their views are adequately reflected in the technical report, key areas of uncertainty have been addressed, and their attendance would not add to the committee discussion.

Participation of company representatives

3.4.13 Two representatives from the company (normally 1 with health economics expertise and 1 with medical expertise) for the technology being appraised can attend part 1 of the appraisal committee meeting discussions. The chair will ask them to respond to questions from the appraisal committee. The chair will also ask the representatives to comment on any matters of factual accuracy before concluding part 1 of the meeting. The chair may ask the representatives to remain for part of the closed session (part 2) of the committee meeting, specifically to respond to questions from the committee about confidential information in the company's submission. Each representative must:

  • be an employee of the company or have been involved in developing the company's evidence submission

  • have relevant detailed knowledge of the technology under appraisal to engage effectively with the appraisal committee

  • be able to comment on the clinical or cost effectiveness of the technology

  • agree to be bound by the terms and conditions of NICE's confidentiality agreement

  • be willing and able to discuss the condition and the technology with members of a large committee at a meeting where there may be members of the public and press observing

  • be familiar with the purpose and processes of NICE.

3.4.14 Company representatives will not receive the confidential appendix that the ERG may create for an appraisal with a comparator that has a confidential patient access scheme or commercial access agreement.

3.4.15 The ACD, FAD and the minutes of appraisal committee meetings will include the industry representation at the appraisal committee meetings but not name the representatives who attended.

3.5 Appraisal

3.5.1 The appraisal phase of the process has 4 possible stages:

  • consideration of the evidence at an appraisal committee meeting to discuss the content of either the ACD or FAD

  • development of, and consultation on, the ACD (if needed)

  • review of the ACD (if produced) after comments from consultation at a second appraisal committee meeting

  • development of the FAD.

Preparing for the appraisal committee meeting

3.5.2 The technical team and the ERG meet to discuss the results of the technical engagement step, if held, and prepare the presentation for the committee meeting.

3.5.3 The committee papers are usually circulated to all attendees (except members of the public) 2 weeks before the meeting, and consist of:

  • a link to the final scope of the appraisal and the list of consultees and commentators

  • the technical report, including comments from technical engagement (if held) and the technical team's summary of them.

3.5.4 Appraisal committee meetings are usually open to members of the public and press. This supports NICE's commitment to openness and transparency. It enables stakeholders and the public to understand how evidence is assessed and interpreted and how consultation comments are taken into account.

3.5.5 To promote public attendance, the meetings in public team at NICE publish a notice and draft agenda on the website at least 28 calendar days before the appraisal committee meeting. Members of the public who wish to attend can register on NICE's website. Up to 20 places will be available, depending on the size of the venue. If any meeting is oversubscribed, NICE may need to limit the number of places offered. To allow wide public access, NICE reserves the right to limit attendees to 1 representative per organisation. The closing date for registration is 14 calendar days before the meeting. NICE will contact applicants to let them know whether they have a place at the meeting. NICE publishes the final agenda on its website 7 calendar days before the meeting.

Appraisal committee meeting

3.5.6 When the appraisal committee meets for the first time to discuss an appraisal, it is intended that a FAD will be developed. Sometimes it may develop an ACD (see section 3.5.26). The committee papers include the written evidence submitted by consultees and commentators. The verbal evidence is drawn from discussions with invited clinical experts, NHS commissioning experts, patient experts, ERG representatives and in the case of a cancer drug appraisal, the Cancer Drugs Fund clinical lead.

3.5.7 Committee decisions are normally based on consensus. If a vote is taken, it will be noted in the minutes. More information on how appraisal committees consider the evidence and make decisions is available in NICE's guide to the methods of technology appraisal.

3.5.8 The committee can conclude that the technology is:

  • recommended for routine commissioning or

  • not recommended for routine commissioning or

  • not recommended for routine commissioning, but recommended for inclusion in the Cancer Drugs Fund, or in some other form of managed access arrangement or

  • not recommended for routine commissioning, but invites the company to submit a proposal for inclusion in the Cancer Drugs Fund, or in some other form of managed access arrangement.

3.5.9 For fast track appraisals a FAD will be developed after the meeting. In exceptional circumstances, the committee may find it is unable to develop recommendations for the technology without further scrutiny, or further submission of evidence. If this is the case, NICE will publish a statement indicating that the committee is unable to make a recommendation.

3.5.10 For fast track appraisals, if a company wishes to resubmit after the committee has stated that it is unable to make a recommendation, the topic will be rescheduled into the committee work programme although it will not always be possible to prioritise the topic for immediate review.

Part 1 (public session)

3.5.11 Part 1 of NICE appraisal committee meetings is usually open to members of the public and press. There may be occasions when a meeting will be entirely closed because it is not possible to conduct business without referring to confidential information, or without discussions being commercially sensitive.

3.5.12 Members of the committee and people having direct input into the discussions declare their interests, which are recorded in the minutes. For further information on how NICE deals with conflicts of interest, please see the NICE policy on declaring and managing interests for NICE advisory committees.

3.5.13 The lead or technical team presents the appraisal topic to the other appraisal committee members and attendees, using the technical report as the basis for the introduction. The lay lead's role is to include the patient evidence in the topic introduction. This introduction does not pre-empt the committee's debate or drafting of the guidance.

3.5.14 Clinical experts, NHS commissioning experts and patient experts will be encouraged to help clarify issues about the submitted evidence, including responding to and raising questions, but they do not make a presentation to the committee.

3.5.15 Company representatives respond to questions from the appraisal committee and comment on any matters of factual accuracy.

3.5.16 The appraisal committee considers the evidence during the public session. However, it will not discuss commercial-in-confidence information, or information contained in a statement from a clinical expert, NHS commissioning expert or patient expert that has been marked as confidential during this part of the meeting. See section 3.1.22 for further details on how academic-in-confidence information is handled at appraisal committee meetings.

3.5.17 The ERG representatives answer questions from the appraisal committee and provide clarification on the ERG report.

3.5.18 Representatives from other guidance-producing teams (for example, guidelines and public health) at NICE who are responsible for developing NICE guidance in areas related to the appraisal may also attend the meeting to observe and advise the appraisal committee. These representatives must declare their interests and satisfy NICE's conflict of interest policy (see section 3.5.12).

3.5.19 NICE staff may present additional evidence, provide advice on NICE policies and procedures, and respond to questions from the appraisal committee.

Part 2 (closed session)

3.5.20 During the closed session, the appraisal committee considers commercial-in-confidence information and agrees the recommendations. Members of the public and press along with the clinical experts, NHS commissioning experts, patient experts, company representatives and the ERG representatives are asked to leave the meeting promptly before this discussion takes place.

3.5.21 The chair may ask clinical experts, NHS commissioning experts, patient experts, company representatives and ERG representatives to remain when confidential information is discussed, but the chair will ask them to leave before the committee agrees the recommendations in the ACD or FAD.

3.5.22 A patient expert can ask to have any personal, sensitive or confidential information heard by the committee in private. The patient expert should formally request this through the project team at NICE and it must be agreed with the chair of the committee before the meeting.

3.5.23 NICE staff and representatives from other guidance-producing teams at NICE who are responsible for developing NICE guidance in areas related to the appraisal may stay at the meeting while the appraisal committee agree the recommendations in the ACD or FAD; however, they play no part in decision-making.

3.5.24 The appraisal committee concludes the discussions and agrees the content of either the ACD (see section 3.5.28), which sets out its preliminary recommendations, or the FAD (see section 3.5.44), which sets out its final recommendations. After the meeting, the ACD or the FAD is drafted based on the discussions at the meeting, including the preliminary or final recommendations agreed by the appraisal committee. NICE may issue an ACD or FAD on a technology before that technology receives final UK regulatory approval (see section 3.1.20 for further information).

3.5.25 The outcome of the appraisal committee meeting will be shared with participating consultees and commentators within 7 calendar days of the committee meeting. This will be a brief statement of the committee decision.

Consultation on the ACD (if produced)

3.5.26 Normally, formal consultation (when an ACD is produced) takes place only if the preliminary recommendations from the appraisal committee do not recommend use of the technology, limit the use of the technology further than the marketing authorisation (or instructions for use) for the indication being appraised, or if the company is asked to provide further clarification on the commercial arrangements in their evidence submission.

3.5.27 NICE usually circulates the ACD to consultees and commentators within 21 calendar days of the appraisal committee meeting. NICE alerts consultees and commentators if a delay is expected.

3.5.28 The ACD summarises the evidence and views that have been considered by the appraisal committee and sets out preliminary recommendations. The ACD is not NICE's final guidance on a technology. The recommendations may change after consultation. The ACD usually contains:

  • the appraisal committee's preliminary recommendations to the NHS on the technology and how it should be used

  • a description of the technology, including its licensed indication and dosage and cost

  • a description of how the appraisal committee has interpreted the evidence together with the key issues raised by clinical experts, NHS commissioning experts and patient experts

  • the appraisal committee's preferred assumptions and maximum acceptable ICER, if appropriate

  • expectations about implementation of the recommendations, if appropriate

  • proposed recommendations for further research, if appropriate

  • the proposed date for considering a review of the guidance.

3.5.29 When a cancer drug has the potential to be recommended for use within the Cancer Drugs Fund, the appraisal committee will state the conditions for its use in the ACD and will identify the nature of the clinical uncertainty that should be addressed through data collection. Details of data collection, including a protocol and analysis plan (when applicable), will be set out in a managed access agreement.

3.5.30 The data collection arrangements for drugs being recommended through the Cancer Drugs Fund will be developed by the company, NHS England, Public Health England, NICE and the Cancer Drugs Fund clinical lead. Input from experts taking part in the appraisal will be requested when needed. The data collection arrangements will be completed before the final guidance is published. Further details can be found in the Cancer Drugs Fund data collection specification.

3.5.31 The ACD and any committee papers are sent to consultees, commentators, the clinical experts, NHS commissioning experts and patient experts for consultation. These documents are confidential until NICE publishes them on its website 7 calendar days after circulation. Information designated as commercial in confidence will be redacted from the public documentation. No clinical confidential information will be shared with consultees and commentators at this stage unless marketing authorisation (or the CE mark) has been received.

3.5.32 The purpose of the consultation is to seek views on the appraisal committee's preliminary recommendations and to determine whether they are an appropriate interpretation of the evidence considered. NICE invites comments on whether:

  • all the evidence available to the appraisal committee has been appropriately taken into account

  • the summaries of clinical and cost effectiveness are reasonable interpretations of the evidence

  • the preliminary recommendations are sound and constitute a suitable basis for guidance to the NHS

  • there are any equality issues that need special consideration that are not covered in the ACD.

3.5.33 Consultees and commentators (and the clinical experts, NHS commissioning experts and patient experts) have 28 calendar days from the date of sending to submit comments on the ACD. They must submit their comments in writing, preferably electronically.

3.5.34 NICE publishes the ACD on its website with an electronic comment facility and any additional committee papers not already shared on the NICE website (with confidential material redacted for public consultation) 7 calendar days after circulation to consultees and commentators.

3.5.35 If a comment contains confidential information, it is the responsibility of the organisation or person who submitted the comment to provide 2 versions, a complete version and another with the confidential information redacted (to be published on NICE's website), together with a checklist of the confidential information. Detailed instructions on sending NICE confidential information about an appraisal are available from the project manager (see section 3.1.22).

3.5.36 After the ACD has been developed, new evidence will not be accepted unless specifically requested by the appraisal committee (see section 3.5.37). The opportunity to provide additional evidence is offered at the technical report engagement stage.

3.5.37 The appraisal committee may find it is unable to develop recommendations for the technology without further scrutiny, or further submission of evidence. If this is the case, there is the possibility for a pause in the appraisal. NICE will ask the company to submit specific information and if relevant, further analyses. If the company has carried out new analyses, it must submit an updated version of the economic model. When the appraisal committee seeks such clarification, NICE will inform consultees and commentators within 7 calendar days of the committee meeting. After this pause, the committee will be required to make a recommendation, as set out in section 3.5.8.

3.5.38 When consultees and commentators submit comments that lead to a substantial revision of the committee's previous decision, involving a significant change in the recommendations, considerations or the evidence base, the centre director or programme director and the chair of the appraisal committee will decide whether it is necessary to prepare another ACD. If so, the consultation process will be repeated. The decision to produce another ACD will extend the timelines for the appraisal. NICE will distribute the committee papers with the second ACD, together with consultation comments and any new evidence not circulated with the previous ACD.

Appraisal committee meeting to develop the FAD

3.5.39 If an ACD is produced, the appraisal committee usually meets again, with members of the public and press observing, to consider the preliminary recommendations in the ACD in the light of the comments received. Before the meeting, NICE sends the appraisal committee members the full text of the comments from the consultees and commentators and a summary of any comments received from other people or organisations.

3.5.40 Representatives from the company, the ERG and from other guidance-producing teams at NICE (for example, guidelines and public health) who are responsible for developing NICE guidance in areas related to the appraisal, may attend the meeting. In exceptional circumstances, if clarification of issues raised during the consultation period is needed, the chair of the appraisal committee can, at their discretion, invite 1 or more of the clinical experts, NHS commissioning experts or patient experts to attend.

3.5.41 The appraisal committee discusses the responses to the ACD consultation in part 1 of the meeting (see section 3.5.11) and moves to a closed session (part 2, see section 3.5.20) to consider any confidential information and to agree the content of the FAD, which sets out the final recommendations. After the meeting, the FAD is drafted based on the discussions at the meeting and the final recommendations agreed by the appraisal committee.

3.5.42 If the company responds to the consultation by making an updated commercial offer and the revised ICER is below the maximum acceptable ICER specified by the appraisal committee in the ACD (see section 3.5.32), the chair can decide, on behalf of the appraisal committee, whether the company's proposal is likely to result in positive guidance. In these circumstances, the chair may decide that another committee meeting is not needed. A FAD is drafted and the final recommendations are agreed by the appraisal committee electronically. The final recommendations will be shared with participating consultees and commentators within 7 calendar days of sign-off. This will be a brief statement of the committee's decision.

3.5.43 If the committee has requested new analyses and the company has carried these out using the appraisal committee's preferred assumptions, if the revised ICER is below the maximum acceptable ICER specified by the appraisal committee in the ACD (see section 3.5.32), the chair may decide that another committee meeting is not needed. A FAD is drafted and the final recommendations are agreed by the appraisal committee electronically.

3.5.44 The FAD contains:

  • the appraisal committee's final recommendations to the NHS on the technology and how it should be used

  • a description of the technology, including its licensed indication and dosage and cost

  • a description of how the appraisal committee has interpreted the evidence together with the key issues raised by clinical experts, NHS commissioning experts and patient experts

  • the appraisal committee's preferred assumptions and maximum acceptable ICER, if appropriate

  • expectations about implementation of the recommendations, if appropriate

  • proposed recommendations for further research, if appropriate

  • the date for considering a review of the guidance.

3.5.45 The centre director or programme director signs off the final FAD and submits a report to NICE's guidance executive. The guidance executive checks that the appraisal committee has appraised the technology in accordance with the terms of the Secretary of State for Health and Social Care's referral and the scope. If satisfied, the guidance executive approves the FAD for publication on behalf of the NICE Board.

3.5.46 NICE issues the FAD to consultees so that they can consider whether to appeal against the final recommendations. They can also highlight any factual errors. Commentators and the experts receive the FAD for information and can also highlight any factual errors. Details of the appeal process are set out in NICE's guide to the technology appraisal and highly specialised technologies appeal process.

3.5.47 Any further analysis done by the company, NICE or the ERG during development of the FAD will be made available to consultees and commentators. When NICE sends the FAD to consultees and commentators, it also sends the comments received from consultees, commentators and experts on the ACD (if produced), together with NICE's responses to them, and the comments received from the public through the website.

3.5.48 NICE usually sends the FAD within 35 calendar days of the appraisal committee meeting to consultees and commentators. NICE notifies consultees and commentators if a delay is expected. NICE publishes the FAD and the committee papers, with confidential material redacted, on its website 7 calendar days after circulation to consultees and commentators.

3.5.49 In highly exceptional circumstances NICE may carry out further analysis. The ERG or Decision Support Unit (DSU) normally does this further analysis before NICE circulates the FAD. The centre director or programme director decides whether this should be done, with the chair of the appraisal committee and the NICE team. The decision is not taken lightly and is made to make sure that NICE is able to provide robust guidance to the NHS. If further analysis is done, NICE will inform consultees and commentators. NICE will distribute any such analysis to consultees and commentators and publish it on the website at the same time as the FAD.

Minutes

3.5.50 NICE publishes unconfirmed minutes of the appraisal committee meeting on its website within 28 calendar days of the meeting. When the appraisal committee has approved them, NICE publishes the confirmed minutes on its website normally within 6 weeks of the meeting. The minutes of an appraisal committee meeting provide a record of the proceedings and a list of the issues discussed.

Table 4 Expected timelines for the appraisal process if an ACD is produced

Calendar days (approximately) since process began

Step 8/10

Appraisal committee meeting to develop an ACD attended by clinical experts, NHS commissioning experts and patient experts.

210

Step 9/11

The ACD is produced. NICE distributes the ACD and publishes it on the website 7 calendar days later.

221

Step 10/12

Fixed 28‑calendar day consultation period on the ACD.

259

Step 11/13

Appraisal committee meeting to consider comments on the ACD from consultees and commentators, and comments received through the consultation on the NICE website. Appraisal committee agrees the content of the FAD.

270

Step 12/13

The FAD is produced. NICE distributes the FAD and publishes it on the website 7 calendar days later.

305

Table 5 Expected timelines for the appraisal process if an ACD is not produced

Calendar days (approximately) since process began

Step 8/10

Appraisal committee meeting to develop a FAD, attended by clinical experts, NHS commissioning experts and patient experts.

210

Step 9/11

The FAD is produced. NICE distributes the FAD and publishes it on the website 7 calendar days later.

245

Publication of the guidance

3.5.51 Unless there are any appeals by consultees, the FAD forms NICE's guidance on the use of the technology.

3.5.52 After receiving the FAD, any consultee (whether or not they are submitting an appeal) or commentator can ask for factual errors to be corrected. Some examples of factual errors are:

  • wrong names or misspelling of technologies or companies

  • errors in figures presented in the FAD

  • incorrect or incomplete quotes from a marketing authorisation or CE mark

  • text describing the facts incorrectly in the FAD.

3.5.53 The guidance executive considers all significant requests for correcting factual errors and decides whether to make changes to the FAD. This decision is made after any appeal proceedings have concluded. NICE then publishes the FAD as technology appraisal guidance on its website. NICE also publishes a lay version for patients and carers (known as 'information for the public').


[1] within the meaning of s.256 of the Companies Act.