Tools and resources

Developing NICE guidelines: the manual – appendices A to P

Appendix J: Call for evidence and expert witnesses

Appendix J: Call for evidence and expert witnesses

There are 2 approaches to seek evidence from stakeholders:

  • A call for evidence: the developer asks registered stakeholders (and other organisations or individuals as needed) to submit evidence in response to a specific question. This can be useful when there may be relevant evidence that was not identified by the searches (see the section on examples of relevant evidence not routinely identified by searches).

  • Expert witnesses: the developer asks experts to provide evidence based on their experience and specific expertise, or to help the committee interpret the evidence. They may attend committee meetings to provide their testimony and answer questions from committee members.

Both approaches are designed to provide committees with additional relevant information for decision making. These approaches may be appropriate when:

  • there is ongoing research because a condition, intervention or service is relatively new

  • studies have been published only as abstracts

  • there are unpublished data on the off-label use of medicines or on harms

  • there are unpublished economic models

  • there are unpublished records (at a national or local level) of the experiences of people using services, their family members or carers, or practitioners working in services.

This appendix only covers approaches that provide additional evidence or expertise. See the chapter on decision-making committees for information on how we ensure that committees have the knowledge and experience to interpret the evidence (for example, by inviting co-opted members to the committee).

Documenting the chosen approach

Evidence can be sought to address a review question, or to support guideline development without directly addressing a review question.

The following criteria should be met and documented before deciding whether to use a call for evidence or expert witnesses:

  • The results of all relevant formal searches for evidence should have been assessed and a gap in the evidence identified (unless it has been agreed upfront that formal searches are unlikely to find evidence and therefore have not been conducted).

  • If additional evidence will be sought to address a review question, the committee should consider whether there is enough evidence to make recommendations, and agree that the following are not appropriate options:

    • making recommendations based on either informal or formal consensus

    • only making recommendations for research

    • making no recommendations.

  • The committee has a reasonable expectation that they will be able to make recommendations based on evidence from stakeholders or experts.

  • Seeking evidence from stakeholders or experts will be an efficient use of the developer's time and will add value to the guideline (as agreed with the quality assurance team). In particular, the developer should be confident that seeking evidence from stakeholders or experts is a better way of getting more evidence than modifying the review protocol to broaden the searches (for example, by including a wider range of study designs or including evidence from indirect populations).

Deciding whether to use a call for evidence or expert witnesses

Once a decision has been made to request evidence, a call for evidence should usually be the first approach. If expert witnesses are used instead of, or as well as, a call for evidence, this decision should be documented (specifically, why a call for evidence was not considered appropriate or sufficient).

Call for evidence

A call for evidence can be made at any point during guideline development, but ideally it should happen early on. The amount of time given to respondents to submit evidence depends on the type of evidence and level of detail needed. It is typically 2 to 4 weeks but can be longer.

A call for evidence needs a protocol that specifies the inclusion and exclusion criteria for the types of evidence that will be considered. If the question used for the call for evidence is identical to an existing review question for the guideline, the call for evidence can use the same protocol. If the call for evidence question used does not match a review question, a new protocol should be developed.

The developer should make it clear to stakeholders that NICE will not consider the following material as part of a call for evidence:

  • promotional material

  • unsubstantiated or non-evidence-based assertions of effectiveness

  • opinion pieces or editorial reviews

  • potentially unlawful or other inappropriate information.

Occasionally, relevant data are held by regulatory authorities. If this is the case, the regulatory authority may be asked to release the data as part of the call for evidence.

Normally, only references or links should be submitted, or details of contacts for unpublished research. The developer will then obtain full copies of all relevant papers or reports, paying a copyright fee if necessary. If the data sources are open access or the copyright holder has given permission, copies of full papers (in electronic or hard copy form) can be submitted.

Confidential information

Information or data that may be considered confidential can include data that may influence share price values ('commercial in confidence') and data that are deemed intellectual property ('academic in confidence', that is, awaiting publication).

Confidential information should be kept to an absolute minimum. For example, information submitted should be limited to the relevant part of a sentence, a particular result from a table or a section of code. NICE does not allow a whole study to be designated confidential. As a minimum, a structured abstract of the study or economic model must be made available for public disclosure during consultation on the guideline. Results derived from calculations using confidential data are not considered confidential unless back‑calculation to the original confidential data is possible.

When the developer sends out a call for evidence, respondents are asked to complete a checklist that identifies the location of all confidential information contained in their submission, and for how long the information is likely to remain confidential. In addition to completing the checklist, respondents should indicate the part of their submission that contains the confidential information. All confidential information should be underlined. Information that is submitted under 'commercial in confidence' should also be highlighted in turquoise; information submitted under 'academic in confidence' should be highlighted in yellow. The underlining and highlighting should be maintained so that the committee knows which parts are confidential.

When documents are prepared for consultation and publication, NICE works with the data owners to agree a compromise between confidentiality and transparency, and strive to release as much information as possible. Any confidential information is removed, and a note added to explain what has been done.

Submitting evidence

Respondents should submit published and unpublished data using a submission form for a call for evidence. They should also complete a:

Information is available to help respondents identify confidential information and fill out the checklist.

Documenting evidence received in response to a call for evidence

Information received in response to a call for evidence should be recorded systematically and checked against evidence identified from other searching (for example, literature searches), to make sure it has not already been assessed.

All evidence provided should be screened against the protocol. If any evidence is excluded, the reasons for this should be explained. The reasons for exclusion should be as detailed as the reasons given when a study identified from literature searching is excluded at full-text level.

All evidence should be analysed and reported in the same way as information found from searches. See the section in the manual on reviewing the evidence. For example:

  • If evidence directly addresses a review question, it should be quality assessed, analysed and reported in the same way as evidence from searches.

    • This includes presenting the evidence in evidence tables, quality assessment of the individual studies (see the appendix on appraisal checklists, evidence tables, GRADE and economic profiles), and analysis and full GRADE or CERQual profiles (if relevant).

    • The call for evidence should be reported in an appendix to the evidence review. The results of the call for evidence should be discussed in the discussion of the evidence review, alongside the discussion of the evidence from the searches.

    • When an evidence review contains evidence from both searches and a call for evidence, it should be clear which recommendations are based on which sources of evidence or combinations of evidence types.

  • If the evidence is used to set parameters for a health economic model, the limitations and uncertainties in the evidence should be considered and discussed in the same way as for parameters from other sources.

When submitting evidence in response to a call for evidence, stakeholders are asked to disclose whether their organisation has any direct or indirect links to, or receives or has ever received funding from, the tobacco industry. Disclosures will be included with the evidence presented to the committee.

Expert witnesses

There are 2 ways to use expert witnesses in guideline development:

  • Expert evidence: getting analysable data in the same way as a call for evidence, but from selected experts rather than all stakeholders.

  • Expert testimony: providing information to the committee that cannot be formally analysed or quality assessed.

Expert witnesses can come from a variety of backgrounds, and include both professionals and lay people. The same processes and reporting standards should be used for all expert witnesses. Additional support and adjustments should be provided if needed for a witness to take part.

Expert evidence

Expert witnesses may be invited to provide expert evidence (appraisable data) to the committee after a call for evidence, or without holding a call for evidence:

  • After a call for evidence: respondents who have provided evidence as part of this process may be invited to provide additional evidence as expert witnesses. There should be a clear explanation of what additional evidence is needed, beyond what was supplied through the call for evidence. There should also be a clear justification for why it is appropriate to invite a particular respondent and not others.

  • No call for evidence: sometimes it is clear that a wider call for evidence would not be helpful and only a limited number of people should be invited to submit evidence. For example, the data needed might be held by 1 organisation (such as a national registry). There should be a specific justification for why a call for evidence was not considered appropriate, and for the choice of experts invited to provide evidence.

As with a call for evidence, the evidence from an expert witness should be analysed and reported in the same way as evidence from searches.

Expert testimony

When seeking expert testimony, there should be a specific justification for why a call for evidence was not considered appropriate or sufficient, and for the choice of experts invited to give testimony.

The expert should be given specific topics and questions to answer at the committee meeting. They should not just be invited to give their views on a broad topic.

The expert should complete a structured form (with support from the developer, if needed) summarising their testimony. The form should contain all the relevant information shared with the committee. If this form was completed before the meeting, it should be updated after the meeting with any additional information not previously included. See the form for expert testimony.

Expert testimony provides data that cannot be reported in the same way as data from searches or calls for evidence. Because of this, an alternative approach to reporting is needed. As a minimum:

  • The forms summarising the expert testimony should be reproduced and summarised in the evidence review document.

  • The committee discussion section should give the committee's views on the validity and applicability of the expert testimony. If more than 1 person gave expert testimony on the same topic, the level of consistency between the testimonies should be discussed, along with the committee's views on possible reasons for any inconsistencies.

  • The committee discussion section in the evidence review and rationale section in the guideline should make it clear when recommendations are wholly or partly based on expert testimony. This explanation should indicate which parts of the testimony link to specific recommendations, and summarise the committee's discussion of how confident they were that the testimony supported the recommendations. The explanation should cover all the standard elements for a discussion section: effectiveness, cost effectiveness, implementation issues, and ethical considerations.

Examples of evidence not routinely identified by searches

  • Ongoing research when an intervention or service is relatively new

  • Interim study results (not yet published) for longer-term studies

  • Studies that have been published only as abstracts

  • Health needs assessments

  • Protocols

  • Local pilot studies

  • Data on health inequalities

  • Business cases

  • Financial reports

  • Analyses of primary data

  • Data from patient registries and healthcare databases

  • Unpublished studies of the experiences of people using services, their family members or carers, or practitioners

  • Unpublished data about the off‑label use of medicines

  • Unpublished data on harms

  • Audit data

  • Implementation case studies

  • Unpublished economic models

Example webpage for a call for evidence

Call for evidence – end of life care for adults: service delivery

What we need

We need the following information for the guideline we are developing on end of life care for adults:

Service delivery models to:

  • identify people who may be entering the last year of their life

  • support people to stay in their preferred place of care (for example, out of hours services)

  • facilitate smooth transitions between care settings (for example, discharge planning teams)

  • facilitate continuity and coordination of care (for example, multidisciplinary team working)

  • reduce inappropriate and avoidable hospital admissions (for example, community health services and telehealth)

  • facilitate discharge back to the community from other settings (for example, rapid discharge pathways).

We are particularly interested in information promoting equality of opportunity relating to age, disability, sex, gender identity, ethnicity, religion and belief, sexual orientation or socio-economic status.

We would like information on:

  • service delivery models that report measurable outcomes, for example:

    • the number of people who die in their preferred place of death

    • quality of life

    • use of hospital and community services (including staff time or any other information on resource use)

    • costs associated with providing or implementing the service delivery model.

We cannot accept promotional material, non-evidence-based assertions of effectiveness or opinion pieces.

Sending information

For published information, send reference details only (authors, title, date, journal or publication details [including volume and issue number, and page numbers]). Do not send a PDF or Word document, or a paper copy.

For unpublished information, send:

  • a link to any relevant trials registered with the Cochrane Central Register of Controlled Trials, or with the US National Institutes of Health trials registry

  • paper or electronic copies of other relevant unpublished information.

Highlight any confidential sections (unpublished research or commercially sensitive information) in unpublished information. For more details about this, see the section on commenting on the draft guideline in the chapter on the validation process for draft guidelines and dealing with stakeholder comments.

Complete the call for evidence form [insert link] and the checklist for confidential information form [insert link], including the declaration of any links with, or funding from, the tobacco industry. Please email these forms with any relevant information by [insert deadline] to: INSERT EMAIL ADDRESS HERE

Alternatively, please send hard copies to:

INSERT POSTAL ADDRESS HERE

We look forward to receiving this information and thank you in advance for your help.

Submission form for a call for evidence

Call for evidence – [insert guideline title]

Please send the information before XX/XX/XXXX

  • When submitting evidence that is published, please send reference details only (authors, title, date, journal or publication details [including volume, issue and page numbers]).

  • We are unable to accept electronic or hard copy of published material unless data sources are open access, or the copyright holder has given permission.

  • We can accept unpublished reports, and local reports and documents.

Respondent organisation or individual Evidence submission Published or unpublished material How the evidence can be obtained

-

-

-

-

Information for respondents on a call for evidence

Section 1 – How to identify confidential information

To ensure that the guideline development process is as transparent as possible, NICE considers that evidence the committee has used to make its decisions should be publicly available. Ideally, all the evidence seen by the committee should be available to all stakeholders. Under exceptional circumstances, unpublished evidence can be accepted under agreement of confidentiality. However, NICE expects stakeholders to keep confidential material within a submission to an absolute minimum. Confidential data includes:

  • commercial in confidence data, which may influence share prices

  • academic in confidence data, which are awaiting publication and deemed to be intellectual property.

Such data should be consistent with the following principles:

  • Information in the public domain, anywhere in the world, may not be marked as confidential.

  • Results of trials relating to products that have received regulatory approval should be available for scrutiny.

  • When trial results are due to be published in a journal after they would be released by NICE, as a minimum a structured abstract should be made available for public disclosure during consultation on the guideline. The structured abstract should follow a recognised format for a full trial report, such as that provided by the CONSORT statement.

  • The same principles apply to the release of information in the form of an economic model. The full economic model, in electronic format, should be available for scrutiny by the developer. As a minimum, a structured abstract of the economic model should be submitted for public disclosure during consultation on the guideline.

Confidential information will be reviewed by the developer and the committee and – when necessary – any external experts (expert witnesses) will be invited to attend committee meetings.

  • If information is confidential, then any specific mention of that information will be removed or 'blanked out' from the draft guideline for consultation and the final guideline published on the NICE website. The confidential information will not appear in the final guideline, unless the confidential status is removed before publication.

  • NICE asks respondents to reconsider any restrictions on the release of data if there appears to be no obvious reason for them, or such restrictions would make it difficult or impossible for NICE to show the evidence for its recommendations.

Section 2 – Important notes to consider before completing the checklist for confidential information

  • Marking a whole submission confidential is not acceptable.

  • For each submission, respondents should complete the checklist below.

  • In addition to the checklist, all confidential information should be underlined. Commercial in confidence information should also be highlighted in turquoise and academic in confidence information should be highlighted in yellow. Please make sure that the checklist is completed accurately and corresponds to the highlighted and/or underlined text in your submission.

  • Results derived from calculations incorporating confidential data will not be considered confidential unless releasing those results would enable back-calculation to the original confidential data.

  • If the status of information changes during guideline development, a new checklist for confidential information must be completed as soon as possible.

  • If the confidential status of information is expected to change during guideline development, exact embargo dates (for example, after a conference presentation) or approximate dates (for example, after an article has been accepted for publication) should be given.

  • If no checklist for confidential information is received with a submission, all information in that submission and any accompanying appendices or attachments will automatically be considered not to be confidential.

Section 3 – Checklist for confidential information

To be completed in full and returned to [insert name], [insert role] (preferably electronically), as a separate file to your evidence submission, by the submission deadline.

If the developer does not receive a completed checklist, then all information contained in your submission will be considered as not confidential.

Respondent:

Guideline title:

Summary of the evidence submitted:

Does your submission contain any confidential information (see previous for more information)? Please check appropriate box

No

If no, please proceed to section 4 – Disclosure

Yes

If yes, please complete the following table in full (insert or delete rows as necessary).

All confidential information should be underlined. Information that is submitted under 'commercial in confidence' should also be highlighted in turquoise; information submitted under 'academic in confidence' should be highlighted in yellow. The underlining and highlighting should be maintained so that the committee knows which parts are confidential.

Checklist for confidential information
Page where confidential information appears Nature of confidential information (commercial in confidence or academic in confidence) Rationale for confidential status Timeframe of confidentiality restrictions

-

-

-

-

-

-

-

-

-

-

-

-

Please state whether the timeframe given is exact or approximate. For academic in confidence material, please state either the date and title of the conference at which the information will be made public, or the date of submission and title of the journal to which the relevant paper has been submitted, together with the journal's stated turnaround time.

Disclosure form




















Please disclose any past or current, direct or indirect links to, or funding from, the tobacco industry.

Section 5 – Declaration

I confirm that all relevant material pertinent to the call for evidence has been disclosed to the developer.

I confirm that any confidential sections of the submission have been underlined, any commercial in confidence sections have been highlighted in turquoise and any academic in confidence have been highlighted in yellow, and that if any change occurs to the above information, a new checklist will be submitted.

Name of person completing checklist:      

Contact details (telephone/email):      

Date:      

Form for expert testimony

Expert testimony to inform NICE guideline development

Section A: Developer to complete
Name:

Role:

Institution/Organisation (where applicable):

Guideline title:

Guideline Committee:

Subject of expert testimony:

Evidence gaps or uncertainties:

[Research questions or evidence uncertainties that the testimony should address are summarised below]

.
Section B: Expert to complete
Summary testimony:

References to other work or publications to support your testimony' (if applicable):

Disclosure:

Declaration of interests:

Note: If giving expert testimony on behalf of an organisation, please ensure you use the DOI form to declare your own interests and also those of the organisation – this includes any financial interest the organisation has in the technology or comparator product; funding received from the manufacturer of the technology or comparator product; or any published position on the matter under review. The declaration should cover the preceding 12 months and will be available to the advisory committee. For further details, see the NICE policy on declaring and managing interests for advisory committees and supporting FAQs.

Expert testimony papers are posted on the NICE website with other sources of evidence during consultation and when the guideline is published. Any content that is academic in confidence should be highlighted and will be removed before consultation and publication if the status remains at this point in time.


This page was last updated: