Evidence generation plan
Open for comments Open until Request commenting lead permission
We have updated this service so that members of the same organisation can now collaborate on a joint online response.
Read our blog to learn more.
3 Approach to evidence generation
Evidence gaps and ongoing studies
Table 1 summarises the evidence gaps and the evidence available to the committee when the guidance was published. Information about evidence status is derived from the external assessment group's report. Evidence that did not meet the scope and inclusion criteria is not included.
REINFORCE trial
The REINFORCE trial is investigating the impact of robot-assisted surgery (RAS) as it is introduced and scaled up across NHS hospitals currently using the technology. The primary outcome measures include:
patient-level outcomes, such as:
quality of life and
complications
surgeon- or team-level outcomes, such as:
precision or accuracy and
surgery-specific workload
organisation-level outcomes, such as:
equipment failure
standardisation of operative quality and
overall economic or cost effectiveness
population-level outcomes, such as equity of access.
The study aims to recruit 2,560 participants and has an estimated completion date of April 2025.
RACER-Knee and RACER-Hip
The RACER-Knee and the RACER-Hip trials are investigating the clinical and cost effectiveness of knee and hip replacement surgery (respectively) of RAS using the Mako SmartRobotics platform, compared with conventional surgery. The studies are likely to collect data on many of the evidence gaps, but they include a 10-year follow up and are anticipated to end in 2032 (RACER-Knee) and 2033 (RACER-Hip). Interim data may become available before then (12-month follow up completes in 2024).
ROSAKnee, ApolloKnee and VELYS are not indicated for use in partial knee arthroplasty or total hip arthroplasty.
Data sources
Data could be collected using a combination of suitable real-world data sources and primary data collection. NICE's real-world evidence framework provides detailed guidance on assessing the suitability of a real-world data source to answer a specific research question.
The National Joint Registry (NJR) is the data source that is most likely to be able to collect the real-world data necessary to address the essential evidence gaps. The registry includes everyone having joint replacement surgery (conventional or robot-assisted) across private healthcare settings and in the NHS. The registry also records the specific robotic systems used, and links to the NHS Personal Demographics Service to get data for revision surgery and mortality outcomes.
NHS England's national patient-reported outcome measures (PROMs) programme records PROMs before and 6 months after surgery. The relevant PROMs measured for joint replacement include the EuroQol 5D (EQ-5D) 3L index score, Oxford Hip Score and Oxford Knee Score. Patient‑level data from the NJR can be linked to other datasets such as NHS Digital's Hospital Episode Statistics. This could support the evaluation of outcomes such as adverse events, further hospital appointments and referral for physiotherapy.
Combining these real-world evidence data sources will address most of evidence gaps around resource use and the impact on people's quality of life. The high-quality data and broad coverage within the NJR should enable relevant subgroup analyses to assess who the technologies might benefit.
The addition of outcomes to the registry is also unlikely within the timeframe of an early value assessment. Outcomes not already collected will need to be collected separately, for example in a prospective audit.
The quality and coverage of real-world data collections are of key importance when used in generating evidence. Active monitoring and follow up through a central coordinating point is an effective and viable approach for ensuring good-quality data with broad coverage.
Evidence collection plan
Most of the evidence gaps can be addressed through a real-world historical control study. For evidence gaps not addressed by the real-world evidence datasets, a prospective audit is proposed to collect data on the impact of RAS on surgical capacity.
Real-world historical control study with propensity score methods
A historical control study could compare outcomes before and after the implementation of RAS. This could assess the clinical impact of RAS as well as resource use associated with RAS, such as:
volume of procedures and RAS uptake
hospital stays
readmission
revision rates and
use of other associated services.
The NJR has data on RAS from March 2020 onwards, but this could enable collection of longer-term data such as revision rates (ideally up to 5 years). Hospital location data could inform evidence gaps around geographical access to RAS. Data on the location of RAS systems may be available through the National Equipment Tracking and Inventory System, which aims to provide visibility around equipment assets. Collection of other baseline patient characteristics such as sex, age, gender, BMI and ethnicity will enable relevant subgroup analyses. These baseline cohort differences may affect clinical outcomes and should be corrected for in future analyses.
Despite consistent eligibility criteria, non-random assignment to interventions can lead to confounding bias, complicating interpretation of the intervention effect. To minimise bias and identify a suitable control group, appropriate statistical approaches that balance confounding factors across comparison groups should be used, for example, using propensity score matching. The comparator group of primary interest is conventional surgery using manual techniques. NICE's Real-world evidence framework provides further detailed guidance on the planning, conduct and reporting of real-world evidence studies assessing comparative effects.
Prospective audit
Some of the evidence gaps around resource use will not be captured by the historical control study. For example, surgical time and total theatre time, or volume and cost of surgical consumables. An audit to collect data on the impact of RAS on surgical capacity is proposed to address these gaps. Technical failure rates should also be reported.
Data to be collected
Patient characteristics and outcomes
Information about individual characteristics at baseline, for example, age, sex, gender, body mass index (BMI), ethnicity, and where in the country the operation was done. Characteristics should include those needed for adjustment to address confounding, and for subgroup analysis.
Patient pain, mobility and functioning PROMs at baseline and post-surgery. Currently PROMs linked to the NJR are collected before surgery and 6 months after. Ideally, this information would also be collected at 12 and 18 months. PROMs should include Oxford Knee scores, Oxford Hip scores and EuroQol 5D (EQ-5D) 3L index scores (outcomes already collected and linked to the NJR).
Resource use
Immediate consumables and resourcing associated with surgery, including:
pre-operative CT imaging requirements
training time and costs
surgical and theatre accessories
staffing (number and NHS band)
total theatre time and total surgical time
volume of procedures per day and
implant costs.
Post-surgery treatment and service use, including:
length of hospital stay
readmission rates
number of physiotherapy sessions and
revision rates (stratified by implant type).
All costs associated with the immediate consumables and resourcing and with post-surgery treatment and services.
Subgroup analyses
Data could be stratified by:
patient characteristics including age, sex, gender, BMI and ethnicity
where in the country the procedure was done
people from Southeast Asian backgrounds
pre-existing medical conditions
people having more complex surgeries, such as alternative alignment approaches
physical status as defined by American Society of Anaesthesiologists risk scores.
Other important covariates should be chosen with input from clinical specialists to support subgroup analysis.
Safety
Adverse events, including conversion to manual surgery and dislocation.
Consequences of additional radiation exposure if more imaging is needed.
Data collection should follow a predefined protocol and quality assurance processes should be put in place to ensure the integrity and consistency of data collection. See NICE's real-world evidence framework, which provides guidance on the planning, conduct, and reporting of real-world evidence studies.
Evidence generation period
This will be 2 years to allow for setting up, implementation, data collection, analysis and reporting.
How are you taking part in this consultation?
You will not be able to change how you comment later.
You must be signed in to answer questions
Question on Consultation
Question on Consultation
Question on Consultation
Question on Consultation
Question on Document