3 Committee discussion
The diagnostics advisory committee considered evidence on home-testing devices for diagnosing obstructive sleep apnoea hypopnoea syndrome (OSAHS) from several sources, including an external assessment (EAG) report and an overview of that report. Full details are in the project documents for this guidance.
Impact of using home-testing devices for people with suspected OSAHS
3.1
The EAG identified 3 papers that reported patient experiences with the home-testing devices: Alsaif (2023) for Sunrise, Devani (2021) for AcuPebble SA100 and Mueller (2022) for WatchPAT 300. The patient experts explained the potential benefits of using home-testing devices for diagnosing and assessing severity of OSAHS. These included their ease of use, time and cost savings, and increased access to diagnosis and treatment. The committee heard that many people have undiagnosed OSAHS and that even before the COVID‑19 pandemic, the availability of diagnosis and treatment was variable. Using newer home-testing devices could increase access to home testing, leading to increased access to diagnosis and treatment. It could also potentially reduce waiting times, by reducing the number and frequency of visits to a sleep clinic. Using home-testing devices may also have a potential positive impact on a child's education by reducing delays for treatment. People often need to collect oximetry and respiratory polygraphy test kits from a hospital, whereas home-testing devices can be sent to people's homes and returned by post. This can save time and costs for the people using them and increase access to testing. This may particularly benefit people with reduced mobility, people who live far from the nearest sleep clinic and people who may not be able to take time off work or afford travel costs. Another benefit noted was that the devices are easy to use and do not involve the complexity of attaching bands or monitors. But a concern expressed was that some devices are smaller than the current tests, so may be more difficult to attach. This would particularly be an issue for people with dexterity issues, such as people with arthritis. It was also noted that for people with mobility issues, returning devices by post may also be an issue. Frailer people or people who have a cognitive impairment may have difficulty using the devices that need more user input, so may need support to set them up at home. The committee concluded that home-testing devices offer several potential benefits for people with suspected OSAHS. But it also noted that it is important to consider whether the devices are suitable for people who have dexterity issues such as arthritis, and if they have support at home to help with attaching and using the devices. A stakeholder highlighted that the devices may work differently based on physiological differences such as chest size. Stakeholders also highlighted that home-testing devices should be used only as part of an agreed sleep care pathway.
Clinical effectiveness
Using accuracy estimates from hospital-based studies to estimate diagnostic accuracy for home testing
3.2
Most of the studies that produced diagnostic accuracy data were done in a hospital setting. Only 2 studies measured test accuracy in the home setting: Devani (2021) for AcuPebble SA100 and Kelly (2022) for Sunrise. The EAG stated that home-based studies are more relevant to the decision problem than hospital-based studies. The clinical experts said that it was routine for the tests, even those intended for use at home, to be validated in a hospital setting, and they would not expect a large difference in accuracy from different settings. The committee noted that testing at home would also affect the reference standard. But, if this was done alternatively in a hospital setting, it would need to be done on a different night, which would make comparing home-based testing with hospital-based testing difficult. Overall, the committee concluded that the diagnostic accuracy estimates from studies done in a hospital setting were acceptable for estimating how well the devices would work in a home setting.
Accuracy evidence for the Sunrise device
3.3
Two test accuracy studies were originally included in the EAG's report for the Sunrise device (Pépin 2020 and Kelly 2022). The EAG initially judged both studies to be at high risk of bias for interpreting the index test, because they reported accuracy data using test cut-off values that were not predefined. This meant that the studies would have overestimated diagnostic accuracy for Sunrise. At the first committee meeting, the committee agreed that this was a substantial cause for concern, and that accuracy estimates should be generated from a different data set to that used to set test cut-off values. So, it concluded that there was considerable uncertainty about the accuracy of the Sunrise device to identify and assess severity of OSAHS. In the draft guidance released after the first committee meeting, the committee did not recommend using the Sunrise device other than in research. During consultation on the first draft of the guidance, the company provided accuracy estimates from the Kelly (2022) dataset using the cut-off values that had been established in the previous Pépin (2020) study (7.63 and 12.65 events per hour). The EAG also agreed with the company's assertion made at consultation that a further study, Martinot (2022) was not a further report of the Pépin (2020) data, as stated initially. But, limited details were available for the study at the time of the second committee meeting. The committee considered the additional evidence that the company provided during consultation. It concluded that the accuracy estimates were acceptable for decision making, and that it was appropriate to consider cost-effectiveness estimates generated using this data (see section 3.17). The committee decided to recommend use of the Sunrise device.
3.4
During the consultation on the second draft guidance, the company provided further details on the Martinot (2022) study. The EAG updated the narrative review and critical appraisal for this study and judged it to be at low risk of bias. The Martinot (2022) study provided diagnostic accuracy estimates using the Sunrise device at the conventional apnoea-hypopnoea index (AHI) cut-off values of 5, 15 and 30 events per hour. The company also provided diagnostic accuracy estimates using the cut-off values established by Pépin (2020) and applied to the study sample from Martinot (2022). The company confirmed that the Sunrise device reported both AHI (using the conventional cut-off values of 5, 15 and 30 events per hour) and obstructive respiratory disturbance index (ORDI; using the cut-off values of 7.63 and 12.65 events per hour that were established in the Pépin study) when used in clinical practice. The committee and clinical experts agreed that the data from the Martinot (2022) study and the additional analysis done by the company provided additional reassurance about the diagnostic accuracy of the Sunrise device.
Accuracy evidence for the Brizzy device
3.5
One test accuracy study was included in the EAG's report for the Brizzy device (Martinot 2017). In its original report the EAG initially judged this study to be at low risk of bias, and Brizzy was recommended for use in the draft guidance produced after the first committee meeting. But during the consultation on the first draft guidance, a consultee stated that accuracy estimates in Martinot (2017) were provided using cut-off values established in the same study, which would overestimate diagnostic accuracy. At the second committee meeting, the EAG issued a correction for its assessment of the Martinot (2017) study. The correction changed its judgement on risk of bias and applicability concern in the index test domain from low to high, assessed using the QUADAS‑2 tool. Based on this, at the second committee meeting, the committee reconsidered its view on the evidence for Brizzy. It recalled its concern about this issue with a different home-testing device from the first committee meeting (see section 3.3). It recalled that this had been considered a strong enough reason not to recommend the device at that time. At the second committee meeting, the committee concluded that there was a substantial cause for concern about the available data that showed test accuracy for the Brizzy device. It reiterated its opinion that accuracy estimates should be generated from a different data set to that used to set test cut-off values. So, it concluded that there was considerable uncertainty about the accuracy of the Brizzy device to identify and assess the severity of OSAHS. The committee changed its original recommendation for using the Brizzy device to a recommendation in the second draft guidance for more research before it can be used in the NHS.
3.6
During the consultation on the second draft guidance, the company provided details of a study in progress, the Sleep Respiratory Disorders in Patients With Moderate to Severe Persistent Rhinitis (JawRhin1) study. This study is evaluating the diagnostic performance of the Brizzy device in detecting sleep disordered breathing in people with moderate to severe persistent rhinitis. The EAG assessed the relevance of the study and concluded that the study population does not meet the inclusion criteria for the systematic review. The committee considered the additional information provided by the company but decided it did not address previous concerns. So, the committee maintained that the Brizzy device should only be used in research.
Using test accuracy data from previous and similar versions of NightOwl and the WatchPAT devices
3.7
The company explained that the disposable NightOwl device that will be available in the UK has the same sensors and software as the CE-marked NightOwl mini. The company stated that the only difference between the devices is that the NightOwl mini's battery can be recharged and the NightOwl's battery cannot. It is awaiting declaration of CE conformity based on the name change. Only 1 study used the disposable version (Lyne 2023) and 2 further studies used the reusable version of the device (Massie 2018; Van Pee 2022). The committee concluded that it was appropriate to use accuracy data from the reusable NightOwl device for the disposable NightOwl device. The NightOwl device was not included in the recommendations section because it was awaiting appropriate CE mark approval.
3.8
There was no accuracy data identified for the WatchPAT 300 and WatchPAT ONE devices. WatchPAT ONE is a single-use version of WatchPAT 300 and includes the same sensory attachments and software. The EAG included data in its report from studies that used the WatchPAT 200U, which was the earlier version of these devices. The company stated that the devices use identical algorithms and produce identical signals. This similarity allowed the company to gain US Food and Drug Administration and CE approval for the new devices, based on technological continuity. The committee concluded that it was appropriate to use diagnostic accuracy data from the WatchPAT 200U device for the WatchPAT 300 and WatchPAT ONE devices.
Evidence in children and young people under 16 years
3.9
The evidence on the test accuracy for the home-testing devices in children and young people under 16 years consisted of 1 published study for the Sunrise device, and preliminary accuracy results from 1 ongoing study for the AcuPebble SA100 device. There are other ongoing studies. The clinical experts stated that it was not appropriate to use data on device accuracy in adults for children and young people. The clinical experts also highlighted that there are potential issues with using a kit for children that is designed to be an appropriate size for adults. They also raised concerns about the possible choking risk of smaller pieces of equipment when the kits are used for children. The clinical experts highlighted that there was uncertainty, particularly for younger children, about how well children may tolerate the home-testing devices. But, if the devices are smaller and less intrusive than current tests, they could be more tolerable, especially for children with sensory or neurodevelopmental conditions. So, there is considerable potential for using the home-testing devices in this group. The ongoing or completed studies were mostly done in hospital, rather than at home. A clinical expert commented that it was feasible to do home-based studies for children and young people. The committee recalled its earlier conclusion that hospital-based studies were appropriate for estimating home-based device accuracy (see section 3.2). The committee concluded that the data from adults was not generalisable to children and young people under 16 years and that further evidence on accuracy is needed in this group.
Effect of home-testing devices on healthcare resource use
3.10
The clinical experts highlighted that NHS sleep services are currently under considerable pressure. They noted that people currently wait 6 weeks or more for a sleep study and that the waiting lists are still growing. Both overdiagnosis and delays in getting a diagnosis can increase anxiety and impact people's earning capacity, ability to drive and other aspects of their lives. The EAG said that estimating a device's effect on the time to diagnosis is not as simple as extrapolating from technical capabilities of the device and the regulatory evidence. This is because there will be variation in how new devices are used by different NHS sleep services when they are adopted in NHS practice. For example, differences in how much reliance clinical services are prepared to place on automated diagnosis. The clinical experts noted that although some devices provide automated analysis, healthcare professionals are likely to still want to check the raw data, particularly if the person being assessed has comorbidities (see section 3.11). The clinical experts also commented that the time needed to review home respiratory polygraphy outputs reported in some studies (up to 2 hours) was far longer than their experience. They stated that the staff time needed to review the outputs was still likely to be a limiting factor. There is some data suggesting that home-testing devices could reduce healthcare resource use such as staff time. The EAG raised concerns about whether the estimates of the time needed to use the devices included all the relevant activities, such as cleaning, device preparation and training. So, the extent to which any time benefits would be seen in practice is uncertain. But, if home-testing devices reduced reporting and staff time this could be a big benefit to help with waiting times. The committee concluded that the devices could plausibly offer some efficiencies for the time taken to review outputs compared with current practice. But based on the current evidence the extent of this is uncertain.
Outputs provided by home-testing devices compared with oximetry and respiratory polygraphy
3.11
The clinical experts noted that in current practice for adults, home respiratory polygraphy is the preferred test if OSAHS is suspected, but if access to this test is limited, home oximetry can be considered. NICE's guideline on OSAHS states that home respiratory polygraphy should be offered to people with suspected OSAHS. If access to home respiratory polygraphy is limited, home oximetry should be considered for people with suspected OSAHS. The NICE guideline also recommends to not use oximetry alone to diagnose OSAHS in people with suspected COPD–OSAHS overlap syndrome. Oximetry alone may be inaccurate for differentiating between OSAHS and other causes of hypoxaemia in people with heart failure or chronic lung conditions. Both tests are used in practice. The newer home-testing devices may differ in the outputs they produce, compared with oximetry and respiratory polygraphy. The clinical experts commented that this could mean that some of the outputs produced by the current tests may not be available if the home-testing devices are used. This could be an issue, particularly if they are used for people with comorbidities such as COPD, for which healthcare professionals review oximetry data to look at oxygenation patterns during sleep. But the clinical experts also noted that some of the newer home-testing devices measure similar features or have the option to use a third-party oximeter alongside the home-testing device. It was suggested that this may indicate that the companies see the value of oximetry alongside the outputs produced by home-testing devices. So, it is important to consider the outputs that are needed, and to choose a device that can provide these. Comments received on the draft guidance during consultation stated that healthcare professionals may prefer to use an oximeter alongside the newer home-testing devices. So, whether a device can be used with a third-party oximeter may be an important consideration for NHS sleep services when they are considering which of the newer devices to purchase. Some home-testing devices may also have contraindications, which could mean they cannot be used for everyone who would currently have testing with home oximetry or home respiratory polygraphy.
Diagnostic accuracy in people with brown or black skin
3.12
The committee considered how well the devices work for people with brown or black skin. It noted the publication of a recent Department of Health and Social Care independent review on equity in medical devices, which reviewed pulse oximeter performance for people with darker skin tones (although not when used to detect OSAHS). The committee noted there would be a large advantage to using the home-testing devices if they improved detection of OSAHS for people with brown or black skin compared with currently used tests. But it recalled that there was no subgroup data in the identified studies based on skin colour or ethnicity. The independent review also considered more broadly devices that send light waves of various frequencies through a person's skin to measure underlying physiology. It highlighted that such technologies could have varied diagnostic accuracy by skin colour, but that it is unknown if this occurs to such an extent that they disadvantage or harm the health of people with brown or black skin. The home-testing devices that were assessed differ in what they measure to detect OSAHS, but NightOwl and WatchPAT devices use light-based technology. AcuPebble SA100 and Brizzy also have the option to use a third-party pulse oximeter alongside the home-testing device (see section 3.11).
3.13
The clinical experts explained that when using measures of blood oxygen levels to diagnose OSAHS, they look at relative changes in blood oxygen levels from the person's baseline. So, they thought that any effect on oxygen level assessment that is related to how the devices work with brown or black skin is unlikely to have an impact on accurately diagnosing OSAHS. The clinical experts also highlighted that a diagnosis of OSAHS was not only based on device outputs, but also other factors such as the person's symptoms and the impact of sleepiness on their life. Overall, the committee was satisfied that the home-testing devices that use light-based technologies for assessment are appropriate to use for people with brown or black skin. But, it agreed it would be beneficial to have data showing how accurate the devices are for people with brown or black skin, to understand if any of the devices should be recommended over others (see section 1.6).
Other considerations
Place of attachment
3.14
The home-testing devices, home oximetry and respiratory polygraphy systems differ in terms of where their sensors attach to the body. Some sensors are attached to the finger (NightOwl, WatchPAT 300 and WatchPAT ONE), while others attach to the neck (AcuPebble SA100), chin (Sunrise) or chin and forehead (Brizzy). Some people have physical features such as skin conditions or scars that may affect how well the devices attach to the skin. If there is hair in the area that the device attaches to, it needs to be removed for the device to attach properly. The committee noted that the positioning of some devices may make them unsuitable for people who have a beard that they do not want to shave for a sleep study, including if it has been grown for religious or cultural reasons. So, an appropriate device should be chosen that attaches to a suitable place for the person.
Access to the internet or a smartphone
3.15
The devices vary in whether they need a smartphone or internet connection to set up, carry out and transmit data from a sleep study. Some companies explained that their devices do not need an internet connection or a smartphone to do the actual sleep study, even if this is needed for some functions such as setting it up or downloading the data. Some companies can provide a fully set-up compatible smartphone if needed. The committee noted that some people may have limited access to a smartphone or internet connection, including having limited internet data or living in areas with poor network signal. Also, some people will be less comfortable using smartphones. This, as well as how the device attaches, should be considered when deciding if a home-testing device is appropriate to use in place of home oximetry or home respiratory polygraphy.
Sustainability considerations
3.16
Some of the devices are reusable (AcuPebble SA100, Brizzy and WatchPAT 300) and others are single-use only (NightOwl, Sunrise and WatchPAT ONE). Any reduction in travel to healthcare centres to collect and return equipment may have benefits in terms of reducing carbon dioxide emissions. The committee discussed that disposable devices would have an environmental cost. But because reusable devices need to be returned, this may cause delays to the devices being available again if they are not returned promptly or are lost. The committee noted that sustainability is an important and growing issue for NHS services. It noted that the descriptions of the devices and the implications described in this guidance could help in considerations about which devices to adopt.
Cost effectiveness
Home-testing devices for diagnosing OSAHS in people 16 years and over
3.17
When comparing the home-testing devices with home oximetry, the committee noted that they were consistently cost effective in the analyses. In its base case, the EAG used accuracy data for home respiratory polygraphy from Xu (2017). But the committee preferred to use the pooled estimates from the NICE guideline on OSAHS, which included data from Xu (2017). This was because the pooled estimates had been discussed at length during development of the guideline and were considered suitable for decision making. The EAG confirmed that it had looked for more recent suitable studies but had not identified any. Compared with home respiratory polygraphy, the home-testing devices were cost effective when modelled using either the pooled estimates or the data from Xu (2017) alone. The committee recalled its concerns with the accuracy data that was available for the Brizzy device (see sections 3.5 and 3.6). So, it concluded that cost-effectiveness estimates generated with the accuracy data from the Brizzy device were not suitable for decision making. For the Sunrise device, more information was provided by the company for the second and third committee meetings. Based on this, the committee concluded that the cost-effectiveness estimates for Sunrise were suitable for decision making. These used data from the Kelly (2022) study and Martinot (2022) study, which were produced using cut-off values set in a previous study (see sections 3.3 and 3.4). The committee concluded that the AcuPebble SA100, NightOwl, Sunrise, WatchPAT 300 and WatchPAT ONE devices were cost-effective alternatives to home oximetry and home respiratory polygraphy. The NightOwl device was not included in the recommendations because it was awaiting appropriate CE mark approval. The committee also noted that the EAG had advised against comparing the cost-effectiveness estimates of the different devices with each other, based on the available data.
3.18
The committee noted that there may be some cases in which it may be more appropriate to use home oximetry or home respiratory polygraphy rather than the newer home-testing devices. For example, the newer home-testing devices may be unsuitable if there are concerns about:
-
whether they provide suitable outputs for detecting OSAHS in people with comorbidities (see section 3.11)
-
how well the device may attach because of facial hair or physical features (see section 3.14)
-
internet or smartphone availability (see section 3.15).
So, the committee noted that even if NHS sleep services do adopt a home-testing device, they will need to retain availability of at least some home oximetry or respiratory polygraphy test equipment for such cases.
3.19
The committee recalled that data on test accuracy for the home-testing devices in people under 16 years is currently limited. Also, it was not appropriate to use data from adults for this group (see section 3.9). So, it was not possible to assess cost effectiveness of the home-testing devices in people under 16 years. Further data on how accurately all the home-testing devices diagnose and assess severity of OSAHS for people under 16 years is needed (see sections 1.5 to 1.7). The committee also decided that it would be beneficial to have data on how often the devices fail to provide useable results from a sleep study in this group, and the reasons for this.