Home-testing devices for diagnosing obstructive sleep apnoea hypopnoea syndrome

Most of the studies that produced diagnostic accuracy data were done in a hospital setting. Only 2 studies measured test accuracy in the home setting: Devani (2021) for AcuPebble SA100 and Kelly (2022) for Sunrise. The EAG stated that home-based studies are more relevant to the decision problem than studies in which the device is used in a hospital-sleep laboratory. Experts said that it was routine for tests, even those intended for use at home, to be validated in a hospital setting, and they would not expect a large difference in accuracy from different settings. The committee noted that assessment at home would also affect the reference standard. But, if this was done alternatively in a hospital setting, it would need to be done on a different night, which makes comparing home-based testing with hospital-based testing difficult. Overall, the committee concluded that the diagnostic accuracy estimates from studies done in a hospital setting were acceptable for estimating how well the devices would work in a home setting.

Two test accuracy studies were included in the EAG's report for Sunrise (Pepin 2020 and Kelly 2022). But the EAG judged both studies to be at high risk of bias for interpreting the index test, because they reported accuracy data using test cut-off values that were not predefined. At the first committee meeting, the committee agreed that this was a significant issue because it meant that the studies would have overestimated diagnostic accuracy for Sunrise, although the size of this was hard to judge. The company stated that the cut-off values determined in the Kelly (2022) study were either the same as, or close to, those the company recommends for use, as established by Pepin (2020). But when asked in the first committee meeting why it had characterised the difference in cut-off values as small, and to what extent this may have affected test accuracy estimates, the company was unable to justify this comment. The committee agreed with the EAG that this was a substantial cause for concern, and that accuracy estimates should be generated from a different data set to that used to set test cut-off values. So, it considered that there was considerable uncertainty about the accuracy of the Sunrise device to identify and assess severity of OSAHS. In the draft guidance released after the first committee meeting the committee did not recommend using the Sunrise device other than in research. During consultation on the first draft of the guidance, the company provided accuracy estimates from the Kelly (2022) data set. These were produced using cut-off values that had been established in the previous Pepin study. The EAG highlighted that these were applied retrospectively, that is, after the study had been completed, but that it did consider this data informative. The committee considered that the accuracy estimates were acceptable for decision making, and that it was appropriate to consider cost-effectiveness estimates generated using this data (see section 3.14). Accuracy estimates from the Kelly (2022) data used the cut-off value for OSAHS set in the Pepin (2020) study (7.63 events per hour) rather than the optimised value set in the Kelly study (9.53 events per hour). This had a sizeable impact on accuracy estimates, especially specificity. This justified the committee's previous concern. The EAG also agreed with the company's assertion made at consultation that a further study, Martinot (2022) was not a further report of the Pepin (2020) data, as stated in the EAG's report. But the EAG commented that limited detail was available for this study, so it was not possible to do a critical appraisal.

One test accuracy study was included in the EAG's report for Brizzy (Martinot 2017). In its original report the EAG judged this study to be at low risk of bias, and Brizzy was recommended in the draft guidance produced after the first committee meeting. But during the consultation on this draft guidance, a consultee stated that accuracy estimates in Martinot (2017) were provided using cut-off values established in the same study, which would overestimate diagnostic accuracy. At the second committee meeting, the EAG issued a correction for its assessment of the Martinot (2017) study. The correction changed its judgement on risk of bias and applicability concern in the index test domain, assessed using the QUADAS‑2 tool, from low to high. At the second committee meeting, the committee reconsidered its view on the evidence for Brizzy based on this. It recalled its concern about this issue with a different home-testing device from the first committee meeting (see section 3.3) and that this had been considered a strong enough reason not to recommend the device. At the second committee meeting, the committee concluded that there was a substantial cause for concern about the available data showing test accuracy for the Brizzy device. It reiterated its opinion that accuracy estimates should be generated from a different data set to that used to set test cut-off values. So, it considered that there was considerable uncertainty about the accuracy of the Brizzy device to identify and assess the severity of OSAHS.

The company explained that the disposable NightOwl device that will be available in the UK has the same sensors and software as the CE-marked NightOwl mini. The company stated that the only difference between the devices is that the NightOwl mini's battery can be recharged and the NightOwl's battery cannot. It is awaiting declaration of CE conformity based on the name change. Only 1 study used the disposable version (Lyne 2023) and 2 further studies used the reusable version of the device (Massie 2018; Van Pee 2022). The committee concluded that it was appropriate to use accuracy data from the reusable NightOwl device for the disposable NightOwl device.

There was no accuracy data identified for WatchPAT 300 and WatchPAT ONE. WatchPAT ONE is a single-use version of WatchPAT 300 and includes the same sensory attachments and software. The EAG included data in its report from studies that used the WatchPAT 200U, which is the earlier version of these devices. The company stated that the devices use identical algorithms and produce identical signals. This similarity allowed the company to gain US Food and Drug Administration (FDA) and CE approval for the new devices, based on technological continuity. The committee concluded that it was appropriate to use diagnostic accuracy data from WatchPAT 200U for WatchPAT 300 and WatchPAT ONE.

There is some data suggesting that home-testing devices could reduce healthcare resource use, for example staff time. The EAG raised concerns about whether the estimates of the time needed to use the devices included all relevant activities, such as cleaning, device preparation and training. It also said that estimating the device's effect on the time to diagnosis is not as simple as extrapolating from technical capabilities of the devices and the regulatory evidence. This is because there will be variation in how new devices are used when they are adopted in NHS practice and how much reliance clinical services are prepared to place on automated diagnosis. The clinical experts noted that although some devices provide automate analysis, clinicians are likely to still want to check the raw data, particularly if the person being assessed has comorbidities (see section 3.9). So, the extent to which any time benefits are seen in practice is uncertain. The clinical experts also commented that the time to review home RP outputs reported in some studies (up to 2 hours) was far longer than their experience. The clinical experts noted that people currently wait 6 weeks or more for a sleep study and that the waiting lists are still growing. Both overdiagnosis and delays in getting a diagnosis will increase anxiety and impact people's earning capacity, ability to drive and other aspects of their life. They stated that the staff time needed to review the outputs was still likely to be a limiting factor. But, if home-testing devices reduced reporting and staff time this could be a big benefit to help with waiting times. They also highlighted that services are currently under considerable pressure. The committee concluded that the devices could plausibly offer some efficiencies for time taken to review outputs compared with current practice. But based on the current evidence the extent of this is uncertain.

The evidence on the test accuracy for the home-testing devices in children and young people under 16 years consists of 1 published study for Sunrise, and 1 ongoing study for AcuPebble SA100 with preliminary accuracy results available. There are further ongoing studies. The clinical experts explained that the main difference between adults and children is their body size. They also stated that it is not appropriate to use data on device accuracy in adults for children and young people. The clinical experts also highlighted potential issues with using kit for children that is of appropriate size for adults, and the possible choking risk of smaller pieces of equipment. The experts highlighted that there is uncertainty, particularly for younger children, about how well children may tolerate the home-testing devices. But, if the devices are smaller and less intrusive than current tests, they could be more tolerable, especially for children with sensory or neurodevelopmental conditions. So, there is considerable potential for using the home-testing devices in this group. Ongoing or completed studies were mostly done in hospital, rather than at home. A clinical expert commented that it was feasible to do home-based studies for children and young people. The committee recalled its earlier conclusion that hospital-based studies were appropriate to estimate home-based device accuracy (see section 3.2). The committee concluded that the data from adults was not generalisable to children and young people under 16 years and that further evidence on accuracy is needed in this group.

Clinical experts noted that in current practice for adults, home RP is the preferred test if OSAHS is suspected, but if access to this test is limited, home oximetry can be considered. NICE's guideline on OSAHS states that home RP should be provided to people with suspected OSAHS. If access to home respiratory polygraphy is limited, home oximetry should be considered for people with suspected OSAHS. The NICE guideline also recommends to not use oximetry alone to diagnose OSAHS in people with suspected COPD–OSAHS overlap syndrome. Oximetry alone may be inaccurate for differentiating between OSAHS and other causes of hypoxaemia in people with heart failure or chronic lung conditions. Both tests are used in practice. The newer home-testing devices may differ in the outputs they produce, compared with oximetry and RP. The clinical experts commented that this could mean that some of the outputs produced by the current tests may not be available if the home-testing devices are used. This could be an issue particularly if they are used for people with comorbidities such as COPD, for which experts review oximetry data to look at oxygenation patterns during sleep. But the experts also noted that some of the home-testing devices measure similar features or have the option to use a third-party oximeter alongside the home-testing device. It was suggested that this may indicate that the companies see the value of oximetry alongside the outputs produced by home-testing devices. So, it is important to consider the outputs that are needed when choosing which test to use and whether the device will provide these. Comments received on the draft guidance during consultation stated that using an oximeter alongside the newer home-testing devices may be preferred. So, having the functionality to be used with a third-party oximeter may be an important consideration for services when they are considering which of the newer devices to purchase. Some home-testing devices may also have contraindications, which could mean they cannot be used for everyone who is currently tested with home oximetry or home RP.

The committee considered how well the devices work for people with brown or black skin. It noted the publication of a recent department of health and social care independent review on equity in medical devices, which focused on pulse oximeter performance for people with darker skin tones (although not when used to detect OSAHS). The committee noted there would be a large advantage to using the home-testing devices if they improved detection of OSAHS for people with brown and black skin compared with currently used tests. But it recalled that there was no subgroup data in the identified studies based on skin colour or ethnicity. The independent review also considered more broadly devices that send light waves of various frequencies through a person's skin to measure underlying physiology. It highlighted that such technologies could have varied diagnostic accuracy by skin colour, but that it is unknown if this occurs to such an extent that they disadvantage or harm the health of people with brown or black skin. The home-testing devices that were assessed differ in what they measure to detect OSAHS, but NightOwl and WatchPAT devices use light-based technology. AcuPebble SA100 and Brizzy also have the option to use a third-party pulse oximeter alongside the home-testing device, which may provide useful additional outputs (see section 3.9).

The clinical experts explained that when using measures of blood oxygen levels to diagnose OSAHS, they look at relative changes in blood oxygen levels from the person's baseline. So, they considered that any effect on oxygen level assessment related to how the devices work with brown or black skin is unlikely to have an impact on accurately diagnosing OSAHS. The clinical experts also highlighted that a diagnosis of OSAHS was not only based on device outputs, but also considered other factors such as the person's symptoms and the impact of sleepiness on their lives. Overall, the committee was satisfied that the home-testing devices that use light-based technologies for assessment are appropriate to use for people with brown or black skin. But, it agreed it would be beneficial to have data showing how accurate the devices are for people with brown or black skin, to understand if any of the devices should be recommended over others.

The home-testing devices differ in where the sensors are attached on the body, to each other and to oximetry and RP systems. Some sensors are attached to the finger (NightOwl, WatchPAT 300 and WatchPAT ONE), while others attach to the neck (AcuPebble SA100), chin (Sunrise) or chin and forehead (Brizzy). Some people have physical features such as skin conditions or scars that may affect how well the devices attach to the skin. If there is hair in the area that the device attaches to, it needs to be removed for the device to attach properly. The committee noted that the positioning of some devices may make them unsuitable for people who have a beard that they do not want to shave for a sleep study, including if it has been grown for religious or cultural reasons. So, an appropriate device should be chosen that attaches to a suitable place for the person.

The devices vary in whether they need a smartphone or internet connection to set up, carry out and transmit data from the sleep study. Some companies explained that their devices do not need internet connection or a smartphone to do the actual sleep study, even if this is needed for some functions such as setting it up or downloading the data. Some companies can provide a fully set-up compatible smartphone if needed. The committee noted that some people have limited access to a smartphone or internet connection, including having limited internet data or living in areas with poor network signal. Also, some people will be less comfortable using smartphones. This, as well as how the device attaches, should be considered when deciding if a home-testing device is appropriate to use in place of home oximetry or home RP.

When comparing the home-testing devices with home oximetry, the committee noted that they were consistently cost effective in the analyses. In its base case, the EAG used accuracy data for home RP from Xu (2017). But the committee preferred to use the pooled estimates from the NICE guideline on OSAHS, which included Xu, because this had been discussed at length during development of the guideline and was considered suitable for decision making. The EAG confirmed that it had looked for more recent suitable studies but had not identified any. Compared with home RP, home-testing devices were cost effective when modelled using this accuracy data. The committee recalled its concerns with the accuracy data that was available for the Brizzy device (see section 3.4). So, it did not consider cost-effectiveness estimates generated with this data to be suitable for decision making. For the Sunrise device, more information was provided by the device developer for the second committee meeting. Based on this, the committee considered the cost-effectiveness estimates for Sunrise that used the data from the Kelly study, which were produced using cut-off values set in a previous study (see section 3.3), were suitable for decision making. The committee concluded that AcuPebble SA100, NightOwl, Sunrise, WatchPAT 300 and WatchPAT ONE were cost effective alternatives to home oximetry and home RP. The committee also noted that the EAG had advised against comparing the cost-effectiveness estimates of the different devices with each other, based on available data.

The committee noted that there may be some cases in which it may be more appropriate to use home oximetry or home RP rather than the newer home-testing devices. For example, the newer home-testing devices may be unsuitable if there are concerns about:

• whether they provide suitable outputs to allow detection of OSAHS in people with comorbidities (see section 3.9)

• how well the device may attach because of facial hair or physical features (see section 3.12)

• internet or smartphone availability (see section 3.13)

• the devices not providing all the required outputs (see section 3.9).
  
  So, the committee noted that even if services are adopting a home-testing device, they will need to retain availability of at least some existing testing equipment (home oximetry or RP) for such cases.

The committee recalled that data on test accuracy for the home-testing devices in people under 16 years is currently limited. Also, it was not appropriate to use data from adults for this group (see section 3.8). So, it was not possible to assess cost effectiveness of the home-testing devices in people under 16 years. Further data on how accurately all the home-testing devices diagnose and assess severity of OSAHS for people under 16 years is needed (see sections 1.5 to 1.7). The committee also considered that it would be beneficial to have data on how often the devices fail to provide useable results from a sleep study in this group, and the reasons for this.

Some of the devices are reusable (AcuPebble SA100, Brizzy and WatchPAT 300) and others are single use only (NightOwl, Sunrise and WatchPAT ONE). Any reduction in travel to healthcare centres to collect and return equipment may have benefits in terms of reducing carbon dioxide emissions. The committee discussed that disposable devices would have an environmental cost. But because reusable devices need to be returned, this may cause delays to the devices being available again if they are not returned promptly or are lost. The committee noted that sustainability is an important and growing issue for NHS services. It noted that the descriptions of the devices and the implications described in this guidance could help in considerations about which devices to adopt.