DIAGNOSTICS ASSESSMENT PROGRAMME

Most of the studies that produced diagnostic accuracy data were done in a hospital setting. Only 2 studies measured test accuracy in the home setting: Devani (2021) for AcuPebble SA100 and Kelly (2022) for Sunrise. The EAG stated that home-based studies are more relevant to the decision problem than studies in which the device is used in a hospital-sleep laboratory. Experts said that it was routine for tests, even those intended for use at home, to be validated in a hospital setting, and they would not expect a large difference in accuracy from different settings. The committee noted that assessment at home would also affect the reference standard. But, if this was done alternatively in a hospital testing, it would need to be done on a different night, which makes comparing home-based testing with hospital-based testing difficult. Overall, the committee concluded that the diagnostic accuracy data from studies done in a hospital setting were acceptable to estimate how well the devices would work in a home setting.

Two test accuracy studies were included in the EAG's report for Sunrise (Pepin 2020 and Kelly 2022). But the EAG judged both studies to be at high risk of bias for the interpretation of the index test because they reported accuracy data using test cut-off points that were not predefined. The committee agreed that this was a significant issue because it meant that the studies would have overestimated diagnostic accuracy for Sunrise, although the size of this was hard to judge. The company stated that the cut offs determined in the Kelly et al. (2022) study were either the same as, or close, to those the company recommends for use, as established by Pepin (2020). But when asked in the committee meeting why it had characterised the difference in cut off as small, and to what extent this may have affected test accuracy estimates, the company was unable to justify this comment. The committee agreed with the EAG that this was a substantial cause for concern, and that accuracy estimates should be generated from a different data set to that used to set test cut-off values. So, it considered that there was considerable uncertainty about the accuracy of the Sunrise device to identify and assess severity of OSAHS.

The company explained that the disposable NightOwl device that will be available in the UK has the same sensors and software as the CE-marked NightOwl mini. The company stated that the only difference between the devices is that the NightOwl mini's battery can be recharged and the NightOwl's battery cannot. It is awaiting declaration of CE-conformity based on the name change. Only 1 study used the disposable version (Lyne 2023) and 2 further studies used the reusable version of the device (Massie 2018; Van Pee 2022). The committee concluded that it was appropriate to use accuracy data from the reusable NightOwl device for the disposable NightOwl device.

There was no accuracy data identified for WatchPAT 300 and WatchPAT ONE. The EAG included data in its report from studies that used the WatchPAT 200U, which is the most recent version of this device. The company stated that the devices use identical algorithms and produce identical signals. This similarity allowed the company to gain US Food and Drug Administration (FDA) and CE approval for the new devices, based on technological continuity. The committee concluded that it was appropriate to use diagnostic accuracy data from WatchPAT 200U for WatchPAT 300 and WatchPAT ONE. WatchPAT ONE is a single-use version of WatchPAT 300 and includes the same sensory attachments and software.

There is some data suggesting that home-testing devices could reduce healthcare resource use, for example staff time. The EAG raised concerns about whether the estimates of the time needed to use the devices included all relevant activities, such as cleaning, device preparation and training. It also said that estimating the device's effect on the time to diagnosis is not as simple as extrapolating from technical capabilities of the devices and the regulatory evidence. This is because there will be variation in how new devices are used when they are adopted in NHS practice and how much reliance clinical services are prepared to place on automated diagnosis. The clinical experts noted that although some devices provide automated analysis, clinicians are likely to still want to check the raw data, particularly if the person being assessed has comorbidities (see section 3.8). So, the extent to which any time benefits are seen in practice Is uncertain. The clinical experts also commented that the time reported in some studies to review home RP outputs (up to 2 hours) was far longer than their experience. The clinical experts noted that people currently wait 6 weeks or more for a sleep study and that the waiting lists are still growing. They stated that the staff time needed to review the outputs was still likely to be a limiting factor. But, if home-testing devices reduced reporting and staff time this could be a big benefit to help with waiting times, highlighting that currently services are under considerable pressure. The committee concluded that the devices could plausibly offer some efficiencies for time taken to review outputs compared with current practice. But based on the current evidence the extent of this is uncertain.

The evidence on the test accuracy for the home-testing devices in children and young people under 16 years consists of 1 published study for Sunrise, and 1 ongoing study for AcuPebble SA100 with preliminary accuracy results available. There are further ongoing studies. The clinical experts explained that the main difference between adults and children is their underlying pathophysiology. They also stated that it is not appropriate to use data on device accuracy in adults for children and young people. The clinical experts also highlighted potential issues with using kit for children that is of appropriate size for adults, and the possible choking risk of smaller pieces of equipment. The experts highlighted that there is uncertainty, particularly for younger children, about how well children may tolerate the home-testing devices. But, if the devices are smaller and less intrusive than current tests, they could be more tolerable, especially for children with sensory or neurodevelopmental conditions. So, there is considerable potential for using the home-testing devices in this group. Ongoing or completed studies were mostly done in hospital, rather than at home. A clinical expert commented that it was feasible to do home-based studies for children and young people. The committee recalled its earlier conclusion that hospital-based studies were appropriate to estimate home-based device accuracy (see section 3.2). The committee concluded that the data from adults was not generalisable to children and young people under 16 years and that further evidence on accuracy is needed in this group.

Clinical experts noted that in current practice for adults, home RP is the preferred test if OSAHS is suspected, but if access to this test is limited, home oximetry can be considered. Oximetry alone may be inaccurate for differentiating between OSAHS and other causes of hypoxaemia in people with heart failure or chronic lung conditions. Both tests are used in practice. The home-testing devices may differ in the outputs they produce, compared with oximetry and RP. The clinical experts commented that this could mean that some of the outputs produced by current tests may not be available if the home-testing devices are used. This could be an issue particularly if they are used for people with comorbidities such as COPD, for which experts review oximetry data to look at oxygenation patterns during sleep. But the experts also noted that some of the home-testing devices measure similar features or have the option to use a third-party oximeter alongside the device. It was suggested that this may indicate that the companies see the value of oximetry alongside the outputs produced by home-testing devices. So, it is important to consider the outputs that are needed when choosing which test to use and whether the device will provide these. Some home-testing devices may also have contraindications which could mean they cannot be used for everyone who is currently tested with home oximetry or home RP.

The committee considered how well the devices work for people with brown or black skin. It noted the publication of a recent department of health and social care independent review on equity in medical devices, which focused on pulse oximeter performance for people with darker skin tones (although not when used to detect OSAHS). The committee noted there would be a large advantage to the home-testing devices if they improved detection of OSAHS for people with brown and black skin compared with currently used tests. But it recalled that there was no subgroup data in the identified studies based on skin colour or ethnicity. The independent review also considered more broadly devices that send light waves of various frequencies through a person's skin to make measurements of underlying physiology. It highlighted that such technologies could have varied diagnostic accuracy by skin colour but that it is unknown if this occurs to such an extent that they disadvantage or harm the health of people with brown or black skin. The home-testing devices that were assessed differ in what they measure to detect OSAHS but NightOwl and WatchPAT devices use light-based technology. AcuPebble SA100 and Brizzy also have the option to use a third-party pulse oximeter alongside the device, which as noted may provide useful additional outputs (see section 3.8).

The clinical experts explained that when using measures of blood oxygen levels to diagnose OSAHS, they look at relative changes in blood oxygen levels from the person's baseline. So, they considered that any effect on oxygen level assessment related to how the devices work with brown or black skin is unlikely to have an impact on accurately diagnosing OSAHS. The clinical experts also highlighted that a diagnosis of OSAHS was not only based on device outputs, but also considered other factors such as the person's symptoms and the impact of sleepiness on their lives. Overall, the committee was satisfied that the home-testing devices that use light-based technologies for assessment are appropriate to use for people with brown or black skin. But, it agreed it would be beneficial to have data showing how accurate the devices are for people with brown or black skin, to understand if any of the devices should be recommended over others.

The home-testing devices differ in where the sensors are attached on the body, to each other and to oximetry and RP systems. Some sensors are attached to the finger (NightOwl, WatchPAT 300 and WatchPAT ONE), while others attach to the neck (AcuPebble SA100), chin (Sunrise) or chin and forehead (Brizzy). Some people have physical features such as skin growths or scars which may affect how well the devices attach to the skin. If there is hair in the area that the device attaches to, it needs to be removed for the device to attach properly. The committee noted that the positioning of some devices may make them unsuitable for people who have a beard that they do not want to shave for a sleep study, including if it has been grown for religious or cultural reasons. So, an appropriate device should be chosen that attaches to a suitable place for the person.

The devices vary in whether they need a smartphone or internet connection to set up, carry out and transmit data from the sleep study. Some companies explained that their devices do not need internet connection or a smartphone to do the actual sleep study, even if this is needed for some functions such as setting it up or downloading the data. Some companies can provide a fully set-up compatible smartphone if needed. The committee noted that some people have limited access to a smartphone or internet connection, including having limited internet data. This, as well as how the device attaches, should be considered when deciding if a home-testing device is appropriate to use in place of home oximetry or home RP.

When comparing the home-testing devices with home oximetry, the committee noted that they were consistently cost effective in the analyses. In its base case, the EAG used accuracy data for home RP from Xu et al. (2017). But the committee preferred to use the pooled estimates from the NICE guideline on OSAHS, which included Xu et al., because this had been discussed at length during development of the guideline and was considered suitable for decision making. The EAG confirmed that it had looked for more recent suitable studies but had not identified any. Compared with home RP, home-testing devices were cost effective, when modelled using this accuracy data. The committee recalled its concerns with the accuracy data that was available for the Sunrise device (see section 3.3), so it did not consider cost-effectiveness estimates generated with this data to be suitable for decision-making. But it concluded that AcuPebble SA100, Brizzy, NightOwl, WatchPAT 300 and WatchPAT ONE were cost-effective alternatives to home oximetry and home RP. The committee further noted that the EAG had advised against comparing the cost-effectiveness estimates of the different devices with each other, based on available data.

The committee noted that there may be some cases in which it may be more appropriate to use home oximetry or home RP rather than the home-testing devices. For example, home-testing devices may be unsuitable if there are concerns about:

• how well the device may attach because of facial hair or physical features (see section 3.11)

• internet or smartphone availability (see section 3.12)

• the devices not providing all the required outputs (see section 3.8).
  
  So, the committee noted that even if services are adopting a home-testing device, they will need to retain availability of at least some existing testing equipment (home oximetry or RP) for such cases.

The committee recalled that data on test accuracy for the home-testing devices in people under 16 years is currently limited, and it was not appropriate to use data from adults for this group (see section 3.7). So, it was not possible to assess cost effectiveness of the home-testing devices in people under 16 years. Further data on how accurately all the home-testing devices diagnose and assess severity of OSAHS for people under 16 years is needed (see sections 1.5 to 1.7). The committee also considered that it would be beneficial to have data on how often the devices fail to provide useable results from a sleep study in this group, and the reasons for this.

Some of the devices are reusable (AcuPebble SA100, Brizzy and WatchPAT 300) and others are single use only (NightOwl, Sunrise and WatchPAT ONE). Any reduction in the need to travel to healthcare centres to collect and return equipment may have benefits in terms of reducing carbon dioxide emissions. The committee discussed that disposable devices would have an environmental cost. But because reusable devices need to be returned, this may cause delays to the devices being available again if they are not returned promptly or are lost. The committee noted that sustainability is an important and growing issue for NHS services, and that the descriptions of the devices and the implications described in this guidance could help in considerations about which devices to adopt.