SYNE-COV for predicting COVID-19 outcomes

Study size, design and location

A retrospective study analysed a cohort of 879 people with a confirmed COVID-19 using viral polymerase chain reaction swab tests. Anonymised patient data was obtained from the Chelsea and Westminster Hospital NHS foundation trust between 1 January and 26 May 2020.

Intervention and comparator

Three machine-learning algorithms were benchmarked to predict patient outcomes from electronic health records:

No comparator.

Predictive accuracy was measured in terms of area under curve (AUC) of the receiver operating characteristic (ROC) and precision-recall curves are provided to assess expected real-world performance relative to random classifiers. F1 scores were calculated to measure of the accuracy of an algorithm based on 2 factors: precision and recall. The higher the F1 score, the better the accuracy.

Key outcomes

Of 1,235 people in the analysis, 630 people had data available from their hospital admissions to discharges. This included 629 people (99.8%) admitted to the hospital through the emergency department and 1 person (0.2%) who was admitted directly to the intensive care unit. Considering the study inclusion criteria, a total of 129 out of 879 people (15%) were admitted for intensive care, 62 of 878 people (7%) had mechanical ventilation, and 193 of 619 people (31%) died in hospital.

Intensive care unit admission: the XGBoost machine-learning algorithm reached an AUC‑ROC of 0.84 and an F1 score of 0.52. This was followed by prediction using the random forest algorithm (AUC‑ROC=0.83; F1 score=0.49) and the logic regression algorithm (AUC‑ROC=0.72, F1 score=0.40). Patient age and measures of oxygenation status were strong indicators for intensive care unit admission, with advanced age decreasing the probability of intensive care admission.

Having mechanical ventilation: both random forest and XGBoost algorithms reached AUC-ROC of 0.87. F1 scores were 0.42 and 0.31 for XGBoost and random forest algorithm, respectively. The logistic regression reached an AUC‑ROC of 0.74 and an F1 score of 0.23. Patient age and oxygenation status were most predictive of having mechanical ventilation, with additional contributions from blood test values, such as lactate and deoxyhaemoglobin levels.

Hospital mortality: the random forest machine-learning algorithm reached an AUC‑ROC of 0.76 and an F1 score of 0.61. This was followed by prediction by the XGBoot algorithm (AUC‑ROC=0.76; F1 score=0.60) and the logic regression algorithm (AUC‑ROC=0.70, F1 score=0.56). Age was an important predictor, with older age contributing to increased risk of hospital mortality.

Strengths and limitations

The study analysed data set from 2 hospitals from 1 NHS hospital trust. The cohort included people who were admitted to hospitals in a 5‑month period. The authors noted that data used for risk predications was limited to the first few hours of a person in the emergency department, and some information such as medical history or primary care were not included for predicting patient outcomes. Several study authors are employees of the company (Sensyne Health).

Study size, design and location

A clinical validation study evaluating performance of SYNE‑COV based on a total of 2,315 people with COVID‑19 admitted to 2 hospitals in 1 NHS hospital trust between 1 January and 25 December 2020. The validation study assessed included study population, demographics of the study population and performance of the SYNE‑COV algorithm.

Intervention and comparator

SYNE‑COV, a machine-learning algorithm uses electronic medical records of patients who have tested positive for COVID‑19. This is to predict each individual's risk of 3 clinical outcomes:

No comparator.

Key outcomes

Study population

Over 10,000 people were admitted to 2 hospitals in 1 NHS hospital trust during the study period. People were included in the study cohort if they met all the inclusion criteria and did not met a single exclusion criterion. Of people included, 3 cohorts were constructed by clinical outcomes: need for intensive care admission and invasive mechanical ventilation, and in-hospital mortality. Each dataset of the cohorts was tested against data quality criteria such as the sample size and the completeness of clinical data for individuals. A total of 2,315 people met the criteria and were included in the study.

Demographics of study population

Of 2,315 people with confirmed COVID‑19 in the study, a statistical analysis was done to examine demographic characteristics of people admitted to the 2 hospitals. The analysis found that age, sex and ethnicity of people differed by hospitals. Age and ethnicity also varied significantly by each calendar month in 2020. Such variability in the study population suggested the population could be generalisable to people with COVID‑19 admitted to NHS hospitals.

SYNE-COV algorithm

The SYNE‑COV algorithm was used to generate a risk prediction for each of the 3 clinical outcomes for each patient. The performance of risk predictions is evaluated using metrics including AUC‑ROC. This shows the probability that the model assigns a higher risk to a randomly chosen positive outcome than to a randomly chosen negative outcome. The higher the AUC, the better the model is at predicting a risk of needing intensive care admission, ventilation or hospital mortality.

Predictions by SYNE‑COV are compared with those made by standard clinical risk scores including the National Early Warning Score 2 (NEWS2), Sequential Organ Failure Assessment (SOFA) score and the Acute Physiology as well as Chronic Health Evaluation II (APACHEII) score. Based on data collected during a patient's attendance at the emergency department, the accuracy of predictions were:

Strengths and limitations

This is an unpublished validation report that has not been peer reviewed. It was based on real-world data obtained from 2 hospitals in 1 NHS hospital trust during a 1‑year period. The performance of algorithms was compared with other clinical risk scores including NEWS2.