Using Computer Vision to Prevent Falls in Care Homes and Hospitals

In the United States, approximately 36 million senior adults experience falls every year. These falls result in about three million emergency room visits and 32,000 deaths. In Western Europe, approximately 8.4 million individuals aged 70 or older suffered a fall-related injury in 2017.

Falling has a huge impact on the quality of life of senior adults. Severe falls can cause mobility reduction, mental confusion, and – in extreme cases – death. For senior adults residing in care homes, the likelihood of falling is three times greater than that of their peers living elsewhere.

Countries that are home to aging populations are at risk of an epidemic of elderly falling. Such an epidemic will not only lead to an immense amount of individual suffering, but it will also consume a considerable amount of healthcare resources.

However, if we knew more about the conditions that led to falls, perhaps we could prevent them. That’s the thinking behind Teton AI, a computer vision company designing fall-prevention tools for healthcare workers in hospitals and care homes throughout Denmark. 

We recently got together with Teton’s Chief Technology Officer Esben Thorius to learn more about the role of computer vision in fall prevention and how Encord’s training data tooling has provided Teton with the flexibility needed to label and manage training data in the ways that result in the best model performance possible. 

‍

‍

The following interview has been edited for clarity and length.

‍

Tell me a little bit about how Teton uses computer vision to help prevent falls.

Esben Thorius: We are building tools focused on capturing and conveying relevant and timely information to healthcare staff when they need it. At the moment, we are deployed in one hospital across three wards and in one care home.  

Our main focus is on understanding patient behavior so that we can prevent adverse events even when care staff is not in the room. This concept is rooted in two core pillars: context and continuity. As patient care is currently practiced at night and in the evenings, medical staff only check on patients every couple of hours, so they don’t really know what’s going on during the time between. A patient can easily have fallen or injured themself in the meantime. Teton aims to be the constant observer that can bring continuous and contextual information to the staff while they are doing other work.

With our technology, staff gain insight into what the patient is doing. Our monitoring system provides information about the patient's behavior. Using deep learning, such as object detection and pose estimation, our models identify movements, and our system alerts care staff of actions that put the patient at risk of falling. For instance, it lets the staff know if a patient has moved their legs off the side of the bed. The information about that position lets staff know that they need to react and go help the patient out of bed to prevent a fall. Because the staff knows that the other patients are lying calm in their beds, they can focus their full attention on helping the one patient currently in need of assistance.

‍

How does having access to this information help prevent falls?

Thorius:  At the hospital wards where we’re working, risk prevention is about detecting the pre-stages of a fall. The patients are neurological, geriatric patients, so the staff has time to react to these pre-stage poses because the patients move slowly when getting out of beds or chairs.

We categorise these pre-fall stages into general movement which is when a patient has started to move a lot in the bed, sitting up, moving legs out of the bed, sitting on the edge of the bed, and starting to stand up. Care staff can determine at what stage they want to receive a notification for each individual patient.

For instance, because the staff knows the patients’ conditions and histories, they know which patients have the highest risk of falling. The nurse can set an alert in Teton’s system so that if one of those patients begins to sit up in the bed, they receive a notification saying, “A high-risk patient is sitting up.” 

Memory care homes are a newer market for us, and the use case is a bit different there. We’re still focused on how to prevent falls, but we are taking different factors into account beyond the pre-fall stages. For instance, maybe the patient fell because their walking aid wasn’t placed near enough to their bed. We’re using AI to figure out why these falls are happening in the first place in addition to preventing them. 

We are currently implementing methods that allow staff to conduct fully remote and privacy-sensitive check-ups on residents. Such check-ups would provide staff with only the necessary information they need to ensure the resident is safe and remove the hassle and risk of waking residents during the night.

‍

Does the technology have any other use cases?

Thorius: I think some of its greatest potentials actually rests in its ability to automate documentation for healthcare workers. At the moment, nurses and care workers spend 30 percent of their time on documentation. Understanding long and complex sequences would unlock the ability to precisely document the entire patient journey. Documenting this information automatically would save the staff a lot of time that could be spent on patient care rather than administrative tasks. 

‍

‍

What sort of hardware does Teton require?

Thorius: At the moment, we have one camera in the room, infrared lights for nighttime visibility, and a small computer. If there’s something a nurse needs to be notified about, the computer sends the data out in a digitised version, and the nurse receives an alert. 

Unlike CCTV footage, our product doesn’t require someone to monitor it constantly, and it doesn’t save video data. Once the model is running, it’s a fully enclosed system. We don’t need to save the data, and we’re not actually collecting data because the only thing we are sending out is a fully anonymised and digitised version of the events in the room.

‍

What are the greatest barriers you encounter when trying to implement this technology?

‍Thorius: Obtaining the proper consent for data and privacy laws can be challenging. 

We need consent from all the patients and care staff. There’s often an initial fear among care staff that we are filming them. Sometimes, they worry our product is a surveillance tool designed to watch them perform their jobs, so we have to communicate our purpose and technology clearly while also obtaining the data in a compliant and legal manner. 

Also, as with all deep learning, we need a lot of data to train the model to perform well, and we need to annotate that data. Figuring out how we should annotate different features in the data based on the outcomes we want to achieve has been a challenge.

At first, we tried using normal bounding box detection, overlapping, and heuristic algorithms. We realized that this wasn’t going to solve our problems, so we moved away from the bounding box idea. Our system worked a lot better if we let the network directly predict different classes or states of patients, for instance, by asking the model: Is this a person? Is it medical staff standing on the floor, or is it a patient in bed? Is it a patient in bed with their legs over the edge?

The flexibility of the Encord platform helped tremendously because it was easy to change to this type of annotation strategy. As we’ve been developing this technology, we’ve had to be adaptive. The Encord platform is capable of being flexible and merging with our progression along the way as we’ve made changes and pivoted.

‍

Tell me a little more about how Teton uses Encord.

Thorius: We started using Encord because they had begun building a pose estimation labeling tool which was an interesting option for us.

We mostly use the flexibility of creating and quickly setting up a project in the platform. There’s a nice feature where you can use the same data for different projects and then quickly set up a new way of labeling things in the data. 

Using Encord’s SDK, we’ve built some internal pre-labeling tools to take that same labeled data and train a much larger model on a more specific task than what is in production. We then use the SDK to upload those labels and data directly, and we have some annotators who can quickly review and correct the labels. After that, we can easily download the labeled data through the SDK and continue to train our models with it. 

We really value that loop with the SDK and having the flexibility to set up new projects and have unique labeling tools that don’t just limit us to bounding boxes.

In the wake of the Covid-19 pandemic, as hospitals and care homes struggle with medical staff shortages, Teton AI is leveraging cutting-edge technology to provide medical staff with timely information, remove unnecessary pressures from their daily routine, and increase job satisfaction. Encord is proud to work with Teton AI as they support medical staff in providing patients with the best possible care.