Ubiquitous Computing – Shwetak Patel
When Shwetak Patel was a student, he wanted to build something that would impact a million people – a lofty goal, but one that is hard to reach for a computer scientist at the start of his career. But Patel, recipient of the 2018 ACM Prize in Computing, does not think of himself as a computer scientist, he likes to find problems and develop the tools to solve them.
Patel is one of the pioneers of a field called ubiquitous computing, which seeks to embed computational capability into everyday objects to enable them to perform tasks and communicate with each other.
The applications for this field are virtually endless, but a key application for this, and one Patel is focusing on, is healthcare.
Shwetak Patel (Image Credits: Peter Badge / HLFF)
Computers for healthcare
Imagine a device that estimates your fitness level. It does not really seem so far-fetched now, with smartwatches and all, but imagine a device that could estimate your fitness level without you actually doing fitness. Researchers from Cambridge gathered data from thousands of participants using wearable sensors and developed a model that can predict VO2 max (maximal oxygen consumption, a key fitness indicator) from this data. This type of data-based wizardry is enabled by vast amounts of data that come from sensors and complex algorithms (commonly based on machine learning) trained on this type of data
Thankfully, there is a lot of data out there.
Every day, many of us carry around an array of sensors. Our smartphones have accelerometers, magnetometers, gyroscopes – and perhaps most importantly for this, cameras.
Cameras have become one of the defining features of our smartphones. They are big and powerful, powerful enough to act as medical sensors. At the 9th Heidelberg Laureate Forum (HLF) in 2022, Patel commented on this:
“The beauty of the phone is that it’s just so powerful when we’ve got 7 billion smartphones in the world, and you can really have an interesting impact on healthcare,” Patel told journalists at the HLF. “Actually, three years ago at the HLF, I gave a talk on all the different ways that you can use the phone for health screening, you can use the microphone for asthma, you can use the microphone for talk detection.”.
Patel referenced a recent study that involved researchers in his lab that developed a way to trace blood oxygen concentration using nothing more than a smartphone camera and a smart algorithm:
“So one of my students, a graduate student at the University of Washington, recently published a paper on using the camera on the back of a phone and the light flash to basically figure out how much blood oxygen concentration is in your finger.”
In a clinic, doctors monitor oxygen saturation using specialized equipment like pulse oximeters that go on your fingertip or ear. But what if we could do this in a non-clinical setting, like at home, without specialized equipment?
The idea is straightforward: You put the fingertip on the camera and flash it. Then, a deep learning algorithm deciphers the blood oxygen from the cameras by measuring how much light from the flash the blood absorbs in each of the red, green, and blue channels. The research team tested it by artificially lowering the participants’ oxygen saturation and the smartphone correctly predicted the low blood oxygen levels 80% of the time.
Image credits: Dennis Wise/University of Washington, press release.
This line of research is very useful with a disease like COVID-19, where a low oxygen level was a key predictor of risk. But the technology is still in its infancy, and it’s just now making the first steps toward the real world.
“So a lot of previous work has shown that when [oxygen saturation] goes down to 90%, it works.” But 90% is not good enough, says Patel: “If you go in an airplane, you get down to 90%. So when he gets an 80% and 70%, that’s when it’s really clinically relevant. Being able to detect something that low is something that we were able to prove with a smartphone.”
Having medical-grade reliability by using nothing but a commercial sensor (and a very common one at that) can be immensely impactful – not just for pandemic-causing respiratory disease, but for monitoring the health of millions of people without accessible healthcare.
This is not the only application, and Patel’s group is not the only one investigating this. In late 2022, another group of researchers found that from smartphone data alone, an analysis of mortality risk could be performed. Meanwhile, Google researchers believe a smartphone’s microphone and camera can offer clues about health problems like diabetes or heart disease.
However, it is not just about smartphones, either.
Cheap, low-power, eco-friendly sensors
Smartphones are great for data, and they are convenient because many people have them. But ubiquitous computing is not only about using sensors and computation that is already there — it is about using things like smart bandages or microphones as sensors, and developing the algorithms that work the ‘magic’ behind the process. For Patel and his group, one other application is detecting tuberculosis.
A tuberculosis cough is different from a ‘regular’ cough, and from a COVID-19 cough. In fact, based on Patel’s work, you can predict if someone is suffering from one of these diseases by analyzing the soundwave from a cough (combined with the heart and respiratory rate). Therefore, all you need is the algorithm and a microphone – even a cheap microphone will do, Patel says. Oftentimes, the sensor does not have to be anything special, it can be a low-cost sensor.
Sound analysis showing the frequency spectrum of a cough. Image credits: Shwetak Patel / HLF presentation.
This is particularly useful for environmental applications, where cost is an important concern. For instance, recent research shows that between 20% to 72% of water usage on agricultural fields can be reduced by placing sensors strategically on an agricultural field and measuring soil humidity, thus only watering where necessary. Something similar can be done to reduce pesticide usage as well as to detect pests and other problems.
This all fits in very well with the Internet of Things (IoT). In fact, several young researchers who attended the 9th HLF in 2022 are indeed working on IoT applied to agriculture or health. The line between ubiquitous computing and IoT is fuzzy; the two overlap significantly, and they share similar challenges.
Communication is one of them. Getting a sensor to communicate from below the soil, or from another site of interest is no easy feat.
“Depending on where you deploy these, it can be hard to communicate back,” says Patel. “So if you’re deploying these in the middle of a forest, where you’re trying to do environmental sensing, for wildfire prediction, for example … it’s not going to talk back to a server somewhere you have many different hops. Do you ever get to these sensors? Do you put in a cell phone tower? Do you know store it [the data] and go find it?”
Another problem is powering these sensors. If you want a neat little sensor picking up information, it has to get energy from somewhere. The common approach is using batteries, but batteries run out and are not necessarily eco-friendly. Because of this, Patel is looking at other options.
“If you have a battery, and if every single one of these devices has one, do you have to replace all these batteries? If that is not practical, then there’s a sustainability argument. It’s like if I have hundreds of sensors around, how do I dispose of the batteries? So one of the big challenges is, how do you look at alternative power sources use power harvesting?”
“We built these patches that look like Band-Aids, that don’t have a battery: it actually uses what we’re measuring – your heartbeat and your respiration. So, that’s kind of a big grand challenge right now… how do we come up with clever ways to power all these ubiquitous sensors? Because that’s a big issue.”
Image credits: Jan Kopřiva / CC BY 3.0.
Patel calls these the grand challenges of ubiquitous computing. You can embed sensors into all sorts of environments, including the human body, but you need to power them, get them to communicate and – ideally – dispose of them in an eco-friendly way.
There is, however, another challenge that is perhaps even greater – although it is not a technical one.
Making it ethical
Diagnosing health conditions from things like smartphone data sure sounds appealing, but having people access your most private data, even if they happen to be well-intentioned researchers, somewhat less so. We saw this when the pandemic lockdowns started: mobility trends from the smartphone giants showed just how society stopped moving and going to places like parks or supermarkets. It was also a bit scary to see just how much information could be derived from this.
Patel thinks there is a way out of this. The data can be anonymized, obfuscated, and masked under a model:
“There’s no meaning to it. It’s obfuscated, it’s meaningless from a human-readable standpoint, but it can still help everybody else. So there are techniques where you can protect one’s privacy, but you can still benefit scientific progress.”
In the end, whether data that needs to be private is actually kept private, will hinge on the public – on each and every one of us – to ensure fair and transparent standards. Of course, no one is going to detect tuberculosis from your cough or measure your agricultural fields for humidity without your approval – but with smartphone data, things are a bit different. Some settings that gather data are turned off by default, but others are not.
Ensuring this standard of fairness and transparency is not exactly easy. The technology is new and most people are not aware of how this obfuscation would work, and what data is and is not accessed. Furthermore, data anonymization strategies can be That’s why it’s important for scientists to communicate what they are doing transparently, and for science communicators to convey this accurately to the general population. Patel points out:
“I think there are some technical solutions that researchers are starting to develop. The big issue with this is how do you educate the public? This is a very difficult concept to convey to an individual. It’s like, what do you mean, you’re still benefiting science, but you’re not giving them my data? How is it possible to not give them my data, right? And so I think coming up with ways to better make that approachable from the general population is gonna be really important right now.”
For better or for worse, the field of ubiquitous computing is just starting to burgeon. The algorithm side of it, the machine learning part, is growing spectacularly. The physical part, the prototyping and construction, is also accelerating with things like 3D printing. With people like Patel on the job, the other grand challenges of communication, powering, and sustainability will probably also slowly be conquered. By integrating more intelligent sensors, we can learn more about the world around us; we can detect diseases without specialized equipment; we may even develop customized, individual diagnoses and treatments.
But we have to do it the right way.
Patel got his wish: his work has impacted over one million people; according to his calculation, it may have impacted 10 or even 100 million. However, there are billions of smartphones in the world, and we’ve merely scratched the surface of what can be achieved.
Ubiquitous computing will soon impact all our lives – if it hasn’t already.