Research in the ISTC-PC is organized around three themes and showcased in two capstone applications described below.

Research Themes

Enabling the next generation of pervasive computing systems that are always aware, and continuously learning and adapting will require significant advances in sensing, interaction and learning. The ISTC-PC is tackling these challenges by focusing its research on the three areas highlighted below.


Pervasive computing systems need to be continuously aware of the environment, the nearby people and the activities they are engaged in, even when there is no explicit user interaction. Thus, saving power whenever possible is crucial and when the sensing components communicate, they must use energy-efficient protocols. We are investigating perpetual power techniques that harvest energy from ambient sources and allow simple sensing and computing systems to run indefinitely. We are also investigating new sensing modalities, both for mobile devices and embedding in the environment, that can be used to infer the state of people and their surroundings. Since our launch, the ISTC-PC has published award-winning work in Infrastructure Mediated Sensing, Backscatter Communication, and RF Power Harvesting.


Next generation pervasive systems require fine-grained activity, object, and social context recognition. We believe this will be achieved using dense, heterogeneous sensors deployed in mobile environments and smart spaces, including depth video and audio as well as classic pervasive computing sensors like GPS, accelerometers, and wireless signals (802.11, cellular and RFID). Much of our ISTC is focused on developing new algorithms to accurately and robustly extract complex context and activity information from sensor data. To be maximally useful, context understanding must run in real-time. Thus the ISTC-PC is also investigating how to partition algorithms between mobile devices and the cloud for maximal efficiency gains. The initial two year effort by the ISTC-PC has developed new state-of-the-art approaches for recognizing objects and activities from vision data.


Successful pervasive computing systems must be able to interactively
learn the environments, objects, schedules and preferences of their users. They must know how to interact effectively with the user, gathering appropriate information and using it to provide useful feedback when needed. For these reasons, the ISTC-PC has
addressed a number of challenges pervasive assistance including sleep monitors, lung function tests, stress recognition and real-time lab task assistance.

Capstone Applications


The Smart Wet Lab Assistant

One of the largest challenges facing science is the documentation and replication of experiments. The goal of the Smart Wet Lab project is to apply sensing, awareness and assistance to the task of performing experiments in a biology lab. Our aim is to produce a system that is capable of monitoring a scientist performing an experiment, recognize their actions and the experiment being performed, and even infer errors and suggest corrections. We see this as a way to auto-document experiments, teach new students and double-check the work of experts.
Key research challenges: The Smart Wet Lab Assistant provides unique research challenges for the understanding and assistance research themes. For it to be effective, we need to accurately recognize both lab equipment (including small, disposable items such as pipette tips and Eppendorf tubes) and lab staff interaction
with the equipment. To recognize experiments and provide meaningful assistance, we need an activity representation simple enough that we can infer and reason about it in our system, but powerful enough to allow the nuances of wet lab work to be represented.

Personal Health Detective (PHD)

The goal of the PHD project is to develop integrated and novel sensing and algorithmic solutions that assist in human-centered automation of journaling lifestyle habits, such as sleep
and nutrition. Continuous, non-invasive pervasive sensing allows for capture of entire day health behavior and supports discovery, prediction, and intervention. Multiple data streams increase the level of context and assist with activity inference and correlation
detection. Our aim is to create a system that captures and smartly integrates multiple data streams, automatically detects opportunities for change, and provides the collected data to users in a meaningful way to promote positive health outcomes.