Environmental Awareness with Wi-Fi Sensing

February 27, 2024

Blog

Let’s start with an exercise: What sensors could you use to detect if an object is moving in a room? Immediately, you would probably shout out, “camera.” But, what if I ask how could you detect if a person is moving in a room?

A camera is still a good choice, but maybe this time you consider using an infrared camera instead. Ok then, what about a room that requires privacy, like a bedroom or a bathroom? Obviously, a camera wouldn’t be the best choice in this case (I’m sure this needs no explanation). So, maybe you consider a microphone or an accelerometer, or both because you found a clever way to associate sound and vibration with a person actually being in a room or not. Still, depending on who you ask, this setup may not eliminate their concerns about privacy.

Ok, the exercise is over. Good job thinking of sensors that could be used. At this point, you’ve identified a lot of great sensors—cameras, microphones, and accelerometers—given their ability to see, hear, and feel, respectively. But, even though these are valid sensors to consider, each one comes with a cost—the dollar cost associated with buying the sensor and the computational cost that comes with using the sensor. Simply put, if your project is budget constrained or if your application is memory constrained, the cost to integrate these sensors could be (and often is) substantial. So, before you plan to incorporate one of these sensors, there is one technology that you may not be aware of that is capable of sensing.

Something that won’t affect cost nearly as much as using one of the sensors identified above. A device you’re most likely already planning to use or that is already equipped on your product to provide it a connection to the internet. I’m talking about Wi-Fi.

Allow me to explain.

Wi-Fi operates by wirelessly transmitting and receiving signals, and when these signals bounce off obstacles in a room (e.g., people, animals, objects, a fan blowing a piece of paper taped to a wall, etc.), the signal’s properties change—changes that can be used to identify motion.

Think of a room with Wi-Fi availability like a swimming pool. When no one is in the pool, or someone is perfectly still (perhaps lounging on a float), the water is undisturbed and therefore calm and flat. Similarly, when no one is in a room that has Wi-Fi availability, or they are sitting quietly in the room (perhaps lounging on their sofa), the Wi-Fi signals are undisturbed and therefore relatively calm and flat, like the pool. But, as soon as someone jumps in the pool, swims a few laps, splashes with friends, etc., the water begins to move and change shape. Likewise, as soon as there is movement in a room, the Wi-Fi signals in the room are disturbed, and their properties change. So, if you walk by a swimming pool and notice the water is moving, chances are something happened and caused the water to move. The same goes for Wi-Fi—certain changes in the signal can indicate if movement is occurring in a room.

So then, how are these changes in Wi-Fi signals captured, stored, and used? Easy. An application can read two parameters—received signal strength (RSSI) and channel state information (CSI) for multiple subcarriers—that are built into most Wi-Fi modules. A software application does this the same way it does when accessing data from any sort of IC, for example a pressure sensor, a GPS module, or a CODEC.    

Allow me to illustrate with a demo.

To illustrate this further, I positioned two ESP32 dev kits in my work-from-home environment and collected the CSI data from one of them. Figure 1. is a photograph of my environment when working from home. As shown by the colored arrows, I oriented two ESP32 dev kits (a project referenced in a later paragraph) across from each other, one acting as an access point (yellow arrow) and another acting as a station (red arrow). I then collected RSSI and CSI from the access point over a 10-second window, as shown in Figure 2.  

Figure 1: (Left) ESP32 dev kit. (Right) Work-from-home desk with ESP32 sensors. The yellow arrow is pointing at an ESP32 that is acting as an access point (AP), and the red arrow is pointing at an ESP32 that is acting as a station (STA).

Figure 2 contains three plots of RSSI and CSI data when no movement was occurring in the environment shown in Figure 1. The plot on the far left is time-series data of RSSI and one CSI subcarrier. The middle plot is a heatmap of all CSI subcarriers vs. amplitude, and the plot to the far right is a 3D representation of time, subcarrier number, and subcarrier amplitude. As you can see, each plot shows relatively small fluctuations over time. This is especially evident in the heatmap and 3D visualization—both are relatively smooth planes (like a calm swimming pool). Even though the time-series plots show what appear to be large fluctuations, their actual values are relatively small compared to those in Figure 3.

Figure 2: (Left) RSSI and one CSI subcarrier. (Middle) CSI subcarrier heatmap. (Right) CSI subcarrier, time, amplitude 3D visualization. RSSI and CSI data collected with no movement in the environment shown in Figure 1.

Figure 3 contains the same three plots of RSSI and CSI over a different 10-second period when movement was occurring in the environment shown in Figure 1. Specifically, I stood at my desk, walked away, and then came back to my desk and sat in my chair. As you can see in each plot (and indicated by the colored arrows), the standing event resulted in a significant change in RSSI and CSI, and a sitting event resulted in another significant change. Furthermore, the pattern of the change is unique with respect to the event that caused the change. In Figure 2, the events in question are standing and sitting. However, if a completely different event occurred (e.g., waving my hand, shaking my head, etc.), a new pattern would emerge.

Figure 3: (Left) RSSI and one CSI subcarrier. (Middle) CSI subcarrier heatmap. (Right) CSI subcarrier, time, amplitude 3D visualization. RSSI and CSI data collected with a standing event (green arrow) and a sitting event (blue arrow) occurring at the desk shown in Figure 1.

In a recent webinar co-hosted by SensiML and DornerWorks, I showed how to build a machine model using SensiML and the CSI data that I collected when standing and sitting. I then followed up in a separate video and explained how to port the machine model to one the ESP32 devkits and then demonstrated the model working in real-time.

Let’s consider a real-world example: a hospital patient falling in a bathroom. In this scenario, an application running on the ESP32 could monitor RSSI to determine if movement occurred. If movement is detected, a machine learning model could be employed to classify the changes in CSI as either a fall or something else. If identified as a fall, the ESP32 could send an MQTT event to a backend service that could then be relayed as a notification to a caretaker’s phone.

For some readers, this type of sensing may seem tricky, or perhaps unproven. But, given the advancements others have made (e.g., Steven Hernandez’s work at https://stevenmhernandez.github.io/ESP32-CSI-Tool/ that utilizes an ESP32, and what I used to collect RSSI and CSI in the above figures) and the introduction of IEEE 802.11bf (https://standards.ieee.org/ieee/802.11bf/10365/) in September 2020 to further explore Wi-Fi sensing, DornerWorks believes that leveraging Wi-Fi is a valid option to consider if you are building new features that rely on motion detection—whether that motion is fall detection in a bathroom, basic hand-gesture recognition in front of a device, or if your air conditioner is ON because the leaves on one of your houseplants begin to sway.

In conclusion, if you’re thinking about building a new feature that involves sensing motion, I hope you consider leveraging the data that your Wi-Fi module may already be providing. Doing so could significantly reduce cost (both in dollars and computational resources) and development time, allowing you to quickly build and test an MVP hypothesis. Or, if you’re still skeptical, hopefully you can appreciate recent advancements in Wi-Fi technology and will consider using one that provides RSSI and CSI parameters—a decision that could better position you to innovate down the road.   

How do you feel about incorporating Wi-Fi sensing into your product?

1. Nope, too risky.
2. I’d have to read more about it first.
3. Sounds promising. 
4. Sign me up!

---

Currently serving as Engineering Project Manager II for DornerWorks in Grand Rapids, Michigan, David Norwood is passionate about state-of-the art technology and how it can help change the world. David holds an MSECE from the University of Tennessee, and since 2011 has worked with ARM, DSP, and FPGA architectures as an embedded software engineer. For DornerWorks, David has helped numerous organizations plan, develop, and commercialize their embedded products.