How a Deep Dive into Alerts allows Tailoring of Response Workflows

In a previous post, Jason Sroka, our Chief Analytics Officer, discussed the importance of data when making accurate business decisions based on objective statistics rather than intuitive hunches. Here at SmartSense, applying data exactly in this way is part of our DNA. We regularly conduct studies to help us measure the performance of our digital monitoring equipment so we can adjust and improve the playbook for our products.

In this post, I want to take you through a detailed example of one of these studies. You’ll see how the SmartSense Data Team evaluated the way time of day impacts environmental behaviors that can disproportionately generate temperature excursion alerts. Using the results of this research, we were then able to tweak our system alarms to create the most accurate alert priorities for our customers.

Setting Up the Study

The primary aim of this study was measuring temperature excursions occurring over a predetermined period of time. Of course, simply becoming aware that an incident has occurred is essential for taking action. Even better, pinpointing the precise time of the excursion helps us determine causes. For example, if an incident happens overnight when employees are offsite, we can usually rule out human error.

To set up the study, the analytics team chose a set of coolers and started counting incidents. Because our database is massive, we had to limit the number of sensors for practical purposes. We chose a representative set of just over a thousand pharmacies, containing 3,117 sensors installed in coolers. All of these sensors have a warm and a cool threshold based on CDC recommendations. We also limited the time frame of the incidents to those beginning and ending during January 2021. Since the sensors are inside coolers, which are indoors, the outside temperature is not relevant to the cooler temperature.

Rather than hour-by-hour views, we were more interested in day versus night comparisons. We used the aggregate of noon to 4 PM as “Day” and midnight to 4 AM as “Night,” then contrasted behaviors during these two-time frames. Since opening and closing times vary from pharmacy to pharmacy, these two windows allow us to consistently compare across the sample set without worrying if a specific store is open.

What the Study Found

Of the 3,117 sensors observed:

• More than 67,000 incidents occurred overall.
• 57% had at least one incident during the window of analysis.
• Of the sensors that alerted:
• 77% had more than one incident.
• 25% averaged more than one incident per day.

Source: SmartSense by Digi The above figure illustrates the count of alerts by hour.

Each wedge spans one hour and is scaled to the number of incidents that started during that hour. For example, an incident that started at 3:55 PM would fall in the 3 PM hour. 

Based on our “Day” and “Night” time frames:

• • • We measured a 24% increase during Day over Night.
    • Nighttime hours averaged about 2,500 incidents per hour.
    • Day time hours averaged just over 3,000 per hour.

Making Our Data Actionable

While this initial comparison of Day and Night is interesting, we wanted data that would suggest specific actions we might take to make improvements. When we divided the incidents into warm (> 8℃) and cool (< 2℃), we found:

• • • One third of the incidents (22,000) were above the CDC’s acceptable range.
    • Two thirds (45,000) were below the CDC’s acceptable range.
    • 1,477 sensors recorded warming incidents.
    • 873 sensors recording cooling incidents.
    • 574 sensors had both warm and cool incidents

The significant gap between warming and cooling incidents had interesting repercussions on the average number of alerts per sensor: 

• • • If a sensor gave a warming alert, it averaged 15 warming incidents during the month, or one every other day.
    • If a sensor gave a cooling alert, it averaged 51 cooling incidents, or one every 14.5 hours.

The figure below shows the distribution of warming and cooling incidents over the course of the day. Cooling incidents are indicated by the blue outer ring. Warming incidents are indicated by the red inner ring.

Source: SmartSense by Digi The above figure compares the type of alert by time of day.

Similar to the distribution of total incidents, both rings are skewed to the day, with warming incidents being much more skewed. Using the same Day/Night comparison as above:

• • • Cooling incidents were 4% more likely to happen during the day.
    • Warming incidents were 76% more likely to happen during the day.
    • Most of the total skew was due to warming incidents.

The high average of cooling incidents per sensor considered together with the relatively low skew in Day vs. Night indicate that cooling incidents are triggered by a mechanical issue in the cooler itself. Most likely, a number of coolers were set at the lower threshold and therefore frequently went in and out of range.

Here is an example of a day of readings from the sensor responsible for the most cooling incidents:

Source: SmartSense by Digi Cool alarms did not deviate from the threshold very dramatically.

As the graphic shows, the cooler spans a very small range of temperatures, but sits at the threshold. Every few readings, it goes out of range and starts an incident. Other than a few fluctuations, the rate of cooling incidents is not at all dependent on time of day.

In contrast, the relatively low rate (1 every two days) and the high Day/Night skew of warming incidents (76%) indicates that they are driven by human agents. Warm incidents are less likely to be a result of a mechanical setting and more likely to be caused by a door opened too often or left ajar. Re-stocking is, in fact, a typical cause of warming temperatures in a unit.

Here is an example of a cooler below the threshold prior to 9 AM, which then drifts in and out of range over the course of the day. It comes back into range after the pharmacy closes.

Source: SmartSense by Digi Warm alerts were less uniform in excursion pattern.

In summary, by looking at the number, direction, and timing of incidents, we can determine how to decrease the number of alarms through temperature tuning and human behavior. The results of this analysis provide some evidence that warm alerts are more likely due to human factors while cold alerts are more likely due to equipment factors. This type of insight can drive tailoring of responses to incidents; warm alarms that are triggered during the day should initiate workflows targeted toward addressing operational issues (e.g. a door left ajar, overstocking) while cold alarms should initiate workflows emphasizing equipment evaluation (e.g. check if the setpoint is too low, assess if a minimum duration for alerts should be used). Using the results of this study, we were able to make modest changes to the system alarms to create a better alarm priority for our customers.

Genevieve Shattow is a Lead Data Scientist at SmartSense by Digi.
If you would like to contact Genevieve with any questions about this post, please email her at Genevieve.Shattow@digi.com.