Of course, when we think about AI monitoring this kind of behavior, we call it corporate surveillance and a slew of dystopian scenarios come to mind: businesses snooping to find out who’s been working and who’s been playing hooky; data gathered to determine who gets fired, gets a raise, or gets promoted. But it doesn’t have to be this way. With all the new data and behaviors these systems now have access to, a far more beneficial application of AI is feasible, one that could vastly improve:

Employee wellbeing. Managers know well the concerns raised by an employee who suddenly starts sending a lot of emails at 3 in the morning. AI systems can not only flag worrisome off hours activity, but also indicate when Zoom fatigue is likely or when a much-deserved break from a given task is a good idea. Changes in facial expressions can indicate even subtle mood changes, and AI systems could suggest taking a short walk, getting a cup of coffee, or even turning the next meeting into an audio-only or walking meeting. Some services, like Receptiviti’s, plug into email and messaging systems like Slack to search for signals that employees are depressed or burned out. Other companies, like Cornerstone OnDemand, take things further and are experimenting with heart-rate data from wearable devices like watches. They’re exploring ways to tie this data to entries from a person’s calendar or project management software to determine whether certain meetings, projects, or even people could correlate with elevated stress levels.¹

Customer experience. As more of our conversations with customers go digital, we can look at calls, video meetings, and chatbot conversations in new ways, looking for changes in customer behaviors and emotions to better understand their context and needs. We can detect in real-time over a video call if we’re losing their attention and can prompt representatives to reengage the customer when needed. Automated summaries of video meetings could be useful in prioritizing product enhancements based on unmet needs. Transcripts of customer video calls can be used to learn more about what customers desire, find annoying or confusing. Changes in customers’ behaviors – whether it’s their voice, expressions, or keystrokes – can also be used to help customers who may be aging into certain disabilities. Researchers are studying whether AI tools that analyze typing speed and speech could be used to better identify people with early-stage dementia.²

Fraud detection. These digital conversations with customers can also be used to find other shifts in customers’ behaviors. Changes in word choices or unusual syntax by customers over chat could potentially indicate that a customer’s identity is being used without their consent. ProctorU, an online exam monitoring service, uses facial recognition software to match students to the image on their ID and verifies their identities with a typing test, to confirm the speed and rhythms of a student’s keystrokes.³ Some insurance companies are using voice analytics to detect whether a customer is telling the truth when submitting a claim.⁴

Inclusive design. There may also be ways to use this technology to increase inclusion for employees and customers. For those with difficulty focusing, an AI prompt could nudge people they’re meeting with to repeat things that may have been missed. When people have difficulty hearing they often make a facial expression that communicates their confusion. AI can read this face and nudge a contact center agent to either speak louder or shift to another interface, like text chat. Other facial expressions by a listener could be a sign that the speaker needs to slow down or explain things in simpler terms. All of this could be done through AI nudges sent in real-time.

Questions to Consider

Even with the best of intentions, deploying this kind of technology can be tricky. Companies need to carefully consider their approach to using the abundance of data we now have, especially in regard to privacy.

Do people want this? Different cultures and age groups may have different tolerances for this kind of relationship with an employer. Many employees are feeling more stressed and exhausted due to the expanded corporate surveillance but are hesitant to speak up.⁵ In April, Zoom removed an “attention tracking” setting, which alerted a call host when a participant was focused elsewhere, following a public uproar about its invasiveness.⁶ Schools faced a major backlash after using cheating software that tracked eye and head movements while students took their exams remotely due to coronavirus. Some students were so afraid that the testing system would brand them as cheating that they cried from the stress, threw up in trash cans, and even urinated at their desks; students with dark skin shined bright lights at their faces, worrying the systems wouldn’t recognize them.⁷

How do we monitor this? Clearly for any level of buy in, communication on this front is essential; we need transparent descriptions of what is being collected and how it’s used. But beyond that, how do we oversee performance? Using facial expressions to identify emotions sounds promising, but some argue that this doesn’t make sense to consider across cultures and contexts.⁸ One person may scowl when angry; another might smile politely. Someone can look downward as a sign of respect; another person may do that out of shyness. In December 2018, a study showed that emotion detection technology assigned more negative emotions to Black men’s faces than white men’s.⁹ In 2016, an algorithm rejected an Asian man’s passport photo for having his eyes “closed”.¹⁰ Perhaps video analytic features will make similar mistakes. Do companies need an AI ethics board to oversee this?

Can we trust this data? As these technologies become more mainstream, people are getting more adept at faking data. A recent analysis found that companies are adapting their language in their forecasts, SEC regulatory filings, and earnings calls due to the rise of AI that analyzes and derives signals from their words.¹¹ And this isn’t stopping with earning calls. Before Zoom removed its “attention tracking” setting, people were easily able to fool the system by using a second device or making sure to head back to the zoom window before 30 seconds passed. Presence Scheduler, which can set your Slack status as permanently active, doubled in sales and traffic in the beginning of the pandemic (until Slack closed the coding loophole).¹² To make people appear more engaged, Microsoft added a feature to its Surface Pro X to switch your eye position so it looks like you’re looking at your camera when you’re actually looking at on-screen faces.¹³ Photoshop’s Neural Filters lets users change facial expressions, strengthening or reducing feelings like joy, surprise, or anger.¹⁴ It’s not hard to imagine this technology being incorporated into videoconferencing.