3 MIN EST READ
Emerging Tech Proof of Concept
My Phone is My Interface
By: Christian Langenberg | May 18, 2020
Coronavirus pathogens can reportedly live on surfaces for up to 72 hours, so minimizing the frequency with which we touch things is paramount. This applies to everyday surfaces such as elevator buttons, door handles and touch screen interfaces. FCAT’s Emerging Technology incubator team developed a Proof of Concept that could help customers and associates control a touch screen, without touching it.

The Challenge

Touch screens are all around us. For example, FCAT uses touch screens at the visitor center to demonstrate our work. Given the new health guidelines around touching surfaces, we needed to find an alternative way for people to interact with the content.

The Value

There are a number of interesting advantages with this technology.

  1. The hardware costs are minimal in that you can operate this setup with a standard computer, display, and mic.
  2. No need to download an app (for this simple use case. More complex use cases with bilateral communication would likely require an app).
  3. No bluetooth wifi or network handshake required

What we Learned

We learned that ultrasound can be used to control a remote screen through a simple web interface. The use of a web application has its limitations. Although two way communication between devices is possible, keeping the devices in sync would require deeper integration if both are dynamically changing using sound as triggers.

Team & Tech

As a potential solution to the problem, the Emerging Tech team explored ultrasonic data transmission. The idea is to use sound signals to connect a smartphone with a touch screen, turning the smartphone into a remote control without the need to download an application or require the remote device to pair with the display.

The user scans a QR-code on the display with a smartphone camera (or QR-code reader) to load a webpage. The site has buttons, which the user can use to interact with the display. When the user interacts with the display by pressing the buttons, high frequency sound signals are played. The display “hears” the sounds and reacts accordingly. This way the only screen you “touch” is your own personal device.

Next Steps

The Emerging Technology team will continue experimenting with ultrasonic data transmission and explore aspects such as interference, multi-user scenarios or the ability to handle more involved and complex interactions with the companion device. The team will also explore various other libraries and offerings to find the best technology to support these experiences.

  • Facebook.
  • Twitter.
  • LinkedIn.
  • Print

The Deep Dive

Touch screens are all around us, in the wake of the COVID-19 pandemic people will likely think twice about interacting with a public touchscreen interface. A new solution is needed.

Above is a sample experience where a customer visits one of our visitor centers. The journey suggests that the user approaches the screen and scans a QR-code displayed on the touch screen with a QR-code reader or iPhone camera. The phone opens a website, that shows the buttons, which the user would typically have needed to press on the touch screen. In this setup, however, the user interacts with the touch screen by pressing the buttons on his or her smartphone, which sends high frequency sound signals to the screen in order to change the page.

For this initial POC, the team uses an open source library called ‘quiet-js’ in order to transmit data with sound using web audio. This setup does not require installing any software as the sending and receiving of the ultrasound signals can be handled via web-services.

Using sound signals as a means of connecting a smartphone to a screen is a much less intrusive and complicated process compared to Bluetooth or WiFi . This connection does not require any action by the user such as setting the phone into pairing mode or selecting a WiFi device. All the user has to do is to scan the QR-code.

Please see below a short demo video.

FCAT - My phone is my interface