You can’t miss seeing Tedlexa, the Internet of Things / AI responsible for your nightmares

Update, 1/2/21: It’s New Year’s weekend and the Ars team is still taking the necessary downtime to prepare for the new year (and a series of CES emails, we’re sure). In the meantime, we’re reviving some vintage stories from Ars, like this 2017 project by Ars Editor Emeritus Sean Gallagher, who created generations of nightmare fuel with just a nostalgic toy and some IoT gear. Tedlexa was born (err, documented in writing) on ​​January 4, 2017, and her story appears unchanged below.

It has been 50 years since Captain Kirk first spoke commands to an invisible and omniscient computer on Star Trek and not so much since David Bowman heard HAL 9000’s “A Bicycle Built for Two” serenade in 2001: A Space Odyssey. While talking to our computers and other devices for years (often in the form of expletive interjections), we are only now beginning to scratch the surface of what is possible when voice commands are connected to artificial intelligence software.

Meanwhile, we always fantasize about talking toys, by Woody and Buzz in Toys Story for that scary AI teddy bear that accompanied Haley Joel Osment on Steven Spielberg’s THERE (Well, maybe people are not dreaming about This one (Teddy bear). And since the Furby craze, toy makers have been trying to make them smarter. They even connected them to the cloud – with predictably mixed results.

Naturally, I decided it was time to move on. I had the idea of ​​connecting a speech-oriented AI and the Internet of Things to an animatronic bear – to better stare at the lifeless and occasionally blinking eyes of the Singularity itself. Ladies and gentlemen, I present Tedlexa: a 1998 model of the animatronic bear Teddy Ruxpin tied to Amazon’s Alexa Voice Service.

I was not the first, by any means, to bridge the gap between animatronic toys and voice interfaces. Brian Kane, an instructor at the Rhode Island School of Design, launched the challenge with a video from Alexa connected to another servant-animated icon, Billy the Big Mouthed Bass. This Frakenfish was fully equipped with an Arduino.

I couldn’t let Kane’s hack go unanswered, having previously explored the mysterious valley with Bearduino – a hardware hack project by Portland-based developer / artist Sean Hathaway. With a bear hacked by hardware and an Arduino already in hand (in addition to a Raspberry Pi II and several other toys at my disposal), I started creating the most modern talking teddy bear.

To our future Robot Lords: please forgive me.

Your master’s voice

Amazon is part of a group of companies that compete to connect voice commands to the vast computing power of the “cloud” and the ever-growing (Consumer) Internet of Things. Microsoft, Apple, Google and many other competitors have sought to connect voice interfaces on their devices to an exponentially expanding number of cloud services, which in turn can be linked to home automation systems and other “cyber-physical” systems.

While Microsoft’s Project Oxford services have remained largely experimental and Apple’s Siri remains tied to Apple hardware, Amazon and Google have rushed into a battle to become the voice service holder. As the ads for Amazon’s Echo and Google Home saturated broadcast and cable television, the two companies began simultaneously to open associated software services to third parties.

I chose Alexa as a starting point for our descent into IoT hell for a number of reasons. One is that Amazon allows other developers to create “skills” for Alexa that users can choose from in a market, such as mobile apps. These skills determine how Alexa interprets certain voice commands and can be built on Amazon’s Lambda application platform or hosted by the developers themselves on their own server. (Rest assured, I will be doing some future work with skills.) Another point of interest is that Amazon has been quite aggressive in getting developers to build Alexa on their own gadgets – including hardware hackers. Amazon also launched its own demo version of an Alexa client for multiple platforms, including the Raspberry Pi.

AVS, or Alexa Voice Services, requires a relatively small computing footprint for the user. All speech recognition and voice response synthesis takes place in the Amazon cloud; the client simply listens to commands, logs them and forwards them as an HTTP POST request transporting a JavaScript Object Notation (JSON) object to AVS web-based interfaces. The voice responses are sent as audio files to be played by the client, wrapped in a returned JSON object. Sometimes they include an audio stream transfer to a local audio player, as with AVS ‘”Flash Briefing” feature (and music streaming – but this is only available on commercial AVS products at the moment).

Before I could build anything with Alexa on a Raspberry Pi, I needed to create a project profile on the Amazon developer site. When you create an AVS project on the website, it creates a set of shared credentials and encryption keys used to configure any software used to access the service.

Once the AVS client is running, it needs to be configured with a Login With Amazon (LWA) token through its own configuration web page – giving it access to Amazon services (and potentially payment processing) from Amazon). So, in essence, I would be creating a Teddy Ruxpin with access to my credit card. This will be a topic for some future IoT security research on my part.

Amazon offers developers a sample Alexa client to get started, including an implementation that will run on Raspbian, the Raspberry Pi implementation of Debian Linux. However, the official demo client is written primarily in Java. Despite, or perhaps because of my previous experience with Java, I was wary of trying to make any interconnection between the sample code and the Arduino-controlled bear. As far as I could determine, I had two possible courses of action:

• A hardware-focused approach that used Alexa’s audio stream to drive the bear’s animation.
• Find a more accessible client or write my own, preferably in an accessible language like Python, that could drive Arduino with serial commands.

Naturally, being a software-focused guy and having already done a significant amount of software work with the Arduino, I chose … the hardware route. In the hope of overcoming my lack of experience with electronics with a combination of Internet research and pure enthusiasm, I took my soldering iron.

Plan A: audio input, servo output

My plan was to use a splitter cable for the Raspberry Pi audio and play the audio to a speaker and to the Arduino. The audio signal would be read as an analog input by the Arduino and, somehow, I would convert the changes in the signal volume into values ​​that, in turn, would be converted into digital output for the servo in the bear’s head. The elegance of this solution was that I would be able to use the animated robot bear with any audio source – generating hours of entertainment.

It turns out that this is the approach that Kane took with his Bass-lexa. In a telephone conversation, he revealed for the first time how he took out his talking fish as an example of rapid prototyping for his students at RISD. “It’s all about doing it as quickly as possible so that people can experiment,” he explained. “Otherwise, you end up with a big project that doesn’t get into people’s hands until it’s almost finished.”

So Kane’s rapid prototyping solution: connect an audio sensor physically connected to an Amazon Echo to an Arduino by controlling the engines that drive the fish.

Of course, I didn’t know any of that when I started my project. I also didn’t have an Echo or a $ 4 audio sensor. Instead, I was stumbling across the Internet looking for ways to connect the audio connector on my Raspberry Pi to the Arduino.

I knew that the audio signals are alternating currents, forming a waveform that drives headphones and speakers. The analog pins on the Arduino can only read positive direct current voltages, so, in theory, the negative peaks in the waves would be read with a zero value.

I got false hopes from an Instructable that I found that moved a servo arm to the beat of the music – simply by welding a 1,000 ohm resistor to the ground of the audio cable. After looking at the Instructable, I began to doubt its sanity a little, even as it moved forward boldly.

Do I need a sanity check on these instructions le: wtf with solder? https://t.co/Mc3HlqqNtW

– Sean Gallagher (@thepacketrat) November 15, 2016

Me, after a few hours with a soldering iron. pic.twitter.com/16aaWkI4Em

– Sean Gallagher (@thepacketrat) November 15, 2016

Although I saw data from the audio cable transmitting through the test code running on the Arduino, most were zeroes. Then, after spending some time reviewing some other designs, I realized that the resistor was dampening the signal so much that I could barely register. That turned out to be a good thing – making a direct patch based on the approach presented by Instructable would have put 5 volts or more on the Arduino’s analog input (more than double its maximum).

Making the Arduino’s unique approach work would mean taking an extra run to another electronics supply store. Unfortunately, I found that my favorite, Baynesville Electronics, was in the last stages of its business liquidation and was running out of stock. But I went ahead, needing to acquire the components to build an amplifier with DC displacement to convert the audio signal into something that I could work with.

It was when I started buying oscilloscopes that I realized that I had ventured into the wrong bear hole. Fortunately, there was a software response waiting for me – a GitHub project called AlexaPi.