Connecting an LLM to the Real World: The Role of APIs for Managing and Supercharging the Smart Home, and How Amazon is Helping
A more intelligent assistant that understands more of what you say and less specific terminology will be the first big change with the new, yet to be named, Amazon product. One of the most frustrating things to have happen with voice assistants is having to repeat yourself after asking it to turn down the thermostat. Please choose unique names and run discovery again,” for the 900th time when all you want is to see where you left the remote control.
Generative AI has looked like their best shot at survival for a while. Digital assistants have a bit of an element of Artificial Intelligence, but it is missing in the processing capabilities and more human-like interactions generativeAI is capable of. This is a big moment for the smart home, as it could take home automation to the next level, moving it from a remote control experience to a home that’s, well, actually smart.
“When you connect an LLM to the real world, you want to minimize hallucinations — and while we think we have the right systems in place … there is no substitute for putting it out in the real world,” says Limp. If you want to be notified when you can join the preview, tell your Echo device, “Alexa, let’s chat,” and your interest will be registered.
This is a machine that may be free at times. Limp said that while Alexa, as it is today, will remain free, “the idea of a superhuman assistant that can supercharge your smart home, and more, work complex tasks on your behalf, could provide enough utility that we will end up charging something for it down the road.”
Say a phrase like “Alexa, I’m cold” and the assistant will turn up the temperature on your thermostat. It’s going to know what room I’m in and what the Seahawk’s colors are, and it’s going to make those translations between the two.
The API is the key, according to Limp. “We’ve funneled a large number of smart home APIs, 200-plus, into our LLM.” This data, combined with Alexa’s knowledge of which devices are in your home and what room you’re in based on the Echo speaker you’re talking to, will give Alexa the context needed to more proactively and seamlessly manage your smart home.
This is similar to how many smart lighting apps — such as Philips Hue’s — make it easy to quickly program lights to react based on the time of day, ambient light, and activity, if you have the right hardware. The advantage here, though, is there’s no programming required in the app at all, and you get a wider compatibility with lights. Any lights that work with Alexa will work with automatic lighting, and that’s basically all of them.
Initially, the multiple command feature will only work with a subset of device types — including lights, smart plugs, and a few others, says Limp. Adding everything is one of the things that the team is working on.
Amazon Dynamic Controller: What is it like to light up a Lightbulb if you screw it up and light it on a speaker?
Dynamic Controller will enable features such as prebuilt scenes for lighting control to be surfaced more naturally. So, if you have GE Cync-colored light bulbs and say, “Alexa, make it look spooky in here,” Alexa will know what to do without you having to program a Routine or import scenes to the Alexa app.
Amazon is opening up the program to more developers, so they can work on the features that it has already worked with. Amazon hasn’t provided details on how or when developers will gain access to these tools; we’ve reached out for clarification.
Automatic lighting is going to be a feature of the Amazon Echo smart speakers. Smart lighting is a popular use case, but can be hard to set up and control. With this new capability, Amazon is looking to make it as easy as just buying and screwing in a lightbulb.
It does not require a setup, but automatic lighting can be changed. You can specify how much brightness will make the lights turn on and what time of day you want them to work.
Amazon’s Featured Routines for the Ring App based on your Ring devices and compatible non-Ring devices like smart lights, locks, and doorbells
Amazon is also bringing its Featured Routines to the Ring app. These are prepopulated Alexa Routines designed to show users how to get the most out of their devices. Previously only offered in the Alexa app, Amazon is bringing them to the Ring app based on your Ring devices.
Ring devices, including lights, alarms, and doorbells, will work with the routines. It will also support non-Ring devices that are compatible with Alexa, such as smart lights and locks.