Download this article in PDF format.
The 1960s television sitcom “The Jetsons” featured an early depiction of a smart home. Conveniences included Rosie the robot housekeeper, video chatting, and food available at the touch of a button. There was also an aerocar so that the main character George Jetson could commute to his job, where he worked for one hour a day, two days a week.
The Home of Tomorrow
Over 50 years on, the true smart home – a seamlessly connected space in which our needs are anticipated, our lives are made easier and healthier, and our carbon footprint is reduced –is still some way in the future. But today’s explosion of technological innovation, and AI in particular, is enabling companies to deliver more and more elements of what will ultimately comprise the smart home.
According to an October 2017 report from The NPD Group1 (Fig. 1), 15% of U.S. internet households own a home automation device, up from 10% in April 2016. NPD reported strong growth across all types of devices, with security and monitoring representing the largest share of dollar sales, and product categories such as video doorbells and smart lighting also growing quickly.
1. Sales are on the rise for home-automation devices.
In addition, NPD reported that ownership of voice-activated wireless speakers (e.g., Amazon Echo and Google Home) had more than tripled over the year, totaling 10% of U.S. internet households at the time of the report.
Smarter Homes Require Smarter Technologies
Building on this foundation, many new devices are entering the market to offer increased efficiencies for our home lives.
One of the most highly anticipated (and possibly feared) innovations is the robot. While today’s digital assistants are confined to certain areas of the home that have wall sockets, we will see this continue to evolve—imagine an implementation of Alexa that can maneuver around your home and help with the laundry. Already Roomba and other robotic vacuums are helping many people keep their floors free of debris; and robotic lawn mowers, while not yet affordable enough for mass adoption, are already on sale. According to a 2018 report by Juniper Research,2 domestic aide robotics like the Roomba will drive consumer robotics hardware revenues from an estimated $6.4 billion in 2018 to nearly $23 billion by 2022—an increase of over 250%.
Our homes will soon also incorporate ubiquitous voice and gesture control. Voice today is already becoming widespread through digital assistants. Alongside gesture control, using this technology will feel even more natural. To work effectively, today’s gesture-tracking technology requires depth sensors backed by high-performance computers, so it’s not yet practical for all devices. This is changing, though, with the help of new software and more efficient hardware.
Today you can already buy products that control the lights, turn on your TV, control the thermostat, and more—all with a wave of your hand. However, in the future, we expect the technology to blend more seamlessly into the living environment, allowing consumers to interact more naturally with their smart home.
More immersive and interactive entertainment and experiences are also coming to the home through the proliferation of untethered VR and AR devices. Today’s AR devices show what are very obviously computer-generated graphics over the top of the real-world view, but higher-performance graphics will make this less obvious and more realistic over time.
On the VR side, the true immersive capability will come when untethered devices have the same graphics capability as tethered devices (Fig. 2). One drawback today is in degrees of freedom (DoF): On untethered devices today, there are only 3DoF, versus 6DoF in tethered devices. The difference is in tracking a user’s position not just by where they’re looking, but also how they’re moving—an important consideration for delivering a realistic experience. These challenges will be overcome, and we will see untethered devices with location services, increasingly better screen quality, 3D room scanning, and low network latency—all leading to true immersion in a virtual world.
2. The true immersive capability of VR will come when untethered devices have the same graphics capabilities as tethered devices.
Another exciting development is in the area of personalization across devices. Whether it’s your television or your digital assistant, AI will enable devices to recognize different family members and deliver personalized services. You will turn on your TV and it will recognize you and adjust the settings accordingly (with parental controls, favorite programming, etc.).
This personalization is also coming to entertainment robots, with several robots already available that leverage a combination of robotics, AI, and computer vision to recognize family members. Interactions in some cases include simulated emotional responses. Makers of these devices are designing them to increase and hone their skills, creating more natural interactions and capabilities over time.
Across these devices and many others, ubiquitous connectivity is coming. It’s this connectivity that will enable the home to provide useful services. Devices will integrate Wi-Fi, Bluetooth, 5G and even location services to provide a better experience.
SoC Design Requirements
All of these innovations will utilize systems-on-chips (SoCs) that integrate IP cores including CPUs, GPUs, neural-network accelerators (NNAs), and connectivity processing engines. If they are to overcome today’s limitations and deliver a compelling user experience, they need IP cores designed with industry-leading performance, power, and area (PPA).
With highly optimized IP, SoC vendors will produce devices that deliver more performance per watt, OEMs will produce products with new features for new markets, and third-party developers can partner with OEMs to deploy advanced algorithms that differentiate their products.
For example, for almost three decades, Imagination Technologies has been creating technology to deliver IP with high-end PPA. As the provider of GPUs at the heart of many smartphones, tablets, smartwatches, smart TVs, etc., the company’s focus on delivering the best possible user experience within mobile power budgets is at the center of every design decision made for its GPUs, NNAs, image processors, and connectivity processors.
Conserving Power
Power efficiency is critical to battery-operated devices in ensuring they can function for an acceptable time before needing a recharge. It’s also critical for many wired products that are designed to operate within energy-efficiency guidelines, and for devices like AR/VR devices that are worn on the face (where a hot device will lead to a very poor user experience). IP that’s designed for low-power operation, and made to be easily integrated into SoC-level power-management schemes, can reduce overall SoC power consumption, leading to longer battery life and less heat output.
Efficient GPUs
For many of the entertainment devices of the future home, immersive, realistic graphics will play a key role. Today’s GPUs are also being used for applications beyond graphics on devices without screens, since they’re better parallel-processing engines than CPUs, tuned for complex mathematics operations. We’re starting to see SoCs that incorporate GPUs for high-end compute to deliver real-time 6DoF computation in VR; precise alignment of graphics overlay on real-world in AR; and a combination of NNA and GPU compute for AI and natural language processing. For these use cases, GPUs must support larger formats, greater bit depth, wider dynamic range, better low-light performance, faster response time, and greater functionality.
GPUs contribute a great deal to overall system power consumption. In mobile devices, which are generally limited to an SoC thermal envelope of 3 to 3.5 W, the GPU accounts for 30% to 50% of the power consumption. For instance, every generation of Imagination’s PowerVR GPUs has employed a number of new techniques to drive down power.
The PowerVR Furian Series8XT GT8525 GPU running at an estimated 35 fps/W in a mid-range mobile SoC (manufactured on TSMC’s 10FF process) falls in the sweet-spot of under 1.5W of power. Compared to the previous generation Series7XT, for the same performance target, the GT8525 will use up to 60% less power. This boost in performance/W enables end users to enjoy devices with increased endurance and longevity, especially when under heavy load.
The foundation of efficiency in PowerVR GPUs is tile-based deferred rendering (TBDR), where the GPU only renders the final elements that will be visible on the screen; there’s no benefit in calculating the positions and drawing hidden objects. This technique ensures the lowest possible bandwidth usage and the lowest number of processing cycles per frame. This leads to high performance efficiency and the very low power consumption per frame.
Connectivity
The home of the future is all about connectivity, and there are multiple standards to support. Wi-Fi of many flavors will play a role across many devices such as 802.11ac and 11ax for high-bandwidth connectivity (Fig. 3). There also remains a strong demand for the older 802.11n standard for low-power, low-bandwidth applications. But because Wi-Fi is traditionally power-hungry, if it’s added it to battery-operated or energy-conscious devices, a low-power connectivity engine such as Imagination’s Ensigma is needed.
3. The home of the future is all about connectivity, and there are multiple standards to support it.
Ensigma low-power connectivity IP has multiple power domains for each of the functional blocks. It also has a sleep controller, which provides power to each power domain at the latest possible time and cuts off power at the earliest possible time. With its optimized architecture, Ensigma low-power connectivity IP consumes roughly half the power consumption of similarly featured IP.
Neural Processing
The many personalized and value-added features that are coming to the smart home will be driven by NNAs. These engines enable natural language processing, voice-recognition technologies, vision-processing algorithms, and more. Because this technology will be implemented in many battery-powered devices with built-in intelligence, power consumption is key.
One such NNA, Imagination’s PowerVR 2NX NNA, employs many techniques to optimize power consumption (Fig. 4). This includes flexible bit-depth support on a per-layer basis for weights and data, which lets 2NX maintain high inference accuracy while dramatically reducing bandwidth and power requirements.
4. The table highlights the precision flexibility for optimized performance, power, and bandwidth from Imagination Technologies’ PowerVR Series2NX neural-network accelerator.
Moving from 8- to 4-bit precision for appropriate use cases enables the 2NX to consume two-thirds of the power while maintaining accuracy. 2NX also allows for offline processing, saving bandwidth across internet connections and improving the speed using local “intelligence” on the device itself. With high inferences/milliwatt and highest inferences/mm2, it efficiently runs all common neural-network computational layers and is claimed as the only IP to meet all requirements for deploying neural networks within mobile/embedded power envelopes.
Reasonable Cost
Consumers will only begin to adopt products en masse when they’re affordable. Unfortunately, smart-home appliances are still expensive. Even smaller items such as networked light bulbs, smart thermostats, door and window locks, and smart electricity sockets aren’t yet attractively priced. Smart-home starter kits can cost several hundreds of dollars. And since adding more sensors can be prohibitively expensive, homeowners often abandon their plans to expand the system.
For silicon design, the biggest contributor to cost is silicon area. For products with GPUs, NNAs, and various connectivity requirements, you can imagine that all of the integrated technology can lead to a fairly large silicon die, but consumer products have a short lifecycle and need to be affordable. Whether creating a highly integrated single-chip design or a companion chip, keeping area to a minimum will decrease costs. IP that’s architected for compact area will absolutely play an important role.
In terms of low cost, Imagination has developed the PowerVR Series9XE/XM GPUs (Fig. 5). According to the company, they break the constraints of typical “low-cost” GPUs to deliver a high level of flexibility for both graphics and GPU compute. One of the keys to achieving the needed performance with reduced silicon area and power consumption was to shrink the required bandwidth between the GPU and frame buffer, along with the required size of the frame buffer. This has implications in terms of the amount, type, and speed of DRAM needed in the design, and leads to reduced area. In addition, these bandwidth reductions lower the power consumption of the graphics subsystem.
5. Shown is the architecture of the PowerVR Series 9XE and 9XM GPU.
In devices such as mobile phones where a GPU is mandated, it’s possible to use a 9XE/9XM GPU to manage the classic vision-processing algorithms and offload the neural-network processing to the PowerVR 2NX NNA. Because of the performance density of the PowerVR NNA and the GPUs, this combination is easily implemented in the same silicon footprint as a competing standalone GPU.
Another way to save on system cost, especially when it comes to connectivity, is through integration. Many of today’s electronic devices implement connectivity as a separate chip, but with IP like the Ensigma low-power connectivity engine, this functionality can be integrated on the main SoC, leading to reduced bill-of-materials (BOM) costs. Because of the multitude of connectivity standards in the home, it’s important to be able to manage multiple standards and prepare for future standards. As communication standards evolve, so too can those devices that use a configurable demodulation engine like Ensigma, via firmware upgrades in the field. This leads to a device with a longer lifetime—and less-frequent costly upgrades.
Advanced Features “as Standard”
All of the affordability and battery longevity in the world won’t make a product successful if it’s not useful and fully featured. Product developers need to deliver true value and a compelling user experience. Silicon IP delivers the underlying performance and features needed to drive these experiences.
For VR and AR devices, GPUs deliver high-performance compute for imaging processing (e.g., stabilization, correction, improvement, face detection, and beautification tools), multimedia (e.g., real-time stabilization, information extraction, and superimposition of information), computer vision (e.g., augmented reality, edge and feature detection), and gaming. These GPUs must support larger formats, greater bit depth, wider dynamic range, better low-light performance, faster response time, greater functionality and more. The PowerVR Furian architecture is designed to scale efficiently to the high-performance points needed, and does so in a way that mitigates performance density issues.
Many home devices of the future will require SoCs that process data locally and relay data to the cloud for onward processing and data aggregation. In many cases, the inferencing could be run on powerful hardware in the cloud; however, for many reasons, it’s now time to move this to edge devices. Where fast response is essential, it’s simply not practical to run neural networks in the cloud due to latency issues. Moving the processing onto the device also eliminates the security issues that could occur.
As cellular networks may not always be available, be they 3G, 4G or 5G, dedicated local hardware will be more reliable, as well as offering greater performance and, crucially, much-reduced power consumption. As applications make use of ever more complex neural networks, manufacturers need dedicated hardware solutions along the lines of the PowerVR 2NX if they’re to offer competitive devices (Fig. 6).
6. The evolution of the embedded-system SoC.
The 2NX NNA represents an inflection point in neural-network acceleration and performance. An image-classification network running on a single-core 2NX achieves up to 2048 MACs/cycle, and can perform image classification on 42,000 images per minute. The 2NX can be used standalone or combined with other processors (e.g., CPUs/GPUs) depending on computational needs. It has an optional memory-management unit (MMU) for use with Android/other modern operating systems, without the need for additional silicon or complex software integration.
On the connectivity side, given today’s many home standards, what’s needed is a connectivity engine that supports them all. The future home gateway will likely combine 802.11ax and 802.15.4 (e.g., Zigbee, Thread) and potentially Bluetooth. In addition, some operators are now considering re-adding DECT in the gateway to support phone handsets, as they retire traditional phone service in favor of providing broadband only. Ensigma’s multi-standard and programmable architecture provides solutions for every form of communications and networking, including 802.11a/b/g/n/ac (up to 2 x 2 MIMO), Bluetooth 4.0 (LE), FM, DAB, and several others, as well as demodulation of all of the world’s digital TV standards.
Visionary Technologies Deliver True Innovation
To achieve the vision of a truly smart home, there are many challenges to overcome in terms of complexity, affordability, interoperability, ease of implementation, and countless others. These challenges will take time for the industry to address. It will also take time for consumers to accept and welcome many of these technologies into their homes. In the meantime, we will continue to see smarter home products that make our lives easier, our interactions richer, and our entertainment more immersive.
The true vision of the smart home will be realized when customers can affordably and easily implement the needed technologies; moreover, when the smart home begins to aggregate the data generated by all of the connected devices therein. At that point, the system will monitor itself, make intelligent decisions, and manage everything autonomously without consumers’ intervention.
With IP like that from Imagination, we’re moving quickly in that direction. Only time will tell if these conveniences will lead to a work week that’s only two hours long, like that experienced by George Jetson.
Simon Forrest is Director of Connectivity & Connected Home at Imagination Technologies.
References
1. “U.S. Home Automation Sales Growth Fueled In Part By Voice Control and Digital Assistants,” retrieved from The NPD Group website: https://www.npd.com/wps/portal/npd/us/news/press-releases/2017/us-home-automation-sales-growth-fueled-in-part-by-voice-control-and-digital-assistants
2. “Consumer Robotics Hardware Revenues to Reach $23BN per Annum by 2022, as Domestic Aide Robotics Dominate,” retrieved from the Juniper Research website:
https://www.juniperresearch.com/press/press-releases/consumer-robotics-hardware-revenues-to-reach-$23bn
