Bringing Laptops into the 2020s: FPGAs Help Deliver AI Functionality

Once predicted to be supplanted by smartphones and tablets, laptop use is growing due to changing work conditions and emerging technologies. Technology trends like cloud computing and ubiquitous broadband connectivity mean employees can stay productive when working off-campus. Thanks to their full-sized keyboard and high-resolution displays and cameras, laptops are generally considered to be the form factor of choice for business productivity applications like email, word processing, and video conferencing.

While many of the latest technical innovations around power savings and artificial intelligence (AI) have been focused on the smartphone platform, those same innovations can now be applied to laptops without significantly impacting the laptop’s power consumption.

For example, since the display is one of the main contributors to power consumption in a laptop, battery life can be dramatically improved by applying intelligent user monitoring to determine when a user is looking at their laptop. Therefore, it can automatically lower display brightness when the user’s attention is focused elsewhere. This application is already being adopted by smartphones, and the same experience can now be implemented on laptops.

AI Acceptance Increasing

While AI remains a nascent field with immeasurable potential, its value is still being proven to consumers. But certain AI applications have achieved mainstream success, and users are comfortable with using them. One is facial recognition, particularly as a security feature in a smartphone. It’s also now widely used in airports worldwide to track travelers entering or leaving a country as they pass an immigration desk.  

In each of those cases, a camera recognizing a face has strengthened confidence that a computer user—or someone entering the country—is who they say they are. Now that consumers are growing comfortable with AI-enabled experiences, expect smart vision applications like facial recognition to be more widely implemented on laptops.

Security and Privacy

While both security and privacy are critical for desktop computers, they’re even more important for mobile laptops. Whether used for work or for personal business, security, and privacy—technically different concepts—merge as a requirement. The need for security to protect sensitive data overlaps with the desire to keep that data private.

We’re all too familiar with the fact that security and privacy often come at the cost of convenience. While burdensome security measures clearly protect a laptop that’s out in public, they can become more annoying (and even unnecessary) when the laptop is in a known, controlled environment like a work or home office.

One of the benefits of a laptop that leverages AI for situational awareness is that it can adjust the security behaviors according to the laptop’s location. At home, convenience can be enhanced, since there’s less physical risk. In a café, by contrast, security can be tightened due to the unfamiliar surroundings and people.

And it’s not just about deciding on the location of the laptop. The same camera that scans the environment can make note of when the user leaves the computer to get more coffee or go to the restroom. The computer can then lock itself down without the user having to remember to do so.

In a similar vein, the camera may notice that another pair of eyes is looking at the screen over the user’s shoulder, causing the laptop to take action to thwart possible viewing by an unauthorized user. Given the multitude of well-publicized computer hacks, AI can also help monitor internal behaviors and activity to alert the user to anything suspicious.

These are a few examples of how laptops are evolving to meet the needs and expectations of the modern workforce.  The balance of this article will focus on two trends in particular—always-on and instant-on behaviors and new laptop form factors (e.g., detachable screens or new laptop shapes)—and how FPGA solutions require far less power than ASIC-based approaches.

Always On and Instant On

Today, there are only three ways to put a laptop to sleep or wake it up: push a button (maybe with the mouse), close the lid, or wait for the laptop to time out. Newer laptops might also be awakened by a voice command.

The challenge with a laptop is its central system-on-chip (SoC). The SoC is in charge of what happens in the laptop, and for that reason it’s involved in most tasks performed by the laptop. It contains the CPU cores and may include other processing and peripheral circuits. It also burns a lot of energy in doing its work.

Because the SoC is involved in so many of the laptop’s activities, there’s little that the laptop can do while the system—particularly the SoC—sleeps. The only capabilities available during sleep relate to the few options for waking the system back up. No other tasks can run on a sleeping system. In some cases, if they do (download email, network monitoring), they consume hundreds of milliwatts of energy, which is a significant drain on battery life.

A laptop can perform many tasks even while the SoC sleeps if it’s enhanced by an AI layer that responds to sensor signals. The AI capabilities can do work or make decisions while the SoC is asleep, ultimately determining whether the SoC should wake up for further work or if it can remain asleep and let the AI layer handle the required task.

Low-power FPGAs make an ideal hardware platform for the AI layer. They provide enough processing power to handle basic AI tasks (presence detection, for example), but do so while consuming much less power than the SoC. If the AI layer determines the SoC needs to be involved with the task, it can then wake it to supplement the AI layer’s processing capabilities or take other action.

But such AI has so far mostly been adopted by Internet of Things (IoT) devices. For example, a smart doorbell might have the following features:

• Detecting the presence of a human and other objects like packages.
• Detecting and/or recognizing a specific face or person. These can be specific people registered as authorized users (like the house’s residents) or people who routinely pass by the doorbell camera, like a uniformed delivery person or gardener.
• Interpreting a few hand gestures or understanding a few verbal commands. Combining facial recognition with a spoken key phrase or password enables dual authentication logins for heightened security.
• Leveraging a sensor-agnostic inference engine to minimize sensor data being sent over cloud.

On their own in an application like this, with no assistance from an SoC, FPGAs such as those developed by Lattice can deliver higher performance with lower power consumption than that of a standalone CPU. That’s because the CPU must execute its tasks as software, while the FPGA uses a far more efficient, hardware-based implementation. For example, a CPU controlling a smart doorbell or security camera might process 1 to 2 frames per second while burning 100 mW; today’s FPGAs can handle 5 frames per second at 7 mW—3X to 5X the performance at 1/14^th the power.

If this FPGA can drive a smart doorbell or camera, then it can also drive the following features in a laptop:

• Detect when the user is looking at the screen. When the user looks away, the laptop can save power by dimming the screen, and then brighten it when the user refocuses on the screen.
• Adjust security according to the laptop’s location.
• Detect where the user’s gaze is directed. This can help with tasks that would ordinarily require a mouse; instead, eye-tracking could provide hands-free operation of some frequent tasks.
• Optimize how energy is spent on the graphics. That energy can be used for rendering the part of the screen where the user is looking. The rest of the screen can remain fuzzier or more poorly rendered, saving energy, until the user shifts their gaze to a different part of the screen.
• Influence the narrative trajectory of a game, directed by the movement of the player’s eyes.
• Detect someone looking over a laptop user’s shoulder
• Dim the laptop screen if no one is using it to save battery life

An FPGA can help offload some of the AI processing from a laptop’s SoC (Fig. 1).

In this implementation, the sensor interface collects the inputs from the individual sensors, processing them in neural-network (NN) accelerators within the FPGA, assisted by on-chip SRAM that stores the weights and activations required for processing the neural nets. Decisions made by the accelerators can then wake the SoC, if necessary, for further processing.

The AI layer is built using multiple components of recently-introduced solution stacks.  For instance, with sensAI from Lattice, designers have access to hardware, neural-network acceleration engines that support implementation of a wide range of networks such as MobileNet, ResNet, SSD on FPGA resources, and compiler tools for adapting networks trained on the major frameworks like Caffe, Keras, and TensorFlow. The stack also includes reference designs and demonstrations covering various common AI applications (like presence detection, object counting, facial recognition, and others).

Some FPGA providers offer custom design services for developers who simply want the AI functionality without having to immerse themselves in how to develop and implement an AI application (Fig. 2).

New Form Factors

The second challenge for modern laptops will be figuring out how to move beyond the familiar traditional form factors. Part of that move is likely to involve flexibility—even if that means simply bending the screen in the middle to close the unit. Laptop engineering today is challenged by the difficulty in routing numerous signals, which run a variety of protocols, all around the laptop.

This can be handled simply by reducing the number of signals through signal aggregation. Multiple low-bandwidth interfaces can be multiplexed onto a single higher-bandwidth signal for de-multiplexing elsewhere, dramatically lowering the wiring footprint in the laptop. It makes board-to-board connections easier; it simplifies on-board routing; it means fewer wires traversing a hinge or other flexible element; and it helps improve reliability by reducing the number of connectors needed.

Signal aggregation can be implemented on single wires or low-voltage differential-signaling (LVDS) pairs. Advanced FPGAs today can implement these signals at various performance points, providing options for the most efficient implementation. For example, some models can drive 7 Mb/s onto a wire and higher-bandwidth aggregation speeds are possible.

The challenge, though, comes from the fact that the different signals may represent different protocols. That challenge can be met by implementing an I/O hub, similar to a sensor hub. Such a hub terminates local I/O signals of different flavors and multiplexes them onto fewer high-speed wires. Those wires travel to an I/O hub at the other end, where the signals are de-multiplexed and reacquire their original protocol nature.

Therefore, FPGAs need to provide the physical hardware for receiving the I/O signals and transmitting the multiplexed versions (Fig. 3). Providing the IP for building the I/O hub would also be very useful to a designer. As signal aggregation is sensor-agnostic, data streams from different sensor types (RGB, IR, LiDAR, radar, etc.) or signals traveling over different bus types (I²C, I²S, UART, GPIO) can be aggregated over the same pin or cable. FPGA specialists with longstanding expertise in enabling signal aggregation in mobile phones can apply that expertise to sensor aggregation in laptops.

Engineers are working on developing laptop computers that will provide users with the kinds of experiences that they’ve already come to expect from smartphones and tablets. The future laptop will be easier to use, have a longer battery life, and be more secure than today’s models. Some of the most noticeable improvements will be in energy consumption and form factor. Laptops will employ the power-reduction techniques that allow batteries to last an entire day or more. And they will be more flexible and easier to carry.

New FPGA designs will be key to enabling these new capabilities and features in laptops. Doing more work outside the SoC allows it to sleep more, waking only when its computing power is truly required. A sensor hub with an AI layer, implemented in pure hardware, will connect multiple cameras, microphones, and other sensors, meaning more downtime for the SoC. And an I/O hub driving multiplexed signals will reduce signal count, making laptops easier to build and providing a richer user experience by enabling AI-based sensor fusion.

These capabilities can help developers create lighter, lower-power laptops that will serve business and personal users through the 2020s.

Hussein Osman is Product Marketing Manager at Lattice Semiconductor.