Autonomy Alone Isn’t Enough: An Intro to Teleoperation

Who doesn’t know about self-driving cars? Perhaps the first time you heard of autonomous vehicles (AVs)—if you’re old enough to remember—was watching KITT in the Knight Rider television series. Or maybe when you saw early talks regarding a self-driving Google car. Surely you read about the fatal collision of Uber’s autonomous car.

The idea of having your own driver is an alluring one. The availability of artificial-intelligence driving means that luxury could be available to the everyman, or woman. Nevertheless, since the announcement of self-driving cars in 2009, we still don’t seem any closer to the dream of actually dreaming while on the road.

Indeed, why aren’t we closer to fulfilling our AV fantasy?

Apparently, the technology that’s required is much more complex than originally thought. Key challenges remain, including perfecting sensors, adaptive machine learning, new and surprising conditions, regulation, and overcoming social apprehension.

Currently, self-driving cars use a combination of LiDAR, radar, GPS, and direction sensors. All of these are supposed to combine and provide an all-weather, all-situation solution enabling an AV to see everything needed to navigate the roads and get us to our destination.

AV Challenges

However, there are many conditions that this complex combination of hardware and programming is still incapable of mastering. These include fog, obstructed road signs, and temporary road changes.

Once the issue of sensors is resolved, all of this data needs to be processed by an on-board computer and turned into actionable information. With the use of AI and machine learning, autonomy companies are attempting to generate algorithms that will understand the dynamic settings around the vehicle so that it knows what actions to take. To date, there’s no definitive answer to this issue. The process of evaluation and re-evaluation is ongoing.

In addition, an AV will confront constantly shifting engagements. Yes, it’s learned from previous experiences what a traffic light is or how to identify a pedestrian. Yet how can we be sure it will know the correct way to utilize things from the past and adapt these for a new road, a new situation, or a new object? Eventually, yes, but not yet. And not by a long shot.

Some issues facing AVs are not technological at all. Governments are in no rush to establish regulations for this futuristic technology. When is an AV allowed to drive? Where? How? What safety systems must be in place? The absence of any empirical information and data makes it difficult to reach decisions regarding a vehicle that could cause damage, injury, and death.

Even if all technical and safety issues are resolved, though, nothing will get people to use an AV if they don’t TRUST it. If one needs to reset their smartphone every other day, or deal with computer freezes, or the wrong tap on a tablet causes a malfunction, how can they feel safe in a car that drives itself and have no direct control? Even an elevator, which relies on 100-year-old technology, depends on the ability—and availability—of a technician to manually override the system and control it, as necessary.

There is a solution. These problems can be solved with an additional technology.

Teleoperation

It’s unlikely that vehicular autonomy will ever be able, or allowed, to function completely independently. That’s why teleoperation is the linchpin to bring about the safe and effective deployment of autonomous vehicles.

Imagine a fleet of robo-taxis navigating the busy city streets. They round corners, pulling over occasionally to pick someone up or drop them off. It doesn’t matter if you’re young or old, able-bodied or infirm, anyone can hail a ride and get to their destination with ease. Meanwhile, a single person watches from a command center, monitoring the numerous cars.

When one gets into a complex situation, that same person links into the vehicle, assuming the role of driver to fix the issue at hand. Once the problem is solved, the operator releases the vehicle to continue on its merry way and enables themself to handle a different situation should it arise in any of the other cars.

Teleoperation is the technology that allows a human to remotely monitor and control an autonomous vehicle.

Currently, owing to all of the aforementioned issues with autonomy, there must be a safety driver seated behind the wheel of each AV. This obviously defeats the purpose of having a self-driving car if you need a “not” driver to sit in each car.

By enabling someone to link in from a remote location and solve the problems currently handled by a safety driver, we get one step closer to getting our own version of KITT. One person can act as safety driver for a number of vehicles, effectively hopping from situation to situation on an as-needed basis.

However, enabling teleoperation is more easily said than done. It may not require the gargantuan effort necessary to achieve vehicular autonomy, but it’s no easy task to allow a person to control a vehicle from hundreds of miles away.

Networking

Teleoperation relies on public cellular networks (LTE, 5G) and/or dedicated Wi-Fi networks to transfer information between the vehicle and the teleoperation center, with virtually no delay. The key to successful operation is continuous two-way data streaming, regardless of shifting network conditions.

Herein lies the challenge—neither LTE nor Wi-Fi are designed to support such high bandwidth and ultra-low latency communication from a moving vehicle, across a vast geography. Meanwhile, 5G will take years to roll out in entire cities and countries.

Furthermore, once a strong and continuous data connection between station and vehicle is established, inherent challenges remain in the act of teleoperating itself. How does one drive a vehicle safely when there are at least a few tens or hundreds of milliseconds difference between what’s happening and what the teleoperator sees?

Upon establishing the connection, the operations can be executed safely.

Nevertheless, there’s another major dragon to slay, one that bedevils every new technology—namely, hackers. It’s bad enough when a bank account is hacked and money is pilfered. The harm is far worse when a teleoperation station is hacked and the hacker seizes control of the AV, thereby sabotaging all of the safety and ingenuity that has brought us this far.

Lastly, there’s the matter of user experience (UX). The importance of UX goes far beyond how enjoyable or easy it is to use a teleoperation platform. Due to the different way a teleoperator may perceive the environment as opposed to an in-vehicle driver, efforts must be made to counterbalance this effect. It’s not enough to just give a video feed and issue commands. A teleoperator requires haptic feedback, situational awareness, and an overlay to utilize sensor systems to translate their information into actionable decisions.

Clearly many questions remain to be answered. Teleoperation, a seemingly simple technology, involves a multitude of technologies and systems to implement it correctly. This is the first article in a series of articles in which each potentially problematic aspect of teleoperation is addressed and answered.