Vision Processing: Its Role in the Future of Autonomous Driving
The following article was originally published in ee DesignNews Europe. It is reprinted here with permission.
The automotive industry is in the midst of massive transformation. Due to advances in technology, the next 10 years will bring more change than we have seen in the last 50. One of the biggest changes will be the move to semi-autonomous and autonomous vehicles, popularly known as the self-driving car. While the extent of the transformation is still to be determined the question on everyone’s mind is, will self-driving cars be the greatest disruptor to the automotive industry, or simply a crucial point in the on-going evolution of the car?
The road to self-driving cars will be gradual, with various stages and milestones that need to be reached. Currently, the timeline is assisted driving in 2016, automated driving in 2021 and fully automated cars by 2025.
Why Do We Need Self-Driving Cars?
A bigger question though is, why the move to self-driving cars? There are a number of key reasons why these cars need to be considered. In my opinion, the most crucial are: increasing levels of congestion and the rise in urbanisation, the need to reduce the number of road deaths and the associated costs, increasing pollution levels and changes in car ownership. Let’s look at each of these in more detail.
Congestion
Congestion is a huge drain on the economy. In the U.S. alone, people are spending over 43 hours per year in traffic jams, time which could certainly be better spent. In many major cities, the average speed of journeys is in decline and in some cities it is below that of a hundred years ago when people used the horse and cart. Also, research has shown that over 30% of driving in cities is simply to find a parking space, another process that technology can automate or that ‘un-owned’ driverless cars could alleviate. Tomorrow’s self-driving car could drop passengers off and then collect new passengers, reducing the number of vehicles on the road at any one time while also reducing the need for parking spaces.
Road Deaths
Today these are approximately 1.2 million road deaths worldwide. Depending on the country, the costs of cleaning up road accidents can be anything from 2% to 10% of GDP. The EU has already set a target for reducing fatalities throughout the continent by 50% by 2025. This will have a huge positive impact on GDP and will enable more money to be spent on upgrading road networks and installing smart infrastructures. Recent research by The Boston Consulting Group has suggested the technologies such as Advanced Driver Assistance Systems (ADAS) could save thousands of lives and about $251 billion each year if widely adopted.
Pollution
Air pollution is now amongst the highest causes of death throughout the world, especially in lower income economies. We only have to look at cities such as Beijing, Shanghai and Delhi to see what draconian methods are being used in order to limit pollution—the use of cars on alternate days and removing older less efficient cars through payments to owners, for example. Diesel cars are increasingly coming under threat because of the high level of particulate output causing respiratory problems. As a result, China is now putting a priority on electrification of vehicles to get control of the inner-city pollution levels.
A report by the Intelligent Transportation Society of America projects that autonomous and connected cars could achieve a 2% to 4% reduction in oil consumption and related greenhouse gas emissions each year over the next 10 years as these technologies infiltrate the market. In truth, the only way to truly decarbonize the transportation system is to move to a low-carbon fuel or completely battery-powered vehicles. However, until that day arrives, autonomous and connected cars will enable us to bring the levels of pollution down.
Ownership
Car ownership amongst city-based millennials is also dropping off and the cachet (perceived prestige) of car ownership is less prevalent, except in certain emerging markets. If car use is analysed, we can see that they are typically only used for less than one hour a day. That’s 23 hours of insurance, depreciation and cost that is sitting outside on the drive or in the office car park. New companies are entering the scene, such as Uber, BlaBlaCar and Lyft who are offering the kind of cost-per-mile that makes owning a car seem expensive.
All of these points are being addressed at government level and through self-driving cars; we can find viable solutions.
How Does the Self-Driving Car Help?
First, let’s look at some definitions of autonomy (below). Level Zero is where we are today, with some ADAS functions enabling Level 1 and Level 2 functionality, i.e. Automatic Emergency Braking. The move through to Levels 4 and 5 will be rapid, with 2021 being promoted by many car manufacturers as to when their first Level 4/Level 5 cars will be available for autonomous taxi fleets.
The levels [of autonomy] of self-driving vehicles.
It is expected that the public won’t be able to buy fully autonomous cars until about 2025. However, the automotive industry is a very fluid market and has some huge players with massive vested interests in self-driving cars, so the timings could be pulled forward.
The legislation that governments put in place, the timescales they agree and the acceptance of the self-driving car by the public will all be critical. Much education needs to be done to ensure the public is comfortable with the idea of self-driving cars and is willing to embrace the change.
One crucial step will be autonomous taxis. If the public accepts autonomous taxis and being driven by a machine, they are more likely to embrace the idea of owning and using a self-driving vehicle—ultimately, they need to see that self-driving cars are safe and viable.
When it comes to particular technologies that will support the move to autonomous driving a key one is a subset of artificial intelligence known as computer vision. The primary sensor for autonomous vehicles will be the forward-facing camera along with the LIDAR (Laser) and Radar systems. However, the greatest amount of extractable information is through the camera subsystems.
The Importance of Vision Processing in Self-Driving Cars
Vision processing has been around for many years, as has the application of convolutional neural networks (CNN). The latter are sets of algorithms that extract meaningful information from sensor input. CNNs are very computationally efficient at analysing a scene and are able to identify things as cars, people, animals, road signs, road junctions, kerb, road marking (objects), enabling them to ascertain the ‘relevant reality’ of the scene. This is all performed in real-time, so that once the sensing part has been done; the fusion/decision making can take place. This is where inputs from one or all of the sensors (LIDAR, RADAR, camera, IR) are assessed and decisions are taken as to the best course of action. For example, if a car suddenly brakes (outside the reaction time of the driver) the car would instantly verify the distance, sense the speed and apply the brakes faster than any human would be able to do.
The network part of the CNN is made up of many ‘layers’ (Fig. 2), which breaks down the problem of image recognition – from finding edges, corners and circles, to identifying them as road-signs, to reading what the road sign says. In order to prepare these networks, they have to be passed through a training phase that involves huge data sets of relevant images so that the CNN can learn to identify the various objects. This then sets the coefficient weights in the network from the learned datasets. This training aspect is normally a very compute intensive offline activity, which can take hours, if not days.
The network part of the CNN is made up of many "layers," which breaks down the problem of image recognition—from finding edges, corners and circles, to identifying them as road-signs, to reading what the road sign says.
Once the system goes live in a car there is the possibility to perform inference. This enables the network to read situations in real time and update the coefficients to provide great accuracy out on the road. Armed with its trained ‘knowledge’ on a vast number of signs in varying lighting conditions and angles, the CNN can extract meaningful information that is then fed to a decision-making element. This takes input from the LIDAR, Radar and camera and fuses them together to analyse the scene and make decisions as to how the car should react.
This set of decisions is then used to either display something on the car dashboard or actually intervene and perform a given action, such as braking or moving lanes.
Looking to the Future
The use of CNNs in automotive applications is going to rapidly increase to form the basis of camera-based autonomous driving and with it, establish a new driving paradigm. Fewer accidents, fewer fatalities, less pollution because of fewer and more efficient journeys, reduced congestion, car sharing and being able to pack cars in more tightly via vehicle to vehicle communication, all bode well for creating a brighter future for global society.