Robert Day, Director of Autonomous Vehicles, Arm

Autonomous vehicle engineers face a mountain of challenges in designing Level 4 and 5 (driverless) vehicles. At this point, many of these challenges are even “chicken or the egg” obstacles, as more optimized hardware is required to execute software stacks that comprise hundreds of millions of lines of code, which themselves have not yet even been proven to behave sufficiently or deterministically in the countless number of scenarios that could be encountered on the road.

In this interview, Robert Day, director of autonomous vehicles in the Automotive & IoT Line of Business at Arm shares his perspectives on the biggest challenges in level 4/5 autonomous vehicle development citing research conducted by Forrester Consulting in their report, “Autonomous Vehicles: Prototype to Production report.” He continues to explain what these indicators signal for autonomous vehicle development over the next five to ten years.

What are the most pressing technical challenges facing Level 4/5 autonomous vehicle developers?

DAY: To help drill down into the particular areas that are causing the most concern, Forrester Consulting interviewed 54 global automotive industry experts for their latest Autonomous Vehicles: Prototype to Production report, sponsored by Arm, which looked at the technical path required to delivering safe and affordable autonomous vehicles at scale, and identified some of the biggest engineering challenges that automotive design teams are facing today as they prepare for production-ready Level 4/5 autonomous vehicles. 

When asked the question “Of the challenges you’re facing with your prototype AV system, which has been the most challenging to overcome?”, Forrester found that the number one challenge is the high cost of components.

This is not surprising as the prototypes being built today are primarily aimed at finding the right suite of sensors, combined with enough hardware performance to let the software algorithms make sense of the environment around the vehicle so that the sensors are able to “see” and make the correct decisions based on that information. That said, neither the sensor suite nor compute complex have been optimized for deployment yet, and therefore the costs of using “off-the-shelf” components are still too expensive, especially as prototype volumes are still extremely low and hence do not benefit from the economies of scale.

Since we're still in the autonomous vehicle prototyping phase, as you point out, the shortcomings of unoptimized hardware components and targets must be presenting more specific engineering challenges. What are those?

DAY: From a hardware technology perspective, companies are currently trying to get the right hardware in place to run the large software workloads needed to make the right decisions in an infinite number of scenarios that occur on our roads.

One of the individuals surveyed as part of the Forrester study stated, “Sensor technology doesn’t exist. Computing horsepower and algorithms don’t exist yet. The amount of human judgement in driving in boundless conditions is such that it will take over 10 years before a computer can make all of the judgments in a guaranteed safer way than a human in all situations.”

The sensors used in today’s traditional ADAS systems will require performance, resolution, and general functionality improvements to be fit-for-purpose for fully-autonomous vehicles. We’re seeing many autonomous vehicle companies now building their own sensors that fit the specific requirements they need as they move closer to deployment. We are also seeing more intelligence coming to the sensor edge to help reduce the central processing for sensor fusion and reduce the internal communication bandwidth requirements, and this is also forcing a change in sensor design.

Regarding central compute, the traditional processors used in high-end automotive applications such as IVI and ADAS do not currently have the performance required for higher levels of autonomy. This is currently pushing autonomous vehicle designers to either use computing from other markets such as the datacenter, which is clearly not suitable for full deployment, or place multiple automotive SoCs together to form autonomous drive solutions.

For AI specifically, the processing workloads that are required for the perception function of the autonomous drive system are currently using acceleration technology from other markets as well. Traditionally, one way we’ve seen this happen is in the use of gaming GPUs to get the performance required to process the vast amounts of data coming from the different camera, radar, and LiDar sensors around the vehicle. For this processing to become the right fit for autonomous vehicle workloads, SoC vendors will also need to provide automotive-specific ML solutions that match performance with efficiency and safety considerations.

The combine hardware/compute challenges cited in the Forrester report include speed and performance (15 percent), reliability (11 percent), power consumption (7 percent), weight (6 percent), cooling (4 percent), and size (2 percent) as the major obstacles. As mentioned, this can be attributed to the fact that the hardware being used is typically the same type of hardware used to power datacenters, so when the hardware optimization phase occurs using automotive-grade SoCs it should minimize many of these challenges, assuming that the performance requirements can still be met. As an example, we are seeing current Level 4 compute systems running at multi-KW power consumption, and this could be reduced to hundreds of watts by using automotive SoC platforms powered by, for example, high-performance Arm application processors.

What about software design? Does the software development paradigm change in these more highly integrated systems?

DAY: Almost a quarter of the respondents to the Forrester study cited software behavior as a major challenge either in unusual situations (13 percent), or in similar or same conditions (9 percent). This points to the fact that today’s prototypes still act as software test vehicles and that optimization of hardware cannot really begin until the software is behaving correctly in a very high percentage of all situations.

For a level 4 autonomous system, it is estimated that software will require hundreds of millions of lines of code to perform the level of thinking and processing that equates to a human driver. Clearly this is very different to traditional automotive systems that are generally reacting to driver requests, and where the focus is more on executing this function rapidly, accurately, and safely.

The autonomous vehicle software paradigm is closer to agile software design, where there is constant testing and iteration around many of the key functions of the autonomous system. It also means that open source software is typically being used as a platform to develop and test the autonomous software stacks, and in the Forrester study, 65 percent of respondents said that they were using open source software to some degree. As the systems get closer to true driverless deployments, commercial automotive-grade software will likely replace many of the open source components, especially where high performance or safety certifications are required.

How significant were safety and security in the report findings?

DAY: Interestingly enough, safety (4 percent) and security (7 percent) were quite low on the list of challenges, and I believe that these will increase dramatically once the aforementioned challenges have been overcome and as the systems get closer to actual driverless deployment. At the moment, the majority of prototypes still have at least one safety driver on all their runs, where that driver can react to any safety or security issue that confronts the vehicle.

Today, any completely driverless trials are typically in a very restricted Operational Design Domain (ODD) operating at low speeds and utilizing either very controlled or closed environments for the vehicles to operate in, thus restricting the damage any safety or security issue could cause. The removal of the safety driver and the increased ODDs that will make these vehicles deployable will now require the compute systems to have built-in safety, redundancy, and security so that the vehicle itself will be able to deal with any issue that arises without causing harm to either passengers or other road users.

New safety and security technologies from Arm, such as the Split-Lock feature on our most recent automotive enhanced (AE) application processors, combine the processing performance required for autonomous applications with high-integrity safety so that CPU clusters in an SoC can be configured in "split mode" for high performance, where two (or four) independent CPUs in the cluster can be used for diverse tasks and applications. Alternatively, in "lock mode," CPUs are in lock-step, creating one (or two) pairs of locked CPUs in a cluster for higher safety integrity applications. We believe this will aid these systems in identifying and dealing with fault conditions and bringing the vehicle to a safe state.

In my opinion, these and the previously mentioned challenges are all solvable, especially with  the functional safety, security, and compute advances that are on the horizon; combined, these will lead to safe and secure Level 4/5 autonomous vehicles.

You can read my point of view on some of the key report findings in one of my recent blogs.

What does an “automotive-qualified” system or component look like for an AV versus traditional automobiles?

DAY: A couple of factors can produce a different level of automotive qualification in AV designs. The first is operation design domains (ODDs), which will be present in all initial Level 4 deployments. This will dictate where and when the vehicles will operate.

A traditional car has to have a really broad ODD for it to be useful to drivers, as it needs to operate in any geographic region, under any weather conditions, and in any temperatures. This leads to the automotive qualification of the components, so they will operate consistently regardless of the external conditions.

As the ODD of autonomous vehicles can be very specific as far as the geofenced area that they will operate in (e.g., places where the weather is clement such as California and Arizona), they can also stipulate when the cars will operate (maybe only in sunny conditions above 32°F and below 90°F), and finally that these vehicles will always return to their garage overnight. This will help reduce the strict automotive qualification requirements around operational temperature range and is also why today’s prototypes often use electronic components from non-automotive sources.

Most of today’s prototypes are based on traditional vehicles, where sensors and compute are bolted onto a platform which already has automotive-qualified components. However, as AVs get closer to mass deployment, the form factors do not have to be based on – or even resemble – a traditional vehicle. This will also allow the AV developers to position components in new areas in the vehicle, such as in the cabin rather than the trunk or under the hood, and this alongside the ODD could lead to different requirements for automotive qualification.

Where and how do industry standards and regulations fit into this new, uncertain autonomous automotive landscape?

DAY: The ISO 26262 standard is certainly being considered for the certification of the safety-critical parts of autonomous vehicles, especially around the actuation phase where the car actually executes the driving commands that the autonomous system specifies. However, if you look at deployable Level 4 systems, where there is no fallback driver, it seems logical that the whole autonomous system should be certified in some way to be functionally safe, at the highest safety level, as any error in its decisions could cause harm to either the passengers or other road users.

Given that a Level 4 system will comprise many SoCs and hundreds of millions of lines of code, this would be a very difficult task to certify using current automotive safety methods. At Arm, we are introducing features in our different processing families, such as Split-Lock in our Cortex-A processors, that can enable high levels of functional safety from a hardware failure perspective without having to change the software running on those processors, but it is still assuming that the software is doing the right thing.

This is where government regulations will also play a role in the safe deployment of autonomous vehicles, and we’ve seen there have been some discussions around a standardized “driving test” for autonomous vehicles. However, given the different scenarios that the autonomous vehicles will face, and the different ODDs that they will operate in, it will be interesting to see how a standardized approach plays out.

In the USA, the National Highway Traffic Safety Administration (NHTSA) has published a “Preliminary Statement of Policy Concerning Automated Vehicles" that :

• Provides a description of developments in automated driving and explains the levels of automation defined by NHTSA.
• Provides an overview of NHTSA’s automated research program.
• Provides recommended principles that States may wish to apply as part of their considerations for driverless vehicle operation, especially with respect to testing and licensing.

The recommendations for testing of self-driving vehicles say that until such time that NHTSA has developed vehicle safety standards pertinent to self-driving technologies, States may want to ensure that self-driving test vehicles in their states adhere to certain basic principles. Many of these basic principles are geared towards limiting the ODD of the autonomous vehicles, and the testing assumes that most of the vehicles will have a driver that can retain control and that testing should make sure that the handover process is “Safe, Simple, and Timely.”

Around the world, different transportation bodies seem to be grappling with autonomous vehicles sharing our roads. This includes not just fully autonomous robotaxis but also higher levels of automation in our passenger vehicles (Level 3 autonomy), and it has led to some governments not approving the use of higher levels of autonomy on their roads even though the technology exists in some vehicles today.

Interestingly, I have just come across another initiative called openGENESIS, which is a collaboration platform within the Eclipse Foundation. Its mission is to provide knowledge, methods, and tools for the assessment of AI that is used within autonomous driving applications and ultimately provide both public and regulatory authorities with approaches to help them deal with the challenges of AI approval and certification.

All that said, there is still a lot of progress to be made for safety certification and testing regulations, but the fact that the industry and the governments are paying close attention to it gives us reassurance that when these vehicles take the streets, we can remain safe.

So, when can we realistically expect widespread deployment (or at least the ability to widely deploy) autonomous vehicles?

DAY: We are still a long way from achieving fully autonomous vehicles and realistically, we can expect that the industry will take approximately another 5-10 years before the first actual fully autonomous deployments, in which time automakers will need to adopt a number of changes.

I anticipate within 5 years we could see some initial driverless deployments, but another 10 years before we see those at scale. The majority, in both cases, are still likely to be operating in a fixed ODD as part of a ride-hailing service.

Robert Day is director of autonomous vehicles in the Automotive & IoT Line of Business at Arm. Interested parties can access Forrester Consulting's  Autonomous Vehicles: Prototype to Production report at pages.arm.com/automotive_forrester_report_2019.html.