Challenges of Advancing Automotive Imaging

November 28, 2022

Blog

This technical article will examine how image sensors evolve to meet the challenge of autonomous driving with the latest technology to deliver industry-required performance at the price point they need.

As vehicles move from being entirely driver-controlled, through offering driver assistance to ultimately taking over the task of driving, they need to be able to sense their surroundings. While vehicles could use several different sensor modalities, image sensors are one of the most versatile and popular due to their unique ability to capture shapes, textures, and colors, and they're relatively low cost.

The road conditions become challenging with poor lighting and high temperatures degrading sensor performance. As such, image sensors must deliver excellent performance in all conditions for autonomous driving.

There are many challenges in deploying image sensors in the automotive environment. Light conditions may produce extreme contrast levels and glare from wet roads, while weather conditions including rain, fog, and snow can hamper visibility. Traffic lights, road signs, and vehicle headlights and taillights commonly use LED lighting. One great advantage of LED lighting is that it is very efficient; however, it is often pulsed. While human eyes do not see this, an image sensor will present this as a flickering image stream.  

One of the primary roles of automotive vision is to detect objects in the vehicle’s path. The further away an object can be seen, the longer the decision-making and reaction time available to the vehicle. That’s why high resolution and high image quality are required to discern objects at a distance.

Cost is crucial as vehicles deploy more image sensors throughout the system—not just for forward viewing, but also for providing a 360-degree view and monitoring the passenger compartment. Some cars have more than a dozen image cameras.

Moving from Assistance to Automation

The American Society of Automotive Engineers (SAE) defined a six-level model that charts the progression from vehicles with no intelligence to fully automated vehicles in all driving conditions, see Figure 1.

 

Figure 1: SAE Model of Increasing Automotive Automation Levels

Currently, many vehicles are capable of operating at Level 2, which includes the most basic control, such as correcting lane drift on the highway. The transition to Level 3 is significant as Level 3 has more automated control of the vehicle movement. Image sensors will need to offer 8 Megapixel (MP) resolution to support this—a four-fold increase on those typically used today. This will be sufficient for some autonomous operations in certain situations, such as on the highway. Moving forward to Level 4 and Level 5 operation, the resolution of image sensors will need to be much higher, thereby supporting autonomous operation in all situations.

Similarly, surround sensing and blind spot cameras also increase the resolution to 3 MP or even 8 MP, depending on their usage, and incorporate simultaneous LED flicker mitigation and high dynamic range (HDR) operation.

Also, non-Bayer filters have increasingly replaced Bayer color filter arrays (CFAs) to improve low light operation while still providing good color performance.

Pixel Sizes

Increasing the resolution of sensors leads to a significant rise in cost if pixel sizes remain the same – currently from 4.2 µm down to 3 µm. However, reducing the pixel size to 2.1 µm will result in a significantly reduced cost of an 8 MP sensor, meaning that an 8 MP sensor with 2.1 µm pixels will have a much smaller cost in comparison to an 8 MP sensor with 4.2 µm or 3 µm pixels. See Figure 2 for a relative cost comparison.

 

Figure 2: Reduced Pixel Size Offers Significant Cost Savings

One might assume that, as a result, there would be some trade-off in critical performance parameters such as low light performance, signal-to-noise ratio (SNR), or HDR. However, this is not the case. The low-light performance metrics (SNR1 and SNR3) of onsemi sensors with 3.75 µm, 3 µm, and 2.1 µm pixels are substantially similar. The SNR and HDR performance of the new onsemi 2.1 µm pixel image sensors is superior to 3 µm pixel image sensors. See Figure 3 for the relative comparison of SNR1 and SNR3 metrics.

 

Figure 3: Peduced Pixel Sizes Does Not Impact Low Light Performance

Furthermore, compared to 3.0 µm 3 MP or 5 MP sensors from another supplier, the onsemi 2.1 µm 8 MP sensor solution improves the detection distance at similar or lower costs.

 

Figure 4: 8.3 MP onsemi Sensor Detect Objects Farther Away Than Competing 3 µm Sensors

In the  challenging example of detecting a rock at night illuminated only by headlights, the 3 µm 3 MP and 5 MP sensors achieved detection distances of 125 m and 150 m correspondingly. In contrast, the onsemi sensor reached 170 m (Figure 4). This extra distance equates to more system reaction time, contributing significantly to increased safety.

Image Quality and Higher Automotive Temperatures

Changing the color filters from Bayer to RYYCy or RCCB and incorporating the high-quality HDR color pipeline like Clarity+ significantly improves sensor performance and image quality. The non-Bayer color filter patterns allow more photons to enter each pixel, improving low-light performance. This allows the sensor to ‘see’ better in challenging conditions while producing color-accurate raw captures processed into high-quality images.

SNR is an important parameter for all image sensors as this is related to the ability of the system to detect objects within images generated by the sensor. At high temperatures, a typical 3 µm split-diode sensor experiences a drop of SNR to around 20 dB. Noise is clearly visible at this level, and object detection is more difficult. A comparable onsemi sensor delivers SNR levels of more than 30 dB. At this level, noise is significantly less, and object detection is far easier, resulting in a much more visually pleasing image for viewing applications.

Temperature is always a challenge for image sensors and can significantly degrade image quality and performance. This is particularly true in automotive applications where sensors run at elevated junction temperatures of 80°C or higher for more than 80% of their lifetime—due to being placed in direct sunlight and designed in small enclosed spaces with other electronics that generate heat during operation.

Even at junction temperatures of 125°C, onsemi image sensors with 2.1 µm pixel size can achieve more than 25 dB SNR performance in mid-to-high light conditions, ensuring that accurate object detection is possible in all operating conditions.

A Modern 2.1 µm Automotive HDR LFM Image Sensor

The newest automotive image sensor from onsemi offers 3840 x 2160 (8.3 MP) resolution with latest-generation 2.1 µm super-exposure pixels. The sensor incorporates true LED flicker mitigation (LFM) pixel technology that generates up to 155 dB HDR images and exceeds 110 dB with flicker-free operation. HDR framerates of up to 60 fps are possible while reducing the framerate to 45 fps will increase HDR from 110 dB to more than 145 dB.

 

Figure 5: onsemi 2.1 µm Sensor (left) and Competitor’s 3 µm Sensor (right) Image Quality Comparison (cropped)

Regarding low light performance, the 2.1 µm sensor is comparable to or better in performance than the very best 3 µm pixel sensors. Figure 5 illustrates the HDR image quality difference of the 2.1 µm sensor compared to the competitor’s 3 µm sensor, highlighting a much better dynamic range and capturing better details and the traffic light’s true color. Transition SNRs are over 30 dB at junction temperatures (Tj) up to 100°C, and even at extreme temperatures (Tj=125°C), the SNRs exceed 25 dB. Under all conditions, the sensor produces sharp images with high color fidelity due, in part, to the range of Bayer and non-Bayer CFAs technology offered—RGGB, RCCB, RCCG, and RYYCy.

Summary

Advanced, autonomous vehicles are ever-more reliant on high-performance imagers that allow them to sense the environment around them. While increasing image sensor performance can be achieved, doing so without increasing cost is challenging.

onsemi imaging device designs have demonstrated that reducing pixel size allowed 8 MP sensors to be at a similar price as current 2 MP 4.2 µm sensors and 4-5 MP 3 µm sensors, without compromising low light performance SNRs and HDR. Furthermore, incorporating non-Bayer CFAs enhances the all-important low-light performance even more.

Temperature is always a challenge, with sensors housed in restricted spaces with heat-generating components inside and exposed to sunlight. onsemi sensors can offer excellent performance at temperatures up to 125°C, ensuring the capture of quality images under all operating conditions.

The next generation of image sensors is crucial in transitioning to safe and capable vehicles with higher autonomy.