Success Story: An Autonomous Tool for the Farmer’s Next-Generation Toolbox

July 06, 2022

Story

A household name in the U.S., John Deere is a leading manufacturer of agricultural machinery and one of the first companies to commercially mass produce fully autonomous tractors.

After acquiring AI startup Blue River in 2017, John Deere began prototyping what would become the 8R Series Row Crop Tractors. Released earlier this year at CES, 8R Tractors are equipped with a chisel plow, GPS guidance system, and six pairs of stereo cameras that pass images through a deep neural network for 360º obstacle detection. The imaging system even allows differentiation between weeds and crops.

Figure 1. John Deere’s 8R 310 Tractor (Source: John Deere).

While mobile autonomous machines are a recent innovation in many industries, you might be surprised to learn that “farmers have actually been using self-driving for over 20 years” says Joe Leifer, Senior Product Manager for Autonomy for the Intelligent Solutions Group (ISG) at John Deere. And when you think about it, it makes sense. Most crops are planted in rows, which means the heavy equipment can be deployed in straight lines to perform tilling, planting, and harvesting. As you can imagine, this saves a lot of time humans would otherwise have to spend driving slowly up and down rows. And for farmers, time saved equals pennies earned.

However, the term “autonomous” has been used loosely in this context, as humans still have to manually turn tractors around once they reach the end of a row. The 8R changes all thanks to the AutoTrac™ subsystem, which includes features like Turn Automation and Implement Guidance that allow the 21'9", 28,000-lb pieces of mobile machinery to completely drive themselves while performing precision agriculture that improves crop yields.

Planting the Seeds of Precision Technology

To do more than just plow straight lines, a tractor – or any mobile system – must have a sense of its location within a given environment. That’s precisely what the 8R’s AutoTrac subsystem enables.

AutoTrac receives inputs from the 12-camera vision system and a StarFire 6000 GPS receiver that communicates with a GNSS satellite constellation for precision guidance within a digitally-fenced field and landform boundaries set by farmers prior to operation. During the configuration process, farmers can set the speed of the 253-horsepower tractors up to 24.9 mph, adjust steering settings, and manipulate other parameters through a mobile app that also displays things like tractor efficiency, agronomic performance, and fuel levels.

Beyond just U-turns, the AutoTrac perception suite helps 8R tractors follow curves along the edges of a field while performing whatever task they’ve been configured for. It even supports figure-eight turns, all while considering the tractor’s dimensions so it can make end-of-pass turns more precisely than even human drivers.

Figure 2. Turns made with AutoTrac Turn Automation (left) and turns driven manually by the operator (right). (Source: John Deere).

The Out-of-the-Cab Experience

As mentioned, much of the AutoTrac perception system’s intelligence is derived from deep neural network algorithms that John Deere trained against more than 50 million images of fields. The 360º vision system captures images and classifies pixels using the object detection neural net in 100 milliseconds.

The image pre-processing and AI model execution is performed on two onboard NVIDIA Jetson AGX Xavier modules.

Figure 3. NVIDIA® Jetson AGX Xavier™ (Source: NVIDIA)

“We have two NVIDIA Jetson GPUs that we use to do image processing,” Leifer explains. “So that’s our onboard embedded compute for our computer model. We’ve got those two GPUs hooked together through Ethernet and that’s how we’re communicating between the front camera pod and rear camera pod.”

After images are captured by the stereo camera system, they’re passed to the massively parallel Jetson AGX Xavier where 512 CUDA cores, two vision accelerators, and a collection of video encode and decode blocks process the image frames. After, 64 Tensor cores integrated alongside the CUDA technology in the Volta GPU architecture work with two deep learning accelerators to deliver up to 32 TOPS of performance in running the object detection algorithm against the processed images.

“Then ultimately, we use the existing onboard CAN bus on our tractor, so as we would potentially see something, we’re processing three frames per second with our perception system, and we’ll then request the stop. We will communicate with the tractor and bring it to a safe stop,” he continues.

After the Jetson AGX Xavier completes the inferencing workload, the output informs AutoTrac’s planning and acting engines. As you might expect, AutoTrac planning engines accept inputs from the onboard AI to plan paths through the environment, while acting engines send commands to control subsystems like steering and brakes to stop the tractor to prevent it from colliding with certain objects, terrain, or people; turn the vehicle so it stays within specific areas; or keep it progressing along pre-defined guidance lines.

At 3 fps, the tractor’s AutoTrac system can recognize potential hazards and intervene in more than enough time to protect the tractor and anything in its path.

“Combining the AI performance that you can get with our GPUs, as well as the ability to process the cameras in real-time using our image signal processor, we have a dedicated processor for doing computer vision functionality, as well as an Arm-based CPU,” says Amit Goel, Director of Product Management at NVIDIA. “All of that is on the same system and our software stack allows you to build out that entire pipeline in a really efficient way to reduce the latency without compromising the performance.”

Reap What You Plow

John Deere’s NVIDIA Jetson-powered AutoTrac system and its various features allow 8R Tractors to leverage precision agriculture technologies for farmers dealing with uncertain labor markets during the pandemic, changing climates, and increasing food and material demands.

The early success of the 8R Series has John Deere looking to expand their autonomous technology offerings. According to a recent press release, the company is currently evaluating whether selling, leasing, or offering an autonomy stack similar to the one used on the 8R line – both hardware and software – is the best path to making these capabilities available to existing tractors and tiller machines without any self-driving features today.

“The need for autonomy and the need for compute on the edge continues to grow,” says Goel. “They have to cover a lot of corner cases, scenarios, and they have to be a lot more reliable, which means they need a lot more compute.”

With the ability to prep 325 acres in 24 hours as a result of that compute, farmers will be lining up to plant their fields with the seeds of technology.