AI-Enabled Yield Sensing and Forecasting for Agricultural Production


This project aims to develop a crop-generalizable, data-efficient, AI-enabled framework for agricultural yield sensing and forecasting. Specifically, we will:

(Objective 1) Design, build, and deploy yield monitoring sensors to accelerate ground-truth yield data collection

(Objective 2) Develop a synthetic data generation pipeline that couples a 3D crop model to a deep learning framework for yield prediction

(Objective 3) Create a deep learning framework that fuses multi-modal groundbased and aerial datastreams to directly predict yield from large-scale ground-truth data that incorporates human decisions

(Objectives 1 and 2). We will pursue these objectives for the three most economically valuable crops in California: strawberry, grape, and almond. We anticipate that substantial groundwork can be laid in achieving these objectives within year one.


Yield forecasting is vital for agricultural management, logistics preparation, and order fulfillment. Moreover, forecasting yield at high spatio-temporal resolution enables precision management activities, mechanization, and automation that can increase grower output and quality, while reducing resource inputs and waste. For example, early yield forecasting can enable automation of variable-rate application of inputs like water, fertilizer, and pesticides, which reduces expenses and environmental impact for growers, all while increasing productivity. However, yield forecasting is very challenging for specialty crops, spanning diverse environmental conditions, crop traits, and management strategies. By developing an AI-enabled framework for yield forecasting that generalizes across many specialty crops, we expect to transform US food systems. Moreover, achieving the objectives outlined here will lay the groundwork for efforts in subsequent years on agricultural automation, mechanization, and robotics.

By accelerating ground-truth data collection through new sensor development and synthetic data generation we will address AI challenges in data-efficiency. Further, we will develop novel multi-modal sensor fusion machine learning frameworks to improve yield forecasting in several economically and environmentally impactful crops.


During the last few decades, substantial research effort has gone into developing computer vision-based techniques related to crop yield forecasting. Researchers have tested a wide range of sensors to detect fruits and vegetables, including standard cameras, depth cameras, multi- and hyperspectral imagers, LiDAR, and microwave sensors. Further, various computer vision and machine learning approaches have been applied to this sensor data to detect, count, and measure fruits and vegetables. Despite such research effort, agricultural producers have not broadly adopted computer vision and machine learning-based yield estimation technologies; likely due to several limitations.

First, no large scale, high-resolution ground-truth yield datasets exist that encompass the broad range of environmental conditions, crop traits, and management strategies present across specialty crops. Not only does this prevent validation of how accurately any model predicts yield, it also strongly limits our ability to take full advantage of state-of-the-art deep learning models designed to extract complex patterns from very large datasets (i.e., millions of images). Thus, there is a need to dramatically accelerate the rate of ground-truth yield data collection.

Second, even if we successfully accelerate ground-truth yield data collection, the task of doing this in a way that covers the diversity of existing crop types, cultivars, environments, and management practices will require massive effort and cost. Previous work from other fields, such as autonomous driving and to a limited extent agriculture has demonstrated the potential for pre-training machine learning models on synthetic data generated using computer models. Despite substantial development in 3D crop modeling, no framework exists for synthetic training data generation in agricultural crops. Given its potential for substantially increasing data-efficiency, there is a need to develop a synthetic data generation pipeline that couples 3D crop modeling software to a deep learning framework for yield prediction that generalizes across many crop types, cultivars, environments, and management decision-making processes.

Third, despite substantial effort focused on computer vision-based fruit and vegetable detection, counting, and measurement in the last few decades, very few studies validate against on-the-ground yield values. When validation does occur, a relatively weak relationship is typically observed likely due to simultaneous variation in canopy density and fruit visibility (i.e. occlusion) along a row, which introduces substantial error into computer vision-based models that simply count or measure visible fruit. Given a large-scale ground-truth dataset, state-of-the-art computer vision models could potentially learn this complex interaction between visible fruit/vegetables and canopy occlusion to more accurately forecast yield. Further, yield forecasting involves prediction under an uncertain future subject to factors across various spatio-temporal scales. Previous work has shown that future yield is driven by ground-sensed variables (e.g., tree size/age and canopy coverage), climate variables (relative humidity, temperature) and remotely-sensed variables (canopy coverage, canopy greenness). Moreover, ground-based sensing provides a different spatial perspective than aerial sensing. Despite the unique and complementary information contained in ground-based and aerial imagery data, no deep learning modeling techniques exist that fuse these multi-modal sensor datastreams. Thus, there is a need to create a deep learning framework that fuses multi-modal ground-based and aerial datastreams to directly predict yield.


For Objective 1) in Y1 are the generation of groundtruth yield data covering 50,000 strawberry, 100,000 grapevine, and 25,000 almond plants at farms where we will also collect or obtain a suite of ground-based and aerial sensor data as well the timing and quantity of input use (see Obj. 3).

For Objective 2) in Y1 are generation of over 100,000 synthetic training data images for each strawberry, grape, and almond, pre-trained models for each validated against real-world ground-truth data from Obj. 1.

For Objective 3) in Y1 are the curation of training datasets of ground-based and aerial sensor data with corresponding ground-truth yield values, and an accompanying end-to-end deep learning training and evaluation pipeline.

Project Team


The projects are jointly conducted with partners through the AIFS Network:

UC Davis

B. Bailey


C. Diepenbrock


M. Earles


Y. Jin


I. Kisekka


Z. Kong


X. Liu


A. Pourreza


A. Smith


S. Vougioukas


UC Berkeley

A. C. Arias


S. Mäkiharju


K. Mosalam


M. Mueller


T. Zohdi