Unmanned aerial system‐based high‐throughput phenotyping for plant breeding

Unmanned aerial systems (UASs) have increased our capacity for collecting finer spatiotemporal resolution data that were previously unobtainable through conventional methods. The use of UAS for obtaining high‐throughput phenotyping (HTP) data in plant breeding programs has gained popularity in recent years. The integrity and quality of the raw data are essential for ensuring the accuracy of predictive tools and proper interpretation of the data. This paper summarizes the standard operation procedures for high‐quality UAS data collection, processing, and analysis for UAS‐based HTP (UAS‐HTP). Plant breeders can follow these procedures to implement a UAS‐HTP system in their germplasm enhancement and cultivar development programs.

height, leaf area loss from diseases, grain filling duration, and rate of senescence.However, obtaining these measurements can be a bottleneck in large breeding nurseries (Chawade et al., 2019;Song et al., 2021).Traditionally used plant phenotypic data collection procedures such as manual measurements or the use of handheld or ground platforms, are destructive, labor-intensive, time-consuming, or expensive.These types of constraints in data collection capability often lead to underrepresentative, inconsistent, and unreliable information due to the limited number of samples with both low spatial and temporal resolution and possible human errors.
An unmanned aerial vehicle (UAV), commonly known as a drone, is an aircraft without any human pilot on board.UAVs are a component of an unmanned aerial system (UAS), which includes a ground-based controller, a sensor, and a system of communication with the UAV (Chang et al., 2017).The UAVs' flight mission can operate under remote control by a human pilot or with an autonomous control system.The technological advancements in UASs and remote sensors have increased our capacity to collect finer spatial and temporal resolution data previously unobtainable through conventional manual measurement methods.A multidisciplinary team of scientists, working in computer science, geomatics engineering, mechanical engineering, agriculture engineering, agronomy, crop physiology, breeding, and extension, was formed at Texas A&M AgriLife to develop UAS-based high-throughput phenotyping (UAS-HTP) tools for crop breeding and precision crop management.The team developed and tested standardized protocols for UAS data collection, processing, and analysis to collect high-spatiotemporal phenotypic data on plant morphological traits such as canopy height (CH) (Chang et al., 2017), canopy cover (CC) (Ashapure, Jung, Chang, et al., 2019), canopy volume (CV) (Ashapure, Jung, Yeom, et al., 2019), and several spectral vegetation indices (Yeom et al., 2019).The obtained UAS-based phenotypic traits have been successfully used to (a) assess disease severity (Bhandari et al., 2020) and drought in wheat (Bhandari et al., 2021), (b) evaluate the effect of tillage management practices on cotton growth and development (Ashapure, Jung, Chang, et al., 2019), (c) select high yielding cotton genotypes (Jung et al., 2018), (d) monitor crop germination (Chen et al., 2018), (e) estimate plant population/stand count (Oh et al., 2020), (f) model crop growth and estimate yield of cotton (Ashapure et al., 2020) and tomato (Ashapure, Oh, Marconi, et al., 2019;Chang et al., 2021), and (g) characterize citrus greening disease (Chang et al., 2020).
This paper takes advantage of a series of lessons learned while developing effective UAS-HTP tools for crop breeding at Texas A&M AgriLife.

Core Ideas
• An essential protocol and procedure for unmanned aerial system (UAS) data collection for small plot research is developed.• The standardized semiautomated UAS image processing procedure is explained.• Potential applications of UAS data for highthroughput phenotyping of few traits is briefly described.
• Lesson 1: transdisciplinary collaborations between engineers, computer scientists, and biologists are essential.• Lesson 2: crop phenotype is an integrated response of genotype × environment interactions that can be measured by remote sensors.UAS is a practical and efficient tool to "read" these responses which can be quantified through further processing and data analysis.• Lesson 3: UAS can generate large datasets for depicting and documenting spatial variability and collect reliable and consistent phenotypic data.• Lesson 4: a data portal system is needed for data management, processing, analysis and communication with users and cooperators.
Realizing the need for a reliable source of information for UAS-HTP by several researchers in wheat breeding programs across the United States (www.triticeaecap.org/thewheatcap-uas-survey),we summarized the standard operation procedures for high-quality UAS data collection, processing, and analysis for the successful UAS-HTP.Although the proposed procedures are focused on wheat breeding programs in this paper, the standardized techniques are transferable to any other field crop (Jung et al., 2018).Specifically, we discuss the following components that breeders can follow to successfully implement a UAS-based HTP system in their breeding programs: 1. Basic protocols and procedures for UAS data collection (RGB and multispectral imagery data); 2. UAS data processing workflow to generate geospatial data products, such as orthomosaic images and digital surface models; 3. Phenotypic feature extraction procedure for multitemporal growth parameters including CH, CC, CV, and vegetation indices; 4. Data analysis and UAS-HTP implementation in a breeding program; and 5. Future directions.

Preplanning
Although In addition to government guidance, there can be additional requirements for research institutions, companies, and universities.It is always advised to check with the compliance office of one's organization.
In terms of equipment, the UAVs flown under Part 107 must be registered before operation.A pilot has to carry out the registration when operating the UAVs.Based on the requirements from FAA, UAVs must be available to the FAA for inspection or testing upon request, and a pilot must provide any associated records required to be kept under the rule.A pilot must also report any operation that results in serious injury, loss of consciousness, or prop-F I G U R E 1 Ground control points (GCPs) distribution in the field.UAS, unmanned aerial system erty damage of at least $500 to the FAA within 10 days (https://www.faa.gov/newsroom/small-unmanned-aircraftsystems-uas-regulations-part-107).Although the UAS mission for agriculture would be conducted in a lonesome region, if an operator is conducting business, flying on behalf of a company, university or institute, or flying for some other kind of nonrecreational purpose where another stakeholder might be involved, it may be necessary to purchase a liability drone insurance policy.
Efficient data collection begins with planning the nursery field layout.Global positioning system (GPS) guidance and autotrip capability on the planting tractor and planting equipment are vital in laying out a uniform plot boundary.Plots with consistent size and shape are necessary for automated data processing.Grouping germplasm or trials for UAS data collection together in the field will maximize the efficiency of data collection.Permanent, semipermanent, or temporary ground control points (GCPs) should be installed and surveyed by GPS devices in the field for precision georeferencing to conduct successful UAS-HTP over the whole cropping season.It is strongly recommended to distribute GCPs around and in the middle of the study area (Figure 1).As the accuracy of GPS measurements affects the quality of UAS-based products, the coordinates of all GCPs should be measured by differential GPS, which provides improved location accuracy.Most quadcopter batteries offer less than 20∼25 min of flight time; therefore, multiple batteries are required for collecting data in larger areas.The flight parameters, such as flight altitude and image overlap should be considered in preplanning to properly collect high-quality data according to various projects/programs objectives.Low-altitude flights (generally <50 m) with high image overlap (>80%) can collect highspatial resolution (generally a few centimeters), but the flight mission will require multiple battery changes restricted to a portion of the entire nursery.In contrast, high-altitude flights (generally ∼100 m) require less image overlap (60∼70%) and fewer battery changes, which means that high-altitude missions can cover larger areas, but lower spatial resolution images will be collected.Other factors to consider are obstructions such as trees or utility transmission lines, interference from other GPS guidance systems, and wi-fi/cellular data service for the aircraft and controller.

UAS campaign preparation and mission planning
To prepare the UAS campaign for agricultural fields, weather conditions and flight parameters should be carefully considered based on the targeted field.These factors can strongly affect the actual flight time, mainly the battery life of the platform.Although the battery life could be varied with specifications of the UAV platform and sensors, the battery can drain quicker to balance its position under high wind speed.In addition, the overheating battery and sensor may not work properly under high temperatures in the summer.In terms of flight parameters for imaging campaigns, there is a tradeoff among flight altitude, image overlap, and field size.With the same image overlap, a high flight altitude can cover a larger area, whereas lower flight altitude can cover a small area with high spatial image resolution.Therefore, an operator must find optimum weather and flight conditions when planning UAS missions (de Lima et al., 2021).Based on our experience with UAS missions under various conditions, we establish the recommended conditions for mission planning: Some of the commonly used software and applications to plan flight missions and control the drones for aerial mapping are Litchi (Vc Technology Ltd.), Pix4D capture (Pix4D), DroneDeploy (DroneDeploy), and DJI GS PRO (SZ DJI Technology Co.).We use the Pix4D capture app for DJI Phantom and Mavic series drones (SZ DJI Technology Co.) with an RGB camera.The software is free and supports most DJI platforms and flight parameters on the UAS models and camera specifications.DJI Matrice 100 (SZ DJI Technology Co.) with Slantrange 3P camera (SlantRange) was operated by DroneDeploy (DroneDeploy, San Francisco, CA, USA) with an additional plug-in to set up flight conditions for the multispectral camera.Based on previous experience and research on UAS data collection for breeding programs (Shi et al., 2016;Yeom et al., 2019), we develop flight specifications on image overlap, flight altitude, and flight pattern to design UAS missions.For example, the RGB platform DJI Phantom 4 Pro (SZ DJI Technology Co., Ltd., Shenzhen, China) equipped with a 2.54-cm (1-inch) 20 MP (megapixel) CMOS (Complementary Metal Oxide Semiconductor) sensor was flown at 20-30 m altitude with 80∼85% forward and side overlap following grid pattern (available in Pix4D capture) to obtain subcentimeters (0.5-1 cm/pixel) ground sampling distance orthomosaics (Bhandari et al., 2021;Yeom et al., 2018).As the multispectral camera has a narrower field of view and needs more time to cover the same area as the RGB sensor, a multispectral platform would fly over the study area at a relatively higher altitude with lower overlap (70∼75%) than the RGB platform.1.2-1.7 cm/pixel ground sampling distance orthomosaic images were obtained from DJI Matric 100 with a Slantrange 3P camera when flown at 30-35 m with a 70-75% overlap (Bhandari et al., 2021;Yeom et al., 2018).

Equipment
For our wheat breeding program in 2018-2021, DJI platforms equipped with RGB and multispectral sensors were adopted (Bhandari et al., 2020(Bhandari et al., , 2021)).The DJI Phantom 4 and Mavic 2 series equipped with RGB cameras and Slantrange 3P, a multispectral sensor mounted on DJI Matrice 100 were used for RGB and multispectral imagery data collection.Although the DJI Mavic, Matrice, Phantom, and Inspire series are some of the popular UAS platforms, platforms developed by Draganfly, autelrobotics (Bothell), and FREEFLY systems can be alternatives to DJI platforms.
In terms of multispectral sensors, radiometric calibration is an important component for converting pixel values to spectral reflectance to see accurate crop traits such as vegetation indices.Traditionally, radiometric calibration is conducted through the relationship between actual reflectance values and pixel values in images of various reflectance panels (Sapkota et al., 2020).Recently, multispectral cameras for UAVs provided two different methods of radiometric calibration: (a) using images, including a reflectance panel taken before and after flights (Chang et al., 2021) and (b) using upwardlight sensor to record illumination conditions (Change et al., 2020).
During the last four years of our work on UAS-HTP development, we found the following basic equipment features for smooth and efficient UAS data collection: (a) a stable and uniform UAS with the autonomous mode is needed to consistently collect high-quality UAS data over a cropping season; (b) UAS that can measure light conditions such as the Ambient Illumination Sensor on Slantrange sensor and Downwelling Light Sensor on MicaSense RedEdge sensor (AgEagle Aerial Systems Inc.) for F I G U R E 2 General unmanned aerial system (UAS) image processing pipeline to process raw images obtained from red, green, and blue (RGB) and multispectral sensors.NDVI, normalized vegetation index.Multi-spectral images may require radiometric calibration radiometric calibration.Reflectance panels can be used for radiometric calibration of multispectral images according to proper method by sensor type, and (c) UASs equipped with real-time kinematic positioning systems can be useful to avoid the need for manual georeferencing during image processing.

UAS data processing workflow (Level 0 to Level 2)
The overall UAS image processing pipeline that we follow is divided into three levels and is presented in Figure 2. The workflow starts with the collection of raw images (Level 0 data product from different sensors and platforms).The Level 0 data are then processed using the structure from motion (SfM) algorithm to generate Level 1 geospatial data products such as digital elevation models (DEM), orthomosaic images, and 3D point cloud data.Level 2 data products are then generated from the Level 1 data products.The Level 2 data products represent relevant biological crop features such as CH, CC, CV, and various vegetation indices from both RGB and multispectral images, such as normalized difference vegetation index (NDVI), soil adjusted vegetation index, and excessive greenness index (ExG).

2.5
Available software for generating geospatial data products (Level 1) Images collected from the UAS (Level 0) need to be further processed to generate merged data products (Level 1) such as orthomosaics and surface models.There are two main approaches: stitching and SfM methods.The stitching approach focuses on assembling L0 data to produce a high-resolution mosaic image.As stitching only focuses on merging neighboring images with minimal seam lines, it often cannot generate georeferenced mosaic images with high precision, and it does not correct geometrical errors induced by the lens mounted on a sensor.However, the SfM utilizes camera location (longitude, altitude, and height) and GCPs to perform a bundle block adjustment to generate the orthorectified mosaic, also known as orthomosaic images from the L0 data.In this process, accurate sensor exterior orientation parameters (EOPs), such as sensor exposure location, and sensor orientation and interior orientation parameters (IOPs), such as a principal point and distortion coefficients, are estimated and accurately georeferenced geospatial data products such as 3D point clouds, DEMs, and orthomosaic images are generated as outcomes of the SfM process.
Currently, both commercial and open-source SfM software packages are available.Pix4D, Agisoft Metashape, and DroneDeploy are the most popular commercial SfM software packages available on the market.Pix4D is available as a standalone (Pix4D Mapper) software package or on a cloud (Pix4D Cloud) platform.The Pix4D Mapper is only supported in the Windows operating system.The Agisoft Metashape is also available as a standalone software package or on a cloud (Agisoft Cloud) platform.However, the Metashape software is supported in Windows, Linux, and macOS platforms, thus providing more flexible deployment options.In addition, Metashape can be configured in a network processing mode without subscribing to the cloud option to speed up the SfM process when multiple computation nodes are available with a high-speed network (Chang, Jung, Yeom, et al., 2021).Otherwise, DroneDeploy is not available as a standalone software package, and it is only available on a cloud platform.
Although commercial SfM software packages dominate the UAS data processing market share, an open-source SfM software package is also available.OpenDroneMap (ODM) can perform SfM to generate accurately georeferenced data products, and it supports Windows, Linux, and macOS platforms.
Because of the open-source nature of this project, various projects are being developed using the ODM as a foundation.For example, WebODM is a project for developing a userfriendly web implementation of the ODM, ClusterODM is a project that enables parallel processing, and PyODM is a project for developing a Python software development kit for the ODM project.

General procedure of generating geospatial data products (Level 1 data)
The general procedure of SfM processing is similar in PIX4D, Metashape, and DroneDeploy.It consists of six steps: (a) Align photos, (b) Geo-referencing, (c) Build Dense Cloud, (d) Build DEM, (e) Build Orthomosaic, and 6) Export DEM/Orthomosaic.In the case of multispectral images, an additional step of radiometric calibration needs to be performed depending on the sensor type.For example, images obtained from Slantrange sensor are calibrated using their software known as SlantView (SlantRange, Inc.).An example of the data processing workflow for RGB images in Metashape is presented in Figure 2. Most UASs for mapping are equipped with GPS systems for measuring EOP during image capture.After importing raw images (Level 0), Metashape estimates six parameters, namely, the camera position (EOP and IOP), using aero-triangulation with tie points between individual images and bundle block adjustment based on the collinearity equations.The estimated IOP and EOP with sparse point clouds of the matched image are generated during the alignment photos step.
After image alignment, users can use the coordinates from the camera data for georeferencing, but the GCP coordinates are usually more accurate for precise georeferencing.The GCP locations can be specified manually on each image as markers.With the marker position in the images and GPS coordinates, the "Optimize Camera" function performs a full bundle adjustment procedure on the aligned photogrammetric block, simultaneously refining exterior and interior camera orientation parameters and triangulated tie point coordinates.The results of camera optimization can be evaluated with the help of geo-referencing error information to determine if more GCPs would be added or eliminated.Depth maps are constructed for the overlapping image pairs considering the estimated IOP and EOP to build a dense point cloud.The depth maps are transformed into the partial dense point cloud and then merged into a final dense point cloud with an additional noise filtering step applied in the overlapping regions.
The DEM can be rasterized from a dense point cloud with height values stored in every regular grid.DEM is used to build an orthomosaic image that is a combined image created by seamlessly merging the original images projected on the object surface and transformed to the selected projection.In Metashape, georeferenced DEM and orthomosaic images are exported as Level 1 products.Once the L1 data products are generated, phenotypic features (Level 2) are computed from the L1 data.Commercial SfM software packages, remote sensing/GIS software such as ENVI (L3Harris Technologies, Exelis Inc.), ERDAS (Hexagon Geospatial) ImageBreed (imagebreed.org),and ArcGIS (Esri) can be used to generate some L2 data products.

Generating multitemporal phenotypic features (Level 2 data products)
Canopy features such as CH, CC, and CV are generated from RGB-based orthomosaics elevation models.CH is generated from the DEM, which represents the surface elevation of objects on the ground.A DEM is acquired prior to crop emergence and considered as a digital terrain model.To estimate plant height for each flight date, a canopy height model (CHM) is generated by subtracting the digital terrain model from the DEM (Figure 3) (Hu & Lanzon, 2018).A classification algorithm is used to obtain CC from orthomosaic images (Ashapure, Oh, Marconi, et al., 2019;Bhandari et al., 2021).The classification algorithm uses RGB spectral bands of orthomosaic images and the red-green-blue vegetation index (Yeom et al., 2018) to generate a binary classification that separates canopy areas from noncanopy areas on the image (Figure 4).A plot boundary layer with plots/grids is overlaid on the classified image to calculate the percentage of green canopy cover within the grids and boundaries F I G U R E 4 Canopy cover estimation from orthomosaic images.A red, green, and blue (RGB) orthomosaic image was collected using the unmanned aerial system (UAS) platform, followed by the binary classification results of the orthomosaic image.White areas represent the canopy, and black areas represent the non-canopy pixels; the last image represents the plotwise estimated percent canopy cover F I G U R E 5 Crop canopy volume (CV) was calculated as the sum of pixels per plot classified as canopy times individual pixel height.We assume that anything that is zero in canopy height model(plant height > 0) when multiplied with pixel dimensions will be zero and be counted as zero canopy volume (Bhandari et al., 2021).Boundaries for each plot can be generated manually in the QGIS software (www.qgis.org)using the add polygon feature tool.The CV provides an estimate of plant biomass as a combination of canopy size and height.The CV for individual grids is calculated by multiplying the pixel dimension with the pixel height obtained from CHM.We assume that anything that is zero in CHM when multi-plied with pixel dimensions will be zero and counted as zero canopy volume (Figure 5) (Asahpure et al., 2019).

Data extraction procedure for multitemporal growth parameters (Level 3)
To extract phenotypic data for each plot, a boundary is delineated manually (Figure 6A), and then the plot boundaries are divided into grids if high resolution data extraction was necessary (Figure 6B).The boundary size is determined by the plot design for the study area.In recent years, GPS surveying of each plot in the early cropping season has been proposed to efficiently delineate suitable plot boundaries.Statistics such as average, standard deviation, and minimum/maximum of pixel values within each plot/grid boundary are extracted as representative crop parameters of each genotype.The grid boundary is also used to extract high-density phenotypic data to make it possible to conduct detailed analysis and to remove bias in the measurements by eliminating the effect of bare ground areas and the surrounding areas within a plot, which may also be affected by the lack of plant competition around the boundary area.

Data analysis and UAS-based HTP implementation in breeding programs
The obtained canopy features and vegetation indices can be used directly or indirectly to assess individual wheat cul-tivars or breeding lines (genotypes) for various phenotypic traits.Plant growth analyses can be performed using the multitemporal UAS-based canopy features to assess genotypes for stand establishment, early-season vigor, winter survival, spring green-up, stem elongation, grain filling duration, and senescence.Simple calculations using growth equations to obtain parameters such as growth rate and relative growth rate can be made (Hunt, 1983), or more complex conceptual models obtained by fitting the nonlinear regression models can be used to obtain several growth parameters and used to assess genotypes (Figure 7).Some of the commonly used functions for analyzing plant growth are the logistic function, Richard function, and Gompertz function (Yin et al., 2003).In addition, both RGB and multispectral-based indices can be used to assess the leaf area loss from diseases, grain filling duration, and the rate of senescence.Either a single phenotypic trait or a combination of multiple traits can be helpful to assess genotypes for the effect of drought on canopy growth or develop yield predictions.It is important to identify traits specific to the environment or crop conditions and assess heritability, multilocation/condition stability, and adaptability of UAS-based traits.

WEB-BASED DIGITAL PLATFORM FOR DATA SHARING, STORAGE, AND PROCESSING
To implement the UAS-HTP in any breeding program, it is important to establish a smooth data collection, data processing, data sharing, data analysis, and decision support platform.A web-based collaboration portal known as UASHUB (https://uashub.tamucc.edu/) was developed for visualization, analysis, and interpretation of Level 1 and Level 2 data and Level 3 data products.The UASHUB F I G U R E 7 An example of performing growth analysis using multitemporal canopy cover (%) obtained from the unmanned aerial system F I G U R E 8 Workflow to develop automated image processing pipeline for high-throughput phenotyping.UAS, unmanned aerial system also facilitates data sharing and collaboration among project scientists.The online data management features enable scientists to upload and download raw and processed geospatial data products to their workstations for further analysis.In addition, the UASHUB is equipped with several tools, including image clipping, plot boundary/grid generation, geotagging, crop growth analysis, and genotypes/entry comparisons, to help researchers perform analytical tasks swiftly and effectively.The current UAS image processing pipeline is semiautomated as we are using GCPs for georeferencing, the Agisoft Metashape software to generate orthomosaics, inhouse Python codes to generate plant attributes, and the QGIS software to extract phenotypic features.Texas A&M AgriLife is currently working to improve the pipeline to facilitate near real-time visualization, analyses, and decision-making (Figure 8).

Future directions
Unmanned aerial system-based high-throughput phenotyping has the potential to enable rapid assessments of large breeding nurseries across time and space by providing high spatiotemporal resolution measurements from small plots.This can increase our capacity to monitor and quantify field data obtained from multiple breeding nurseries and improve genetic gain.The ability to obtain precise phenotypic information can replace tedious and subjective ratings of breeding plots with highly dependable phenotypic information about a certain genotype.Considering the breeder's equation of genetic gain, the UAS-HTP system can improve genetic gain by improving selection intensity and selection accuracy directly and indirectly to all components of the equation.Developing automated UAS data processing pipeline with F I G U R E 9 Integrating multiple data platforms for high-throughput crop breeding user-friendly platforms for smooth data flow from data collection to decision-making will maximize the use of UAS-HTP and support real-time decisions.In addition, future research should be directed toward automating the integration of genomics, UAS-based phenomics, weather information, and artificial intelligence (Figure 9) to develop a decision support system for breeders to accelerate and improve cultivar selection.

A C K N O W L E D G M E N T S
This project was supported by the Agriculture and Food Research Initiative Competitive Grant 2022-68013-36439 (WheatCAP) from the USDA National Institute of Food and Agriculture.In addition, authors would like to acknowledge the Texas Wheat Producers Board (TWPB) funding support for collecting UAS data and testing the tools developed.

C O N F L I C T O F I N T E R E S T
The authors declare no conflict of interest.
(a) prepare sufficient, fully charged batteries including one or two extra; (b) set up the optimum flight altitude and overlap according to the required image resolution and field size; (c) conduct UAS missions under low wind speed (<15 mph) and clear sky; and (d) select a bright-colored platform and sensor, if possible, avoid overheating.

F
An example of the crop canopy height estimation procedure.Digital elevation model (DEM, ground elevation + crop height) images taken frequently are adjusted by the digital terrain model (DTM, ground elevation), resulting in the canopy height model (CHM, crop height estimate).Measurements are in meters