Uncrewed aerial vehicle radiometric calibration: A comparison of autoexposure and fixed‐exposure images

Remote sensing with uncrewed aerial vehicles (UAVs) is increasingly being used in agriculture to provide data on the physical characteristics of plants under field conditions. Data accuracy is critical for decision making with a high degree of confidence. In this work, we compared two multispectral camera calibration methods for image data collected with a UAV: (1) an autoexposure method that relies on a single calibration panel and a post hoc calibration, and (2) a fixed‐exposure system that uses three in‐field gray calibration panels using the empirical line calibration method. Both methods were compared to reflectance data from (a) four ground calibration targets measured with a spectroradiometer and (b) a single manned aircraft image calibrated with commercial calibration tarps. In a band‐by‐band comparison, the autoexposure method produced almost twice as much radiometric error on average compared with fixed exposure. Because remote sensing data are commonly converted to spectral indices, the calibration methods were also evaluated by calculating the visible atmospherically resistant index (VARI) and comparing the resulting data to the manned aircraft image. Similarly, the autoexposure method in this case produced twice the error of the fixed‐exposure method. The effect of the error was considered in a production agriculture context by simulating a remote sensing‐based prescription map for pesticide application in a cotton (Gossypium) field and calculating the number of mislabeled management zones. The simulation showed that the autoexposure method would be more costly to the farm because of its higher error, roughly $8.00/ha based on the assumptions made.

indicate numerous crop stresses such as spider mite infestation (Nansen et al., 2013), nitrogen deficiency, and water stress (Peñuelas et al., 1994).Spectral information is often used in vegetative indices (VIs) such as the normalized difference vegetation index (NDVI) (Zhang et al., 2016).Having accurate reflectance data is a key factor for generating actionable information from remote sensing (Brook & Dor, 2011).The error in calculating the relationship between digital number (DN) and reflectance is reduced by including corrections for known biases such as atmosphere, illumination, and camera properties (Smith & Milton, 1999).The conversion from DN to reflectance is a critical step in generating reliable VIs such as NDVI (Guyot & Gu, 1994).Accurate NDVI values must be generated from calibrated reflectance maps-that is, images converted from DN to reflectance-or they are not comparable to other NDVI values (Price, 1987) collected at different times or locations or by different sensors.To produce the most useful (i.e., low-error) data from remote sensing platforms, calibration must be performed to generate a relationship between the DNs and reflectance values.Radiometric calibration is used to convert the DN output from the sensor to target reflectance, which can then be compared across dates, locations, and sensors.
Traditionally, researchers have conducted remote sensing surveys for precision agriculture and high-throughput phenotyping using satellites and crewed aerial vehicles (CAVs).Each has its own applications and limitations.In attempts to address the limitations, uncrewed aerial vehicles (UAVs) have become prevalent as imaging platforms in research studies because of their unique potential.They come in a variety of size classifications (Anderson & Gaston, 2013) and are able to carry a wide variety of sensors (Sankaran et al., 2015), with most UAVs in agriculture falling under the small UAS classification, as provided by the U.S. Federal Aviation Administration (FAA, 2018).Their ability to carry sensors at a low altitude and relatively low speed presents opportunities for even higher spatial resolution.Moreover, in one study, researchers performed a comparison of image data from satellites, CAVs, and UAVs over a 2.5-ha vineyard in Italy.The UAV had a larger range of NDVI values (0.08 vs. 0.04 and 0.02 for CAVs and satellites, respectively) as well as a larger standard deviation than the other two platforms (Matese et al., 2015), suggesting that the UAV can capture more spectral variability due to its higher image resolution.The most common sensors used for both precision agriculture and plant breading applications have been multispectral cameras including standard red, green, and blue (RGB) visible light cameras (Yu et al., 2016), color infrared and other multispectral cameras (Wang et al., 2020;Zheng et al., 2018), and thermal cameras (Ludovisi et al., 2017).
As researchers and producers investigate ways to use UAVs in agriculture, reducing the error in reflectance data by improving calibration continues to be an important area of

Core Ideas
• The impact of two different UAV radiometric calibration techniques was compared.• Sensor settings and calibration methods on UAV images were compared.• Error comparison for autoexposure and fixedexposure images were collected with a UAV system.
investigation.Del Pozo et al. ( 2014) tested a vicarious radiometric calibration system for individual images and produced low errors of approximately 2%.A multispectral camera (Mini-MCA 6; Tetracam) operating with fixed-exposure settings was used to investigate low-cost calibration targets and methods.In this instance, vicarious calibration was defined as using ground-based (though man-made) targets with known (measured) reflectance to create the calibration equation that relates DN to reflectance.The calibration was only applied to individual images that had calibration targets present.
Selected areas inside each calibrated image were then compared to the measured reflectance to assess the calibration error.The low error produced with this system indicates that a vicarious method may be useful for UAV applications.Padro et al. used an octocopter with a five-band multispectral camera (RedEdge; Micasense) to collect images that were then stitched together into a mosaic.The UAV image mosaic was calibrated with the empirical line method (ELM) based on 17-color ethylene-vinyl acetate targets placed in each image.Exposure settings were not discussed in the article.These researchers found that the UAV data overestimated reflectance when compared to ground-based measurements, with reflectance values calculated with the near-infrared radiation (NIR) band being 15%-18% higher than the measured values (Padró et al., 2018).The goal of this work was to use UAVs to augment satellite data, but it is clear from the error level that care must be taken in how the sensor, calibration target, and correction methods are managed to produce accurate reflectance data.Iqbal et al. investigated a simplified calibration system for UAV images using five calibration targets that were placed in the field along with 20 pseudo-targets, which in this context referred to targets with unknown Lambertian properties.A multispectral camera (Mini-MCA 6; Tetracam) operating with fixed-exposure settings was flown on a multi-rotor UAV at solar noon.Each image had a pseudo-target visible, and each image was calibrated individually.A simple linear relationship was found between the measured reflectance of the calibration targets and the DN from the images.Each target appeared in several images; to improve the calibration, the DNs were collected from all images with the same target and used in creating the calibration equation.R 2 values from 0.97 to 0.99 were found for the camera's six spectral bands.The root mean squared error (RMSE) values across all bands ranged from 2.5% to 6.3% reflectance (Iqbal et al., 2018).
The calibration methods put forth in Padró et al. (2018) and Iqbal et al. (2018) all require a calibration target in each image.For large research fields or commercial agriculture, using a calibration target in each image would commonly require hundreds of targets and would be an impractical use of time and money.Mafanya et al. addressed this issue by investigating a calibration method for large-scale mapping at high resolution.The calibration relationship used was unconventional because it used a logarithmic equation (Wang & Myint, 2015), but the overall method is noteworthy.By placing calibration targets only in a single part of the survey area and then using information from that area post-mosaicking, the authors demonstrated that calibration points do not need to be in each image to generate consistent and reduced error across the mosaic (Mafanya et al., 2018).Laliberte et al. investigated the effect of image calibration pre-and post-mosaic using a six-band multispectral camera (Mini-MCA 6; Tetracam) and an RGB camera (Canon SD 900; Canon).Both cameras were operated with fixed-exposure and gain settings.Two ground-based calibration targets (2% and 85% reflectance) were used as reference for reflectance.The first method used the calibration information from a single image and applied the ELM to all images pre-mosaicking.The second method mosaicked uncalibrated images and then applied the ELM to the mosaic.Each method was compared with ground measurements made with a field spectroradiometer.Applying the calibration post-mosaic across all six bands resulted in a 2.2% reflectance RMSE, while applying the calibration to each image pre-mosaic produced a 5.6% reflectance RMSE (Laliberte et al., 2011).The authors noted that sensor corrections such as vignetting are performed during the mosaicking step and that the order of calibration steps is important for reducing error.
As remote sensing has progressed through satellites and CAVs to UAVs, the method of calibrating the sensors during image acquisition has been investigated as each new platform and sensor have been introduced.Past work shows that each image can be calibrated with in-field targets, resulting in low error, but doing so is unrealistically cumbersome and expensive for production farming (Han et al., 2019).ELM has been shown to work well in many instances where ground-based targets are used with fixed sensor settings and when the sensor is known to have a generally linear response.As UAVs become more common, and sensors are being designed specifically for UAV platforms, autoexposure camera settings are being commonly used because they are easier and faster to set up.As these systems and operating methods gain traction, they need to be tested to understand the impact of the sensor and acquisition method on the accuracy of the data.
By extension, the appropriateness and benefit of agronomic decisions based on the data depend on such testing.Thus, far little consideration and research have been published on the impact of autoexposure on radiometric calibration of UAV image mosaics.
The key hypothesis of this work is that using a sensor with fixed-exposure settings and ground-based reference targets for post-mosaic calibration (UAV-fixed) will result in smaller reflectance errors compared to the same sensor with automatically adjusted exposure settings and single-panel preflight calibration image (UAV-auto).Thus, the primary objective of this research was to collect UAV-based remote sensing images with the same sensor and both methods and then to compare both data sets to ground-based measurements and to a single CAV image calibrated with the standard ELM process to compare reflectance error between the two methods.A secondary objective was to simulate the effects of each calibration methods on practical agricultural decision making.Preliminary data were also presented as a conference preceding in 2018 (Bagnall et al., 2018) and is included and expounded upon here.

Site description
Flights were conducted between mid-August and late November of 2017 and involved two field locations at the Texas A&M AgriLife Research farm near College Station, Texas.Field 1 was a long and narrow field with rows running along the short side and total area of approximately 12.1 ha (Figure 1A).The field was bordered on one short side and one long side by perennial grass and row crops on the other two sides.It was broken up into several plots with row crops of soybeans (Glycine max), corn (Zea mays), wheat (Triticum aestivum L.), and a weed research plot.Field 2 covered a 31.5-hacotton (Gossypium) field with a perennial grass border on all sides (Figure 1B).The cotton, soybeans, and winter wheat were all harvested at typical dates for central Texas (cotton, August; soybeans, September; winter wheat, June), but the corn was part of the weed study and was not removed until mid-November.After harvest, each plot was tilled and left as bare soil.The land cover changed from predominantly row crops to predominantly bare soil through the course of this experiment.The use of two fields over the course of several months enabled the study to include broad variability in land cover and range of reflectance.

Equipment description
A Cessna 206 (Textron Aviation) was flown by the USDA Agricultural Research Service (USDA-ARS) Aerial Application Technology Research Unit as the CAV for this research.A Tuffwing mapper UAV (Tuffwing; Figure 2) was used for all UAV flights.It is a fixed-wing UAV with a wingspan of 1.2 m and takeoff weight of 2.2 kg fully loaded.This UAV uses a Pixhawk on-board flight controller, and Mission Planner (Ardupilot) was used for the mission planning software.
A single RedEdge camera (Micasense; Figure 2) was used for both UAV and CAV flights.It is a five-band multispectral sensor with discrete bands in the blue, green, red, red-edge, and NIR spectral regions (Table 1) and has a downwelling radiation sensor that allows the sensor to gather information about ambient light conditions.For this work, flights were conducted on cloud-free days, but care was taken to consider variation in illumination.The downwelling data were not included in the comparison between UAV and CAV data because the CAV data did not have the downwelling sensor information.
The RedEdge camera allows for two different exposure setting modes.The default mode is autoexposure, which adjusts the exposure time and gain settings for each band indepen- dently in real time.The second is fixed exposure, in which the user chooses from 15 exposure time settings and four gain settings, with each band being set independently and maintained for the duration of the flight.Geographic registration was important for mosaicking images properly so that locations within mosaics could be accurately compared between image collection methods and between flights.Ground control points (GCPs) served as geographic registration points for both UAV and CAV, and the particular GCPs used also served as radiometric calibration The ground control points were two levels with radiometric targets placed on both.The GPS information was collected from both front corners of the lower deck, shown here with black circles.
references for the fixed-exposure UAV flights.The GCPs had two height levels, one at 0.92 m above the ground and the other at 1.83 m (Figure 3).In this study, only the top level was needed for calibration purposes, with the additional lower level being designed for a separate experiment and used here for geographic correction of the mosaic.Each level of a GCP had three radiometric targets painted with diffusely reflecting paint in dark gray, medium gray, and light gray, along with a lid to protect the tiles from the weather and sunlight when not in use.The radiometric targets had dimensions of 60.9 × 60.9 cm, with a thickness of 0.64 cm, and were constructed from recycled rubber.The targets were painted black, dark gray, and light gray and then measured using a field spectroradiometer (PSR+ 3500; Spectral Evolution).The reflectance for each grayscale grouping was found to be approximately 6%, 25%, and 50%, respectively.For geographic registration, a Trimble R7 GPS base station was used along with a Trimble R8 rover to collect two GPS points on the lower front corner of each GCP (circled in Figure 3).Postprocessing kinematic correction was performed on the data after field collection, resulting in position data with an accuracy of ±2 cm in the xy-plane.
Four radiometric calibration tarps (Group 8 technology) measuring 8 × 8 m were also placed in the field prior to each flight for calibration of the CAV data and as groundtruth measurement targets for both UAV calibration methods.These tarps had reflectance of approximately 8%, 16%, 25%, and 40% (Figure 4).

Data collection
Flight missions and associated data collection were conducted on four dates: August 18, 2017; October 6, 2017; October 25, The autoexposure setting allows the camera to change the exposure time and sensor gain immediately before each image is acquired based on how much incident light is reflected from the scene.The reflected light can change drastically from image to image depending on the dominant ground cover in the scene, so the exposure and gain can change drastically as well.The fixed-exposure mode involves setting the exposure time and sensor gain preflight and maintaining those settings constant for the duration of that flight.Twenty preliminary short-flight tests were conducted to establish optimal exposure settings for the fixed-exposure mode.The UAV was flown over the calibration targets (6%, 25%, and 50% reflectance) at 120 m AGL on a cloud-free day near solar noon.The gain and exposure settings were systematically changed for each flight, and the DNs associated with the dark gray, medium gray, and light gray calibration targets were determined.The settings that maximized the dynamic range-that is, the difference in DNs between dark and light targets-without zeroing or saturating any pixels of interest were selected for each band.Based on these preliminary tests, exposure time settings of 0.44, 0.44, 0.44, 0.44, and 1.00 ms, and gain settings of 1×, 1×, 2×, 2×, and 2×, were selected for the blue, green, red, red-edge, and NIR bands, respectively.For the CAV flights, the camera had fixed-exposure settings of 0.33 ms for all bands and a gain of 1×.These settings were based on the experience of the USDA-ARS flight crew in using this sensor on a crewed aircraft traveling at comparable AGL and speeds to those used in this experiment.These preliminary tests were also used to determine if the response curve for the RedEdge camera was linear, and thus whether the ELM was an acceptable equation for calibration.
The GCPs were placed in the field at the beginning of the growing season in April 2017 and secured in place until the end of the season with four metal posts per GCP.Eight GCPs were placed in Field 1, with one at each corner and two spaced evenly along the long edge of the field.Nine GCPs were placed in Field 2 and spaced fairly evenly around the perimeter but out of the way of the center-pivot irrigation system.
On all four flight days, the CAV was flown first, and the ground-based radiometric target data collection was started while this flight was being performed.After the CAV flight was finished, the RedEdge camera was removed and mounted on the UAV, and the autoexposure flight was flown followed by the fixed-exposure mission.Each UAV flight lasted approximately 20 min.This protocol allowed the completion of all three flights of the same field inside the 2-h flight window centered on solar noon.At approximately the same time the CAV flight was beginning, a spectroradiometer was used to collect ground-truth measurements of the CAV calibration tarps and the UAV calibration targets.The PSR+ 3500 has a spectral range of 350-2500 nm with a nominal spectral resolution of 3 nm from 350 to 1000 nm wavelength, 8 nm at 1500 nm wavelength, and 6 nm at 2100 nm wavelength.It uses an active sensor and shields the object it is scanning, making it insensitive to solar illumination, and provides data output in reflectance.Each calibration target was measured at five points spread across the surface.The five readings of each target were then averaged to generate each target's reflectance value for the given flight.The reflectance was then clipped to match the red, green, blue, red-edge, and NIR bands based on the full width at half maximum reported in Table 1, providing a calibration reflectance value for each target in the spectral range matching each band.During the flights on October 25 and November 30, ground-based radiometric calibration readings were collected on all reflectance calibration targets with the spectroradiometer.For flight day August 18, the target measurements were collected on August 16.The measurements for October 6 were not collected on a day close to the fight; instead, the data for this flight day were interpolated from data collected on June 6, October 25, and November 30, assuming that any reflectance changes over this period would be linear.The solar zenith angle at the time of flight for each flight day is shown below in Table 2.
The CAV calibration tarps were also measured using the PSR+ 3500.For these calibration points, the tarps were measured in five places at the beginning of the season.These points were compared with data collected on these tarps the previous year from another project and found to be within the instrument error showing that these calibration targets were spectrally stable over the time frame we would be using them.The measured reflectance was averaged for each tarp across all five bands to create a ground-truth measurement against which each UAV calibration method was then compared.

Image preprocessing
Physical parameters of the RedEdge camera as reported by the manufacturer were stored as metadata in the image files and later used in image processing to provide corrections for dark current and vignetting for all images and all flights.The ELM was used to generate the relationships between DNs and reflectance for the whole-field CAV image.The reflectance was collected from the four calibration tarps on the day of flight based on the spectroradiometer data.For each band, a regression equation was calculated to relate calibration tarp reflectance to DN.The specific DNs used were selected by inspecting an area of interest (AOI) composed of the middle two thirds of the calibration tarp in the CAV image.The linear relationship found between measured reflectance and DN was then used in ENVI software to convert the pixel values of the image from DN to percent reflectance.This process was repeated for each image of each band, such that each CAV image had a unique calibration equation constructed from the calibration tarps present in that image.After calibration, the image was loaded into ArcGIS 10.2 software for georeferencing, which was performed by locating the GCPs in the image and applying the measured GPS coordinates to the appropriate corners of each GCP in the image.
For the autoexposure UAV flights, the camera manufacturer's recommended methods were followed to generate a reflectance mosaic.This method includes using a single white calibration panel of 15 × 15 cm provided by the camera manufacturer.An image of the calibration panel was taken before and after the flight by holding the aircraft over the panel and manually triggering an image while being careful about certain considerations such as light direction so as not to induce shadows.The raw flight data were loaded into Pix4D mapper software (Pix4D) along with the images of the calibration panel and the albedo value for each band.Pix4D uses the metadata imbedded in the images to create a rough layout of the mosaic.The user then applies the GPS information to the images containing the GCPs.The software then completes the mosaic and the calibration.The output was a geo-rectified, calibrated reflectance map of the entire field in which each pixel value was reported as percent reflectance.
The fixed-exposure flights required a combination of software due to the proprietary nature of the available systems.The fixed-exposure flight data were loaded into Photoscan software (Agisoft) to create an orthomosaic.The reflectance data collected from the three calibration targets located on the top levels of each GCP were loaded into a custom software program that performs radiometric calibration of the mosaicked image based on ELM.In this program, the measured reflectance of each calibration target was compared to the DNs collected from the image for that target.The program then calculated a linear equation to transform the image to a reflectance mosaic.The calibrated mosaic was then loaded into ArcGIS to apply GPS-measured position data to the GCPs and create a geo-rectified, calibrated reflectance map of the entire field.
The reason for using different mosaicking processes between autoexposure and fixed-exposure images is twofold.First, at the time this work was performed, Pix4D was the only program recommended by the camera manufacturer to handle the changing exposures generated by the autoexposure setting, so Pix4D was used for the autoexposure images.For fixed exposure, it was important to ensure that no unseen pixel smoothing algorithms were being applied by Pix4D.Photoscan offers more control over image processing procedures such as color smoothing.Thus, Photoscan was a more appropriate mosaicking software for the fixed-exposure data.Any relative error attributable to the difference in software is expected to be trivial.

Data analysis
Image reflectance data from the two UAV remote sensing calibration methods were compared on each flight date and across flight dates in three ways: (1) band-by-band comparison relative to the ground-based spectroradiometer measurements of the four radiometric calibration tarps placed in each field during the flight; (2) band-by-band comparison across 31 AOIs relative to the single calibrated image captured from the CAV; and (3) by the spectral index, visible atmospherically resistant index (VARI), across the 31 AOIs relative to the CAV VARI.

Comparing reflectance error of autoexposure versus fixed exposure
To extract pixel-level data from commonly located AOIs, the 40 reflectance mosaics (four image dates, two calibration methods, and five spectral bands) and 20 CAV reflectance images (four image dates and five spectral bands) were processed with custom code written in R (R Foundation for Statistical Computing).The code retrieved pixel reflectance values from 31 AOIs (Figure 5) defined by a set of coordinates in the image space.These AOIs were selected using stratified random sampling based on knowledge of each field so as to span the reflectance range and ensure each AOI was as homogeneous as possible.Each AOI used for the bandto-band comparison was approximately 10 × 10 m, with the mean pixel value of each AOI being calculated by the R code.The AOIs allowed comparison of reflectance at the same position in the UAV mosaics and the CAV images as well as comparisons of the same area across dates.
Plots of mean UAV reflectance versus mean CAV reflectance for each AOI were constructed for comparison of each band on each date and spanning all dates.Regression lines were determined, and coefficients of determination (R 2 ) and RMSE were calculated.The closer the UAV data matched the CAV data, which was considered ground truth, the more accurate the method was considered.A perfect match would follow a 1:1 line, thus the RMSE of the data from each method was calculated with respect to the 1:1 line.A paired T-test was also conducted to compare the mean differences of the residuals between the two UAV calibration methods.
Since the CAV calibration tarps were not used in the calibration of either UAV mosaic, we also analyzed the resulting estimated reflectance for these ground targets and compared them to the measured values.To accomplish this comparison, the four calibration tarps were identified in each mosaic and the mean reflectance was collected for all four flights.Each UAV calibration method was then compared with the known reflectance values, as measured with the PSR+ 3500, for the four calibration tarps (Figure 4).As mentioned previously, these tarps were used to calibrate the CAV data, but not to calibrate the UAV data.The mean reflectance, standard deviation of the reflectance across the four flights, the mean of the difference, the RMSE, and the bias relative to the 1:1 line were calculated for both UAV calibration methods.

2.7
Comparing spectral index error of autoexposure versus fixed exposure For the third comparison, the VARI index was calculated from the mean RGB pixel values for each of the 31 AOIs with the following equation.

VARI =
Green − Red Green + Red − Blue . (1) The mean AOI reflectance values in each of these bands were used in place of the corresponding pixel-level variables in the equation.The VARI for both UAV calibration methods was then plotted against VARI for the CAV, and the RMSE was calculated.This comparison enabled a better understand- ing of how reflectance error would affect a common spectral index like VARI.
As mentioned previously, most users of agricultural remote sensing data use VIs as principal metrics in their decisionmaking processes, so the difference in reflectance error between autoexposure and fixed exposure is important in spectral indices-that is, not just in a theoretical sense but also in a practical sense.These VIs are commonly classified into management zones for the application of inputs like nutrients, irrigation water, and pesticides.This process is well described in Willers et al. (2005), where NDVI maps were classified into four management zones for cotton plants and four other zones for all noncotton regions of a 218-ha field.
To demonstrate the effect that each calibration method's error could have on on-farm decision making, a Monte Carlo simulation was conducted that mirrors a particular actual field usage.This usage was reported in the literature and commercialized as a means of applying insecticide to a cotton crop at appropriate variable rates to control the insect, tarnished plant bug (Mckinion et al., 2009;Willers et al., 2005).These insects tend to thrive in high-vigor field areas, so high-vigor areas in the previous studies were identified with NDVI and assigned a higher rate of insecticide.In the current simulation, the full rate of insecticide would notionally be applied in high-vigor areas and zero insecticide in low-vigor areas.The VARI was used in the simulation to classify crop vigor instead of NDVI due to the limitations of the data collected in this study, specifically that NIR data had unacceptable error.
A simulated ground-truth matrix representing a field of cotton with roughly equivalent areas of high vigor and low vigor (an arbitrary central VARI value was selected to separate these classes) was created for the RGB bands.Each band had 10,000 data points assumed to be representative of a field area of reasonable size for precise variable-rate spray application.Each T A B L E 3 The crewed aerial vehicle mean percent reflectance value and standard deviation (SD) for the blue, green, and red bands, which were then used in the Monte Carlo simulation for creating the ground-truth simulation data.

Crewed aerial vehicle
Mean (% reflectance) SD Blue 5.14 6.14 Green 8.60 7.88 Red 11.34 9.46 matrix was generated with the rnorm function in R, which creates a vector of random numbers such that the values of the vector fit a normal distribution with a specified mean and standard deviation.For the ground-truth matrix, the mean and standard deviation were set to match those of the CAV data (Table 3).The same random number generator was also used to generate two error matrices with the means and standard deviations of the residuals for the fixed-exposure and autoexposure methods (Table 4).
The ground-truth and error matrices were combined to create two additional simulated data sets, one for each calibration method, of values in the RGB bands.These RGB data were then used to calculate VARI for ground truth, autoexposure, and fixed exposure on each of the 10,000 field units.These VARI data were then classified into high-vigor and low-vigor areas, and comparisons were made between the CAV classification and the autoexposure and fixed-exposure classifications to determine whether a particular field unit was correctly classified, underclassified (e.g., classified as low vigor when it was actually high), or overclassified (e.g., classified as high vigor when it was actually low).The Monte Carlo simulation was conducted 10,000 times resulting in a classification comparison of 100,000 points.

T A B L E 4
The mean of the residuals and the root mean squared error (RMSE) were calculated from the uncrewed aerial vehicle (UAV) autoexposure and fixed-exposure systems.These values were used to create the Monte Carlo simulation error data for each exposure method.The mean of residuals and the RMSE are both unit-less values but are calculated from the UAV reflectance data that have a unit of percent reflectance.

Autoexposure
Fixed exposure

Considering sources of error
When autoexposure is used, the exposure time and gain settings can conceivably be different for all adjacent images of a field.Unique pairs of exposure and gain settings require unique equations to convert DNs to reflectance.These equations are calculated in Pix4D automatically during the orthomosaicking process based on these settings as well as preflight calibration panel data, downwelling light sensor data (if available), and the manufacturer's camera corrections regarding vignetting, dark current, and so forth.Any error associated with these variations in conversion equations is a potential reason for extra error with the autoexposure method.An analysis of the exposure and gain variability with the autoexposure method was conducted to better understand how often the system changed the exposure and gain settings.The exposure time and gain were extracted from the metadata of each autoexposure raw image, and a bubble plot was created for each band and flight date so that the overall variability could be observed.The number of times the system changed exposure and gain was also found for each band and flight date.Furthermore, it is important to consider how the camera's dynamic range is affected between autoexposure and fixed exposure.The RedEdge camera used in this study encodes 12-bit data in a 16-bit DN format.Select raw images from two regions of Field 1 were also used to compare the dynamic range of DNs between the autoexposure and fixed-exposure methods.Two images each were selected from flight days October 6, 2017 and November 25, 2017 that contained AOI #A (Figure 6), and one image each was selected from flight days October 6, 2017 and October 25, 2017 that contained AOI #B (Figure 7).The two AOIs each covered an area of 26 × 66 m and were chosen to have consistent land cover of perennial grass, providing reflectance levels expected in agricultural crops, based on a knowledge of the field conditions at the time of flight.For agricultural surveys, it is ideal for the plant reflectance to occupy the largest portion of the dynamic range, because this provides high spectral resolution in the region of interest and the greatest amount of information about small changes in plant reflectance, which may relate to various stresses.Thus, a higher dynamic range in the reflectance of interest should minimize the error associated with dynamic range.The exposure time and gain of the selected images were determined, and the DNs from AOI #A and AOI #B were collected.The mean, minimum, and maximum DNs for the AOIs were determined and used to calculate the portion of the camera's dynamic range used by the autoexposure and fixed-exposure methods.The range of values from each AOI was divided by the possible dynamic range of the DNs to provide a percentage of the dynamic range related to green vegetation, and these percentages were compared between autoexposure and fixed exposure.

Comparing reflectance error of autoexposure versus fixed exposure
As mentioned in Section 2.3, the CAV calibration tarps were not used in the calibration of either the UAV-auto or UAVfixed calibration methods.These calibration tarps therefore were useful for ground-truth comparison.Figure 8 and Table 5 show a comparison of measured reflectance values for each band.Distinct clusters of data points correlate to reflectance regions associated with the four tarps, and in each cluster, there are four points for each UAV exposure method, which correspond to the four flight days.A 1:1 line is shown in each plot, indicating where perfect agreement between the UAV data and ground measurements would lie.While both calibration methods follow the 1:1 line fairly well, autoexposure tended to overestimate reflectance and also to have more scatter than fixed exposure.The scatter for both exposure methods increased as reflectance values increased.Table 5 also shows the standard deviation and the mean difference for each tarp and each band.In most cases, the fixed-exposure system has smaller standard deviation and smaller mean difference.Table 6 presents the RMSE and bias of this comparison showing that the autoexposure data error was higher in all bands than that of the fixed-exposure data, with the difference between errors ranging from 0.5% reflectance in the green band to 2.5% in the red band (Table 5).Table 6 also shows the bias relative to the 1:1 line for UAV reflectance versus measured tarp reflectance.The bias for the autoexposure method was more than double the fixed-exposure bias for each of the RGB bands.
The results of the second comparison of calibration methods are shown in Figure 9.The AOIs were classified according to ground cover type and denoted by a data point's color, while calibration method is denoted by a data point's shape.A 1:1 line is plotted on each graph, marking where perfect agreement between UAV and CAV data would be.A clear trend can be seen in the RGB bands for both autoexposure and fixedexposure methods.Autoexposure points are mostly above the 1:1 regression line, with increased scatter as the reflectance value increases.The fixed-exposure scatter is smaller and the data lie more closely along the 1:1 line, with little to no increase in variance at different reflectance values and across flight days.It is noteworthy that the red-edge and NIR bands F I G U R E 8 Reflectance for measured tarps reflectance plotted against the uncrewed aerial vehicle (UAV) mosaic tarps reflectance for red, green, and blue bands, for all four flight days.NIR, near-infrared radiation.

T A B L E 6
The root mean squared error (RMSE) and bias calculated from the comparison of the uncrewed aerial vehicle mosaic reflectance and the measured tarp reflectance.had unexpectedly large errors with both the autoexposure and fixed-exposure methods.For both methods, the error for the red-edge (RMSE auto = 5.75; fixed = 5.67) and NIR (RMSE auto = 9.06; fixed = 8.11) bands was roughly double that of RGB error (RMSE auto_avg = 3.45; fixed_avg = 1.97).Both the red-edge and NIR bands also showed greater variability between flight days in terms of AOI classification reflectance compared to the RGB bands (Figure 9).It is not clear why this was the case-it is most likely attributable to slightly malfunctioning detectors for those bands-but in any case, a decision was made to deemphasize the red-edge and NIR bands in subsequent analyses.
The RMSE for each UAV calibration method relative to the CAV data can be found in Table 6.The fixed method had significantly lower RMSE along with slightly higher coefficient of determination (R 2 ) for the RGB bands.The rededge and NIR bands had significantly higher error than the RGB bands, and the red-edge R 2 was actually higher for autoexposure than for fixed, but little weight was placed on this finding due to the higher error in red-edge data.For the RGB bands, the autoexposure RMSE was 2.6%-4.5% reflectance, nearly twice as high as for fixed exposure, which had RMSE of 1.7%-2.5% reflectance.It is worth noting that the rededge and NIR RMSEs were almost double those of the RGB bands, and autoexposure had higher RMSE than fixed exposure for these two bands as well.A paired T-test was run on the residuals of the UAV autoexposure and UAV fixedexposure data and showed a significant difference at the 0.01 confidence level (Table 7) in all spectral bands, indicating that the RMSE differences in the band-to-band comparisons  are significant, clearly establishing the superiority of the fixed-exposure method in data accuracy.

Comparing spectral index error of autoexposure versus fixed exposure
As mentioned previously, the VARI was calculated with the RGB bands, which had relatively low error and a distinct difference between autoexposure and fixed exposure.There-fore, the evaluation of VARI for practical usage of UAV data should present a strong case for why these differences in error are important.Figure 10 shows plots of the VARI for the UAV-auto (left) and UAV-fixed (right) compared to the CAV data.
Again, a 1:1 line is shown to indicate the position where perfect agreement between UAV and CAV VARI would lie.VARI is a ratio of combined reflectance and as such has no units and thus is reported simply as a number.It is clear from comparing the plots that autoexposure data had greater scatter, and at the high end of VARI values, an exponentially increasing error can be seen.There is also a large amount of variation seen in the UAV versus CAV VARI trends existed among flight days for autoexposure.Overall, autoexposure had a higher RMSE (0.44) than fixed exposure (RMSE = 0.23).Furthermore, while the CAV data had a VARI range of −0.62 to 0.59, autoexposure had a much higher range (−0.66 to 2.79), whereas fixed exposure also had a higher range (−0.69 to 1.15) but was more in line with the CAV data.The mean, SD, and MD found in Table 5 for the VARI also indicate a closer fit when using the fixed-exposure system.These results indicate that when calculating a vegetative index from reflectance values, the error levels in the original data are important, and fixed exposure produces results with much less error than autoexposure and is more consistent across flight days.
When the Monte Carlo simulation was conducted to enable comparison of reflectance-error effects on classifications for insecticide spray decisions, autoexposure produced correct classifications 77.0% of the time.Of the misclassifications, 12.5% of the time a lower-than-necessary spray rate was selected, and 10.5% of the time a higher-than-necessary spray rate was selected.The fixed-exposure method enabled correct classification 88.8% of the time.Of the misclassifications, 4.5% of the time a lower-than-necessary spray rate was selected, and 6.7% of the time a higher-than-necessary spray was selected.Essentially, the UAV-auto method enabled 129% improvement in accuracy over a random guess, which would have a one-in-three chance of correct classification.The UAV-fixed method, on the other hand, enabled 166% improvement over a random guess.
Figure 11 is color coded to show the proportion of units correctly classified (green) and of those misclassified, with red zones overclassified and blue zones underclassified.Figures 11A and 11B show the results of the simulation based on autoexposure and fixed-exposure error, respectively.
Assuming a 404.7-ha (1000-ac.)field, each of the 10,000 management units would be 0.047 ha (0.10 ac.) in size.A cost of $24.46/ha was assumed for the cost of full-rate insecticide to treat tarnished plant bugs with the Centric 40WG insecticide.The correctly classified units of the field were assumed to receive the correct spray rate.The overclassified units (red zones in Figure 12) were assumed to receive a full rate of the insecticide when it was not needed, resulting in overapplication.The underclassified units (blue zones in Figure 12) were assumed to receive no insecticide when the full rate was needed, resulting in lost yield.An average lint reduction of 6.5% due to tarnished plant bugs was assumed based on a high spray threshold of five tarnished plant bugs per plant (Musser et al., 2009).The 2019 average yield in Mississippi was 1246.4 kg/ha of cotton lint (Jackson, 2020).On this hypothetical 404.7-ha field, the autoexposure system would have resulted in spending $1039 extra for overspraying and would The Monte Carlo simulation is displayed graphically with the green squares representing a unit that is correctly classified and the red squares representing a unit that was misclassified into a higher management zone, while the blue squares represent a unit that was misclassified into a lower management zone due to the presents of error.Panel (A) shows the visible atmospheric resistance index (VARI) calculation when the error present in the autoexposure method is used, resulting in 67% correctly classified and 33% incorrectly classified.Panel (B) shows the VARI calculation using the fixed-exposure error, showing 82% correctly classified and 18% incorrectly classified.
have lost $5125 from not spraying enough due to the misclassification.Fixed exposure would have resulted in spending $666 extra due to overspraying and would have lost $1861 due to not spraying enough.Autoexposure thus resulted in a loss of 0.9% of the total expected revenue from the field, while fixed exposure resulted in a net loss of 0.38%.This translates to a cost difference of $8.96/ha or $3627 more for the entire field when autoexposure was used instead of fixed exposure, due to the increased misclassification of management zones.

Considering sources of error
When image exposure times and gain settings of the autoexposure calibration method were considered, the minimum number of changes observed during a flight was 79 out of 268 images (red-edge band, October 6, 2017) (Table 8).The maximum number of changes was 120 out of 267 (NIR band, November 30, 2017), almost half the images collected.Thus, between 29% and 44% of the images in any mosaic had different exposure time than other images in the mosaic.Table 9 shows the number of times the gain setting changed for each flight day and spectral band.The smallest number of changes was 4 out of 346 images (green band, August 18, 2017).The largest number was 61 out of 267 (blue band, November 30, 2017).Thus, between 1% and 22% of the images in a mosaic had a different gain setting than other images in the mosaic.
A bubble plot of exposure time versus gain setting is given in Figure 12, where the size of each bubble indicates the number of images with that exposure and gain setting.Each graph encompasses one flight day and one spectral band.Figure 12A shows the exposure and gain pairs from October 6, 2017, and the exposure range can be seen to be roughly from 0.5 to 2.0 ms, with two gain settings (1 and 2). Figure 12B shows three gain settings (1, 2, and 4) and exposure times of roughly 1.0-2.0ms.This trend of increased gain values and exposure times is found throughout the data as the date moves from earlier in the season to later in the season.Crop

T A B L E 9
The number of times the gain settings for the autoexposure system change for each flight day on each band.T A B L E 1 0 The range of the digital number for the chosen area of interest of perennial grass, shown as a percent of the dynamic range possible given the nature of the sensor.reflectance decreases as the season progresses, and the exposure times and gain settings also increase as autoexposure appears to adjust for low reflectance in the field by increasing the sensitivity of the system.When the dynamic ranges of the autoexposure and fixedexposure methods were compared with respect to green vegetation, the autoexposure DNs covered a larger percentage of available dynamic range than the fixed-exposure DNs (Table 10).The autoexposure DNs actually covered between 40% and 80% of the available dynamic range on average, while the fixed-exposure DNs covered between 10% and 40%.Because fixed-exposure and autoexposure data were produced with the same camera, any digitization error should have been consistent on a per-DN basis across both systems.Since the autoexposure DNs covered a larger amount of the dynamic range, the autoexposure data should have had more precision in digitization of object reflectance.Yet, the error was signif-icantly greater with autoexposure, so it appears likely that the majority of the error in the autoexposure data was introduced in the conversion from DN to reflectance.

Number of times
The difference in error between the calibration methods has several potential sources.One possible source of error with autoexposure comes from how the calibration panel is used.The process involves a single high-reflectance target, which means the reflectance values and DN relationships must be extrapolated from that single point, likely resulting in more error than interpolating between a low and high point as was done with fixed exposure.The autoexposure method collects an image of the calibration target under relatively unstructured conditions at ground level preflight, while the images that are collected during the flight mission are collected at altitude (120 m AGL in this research), meaning calibration is not performed under the same lighting circumstances as image collection during the mission, so parameters that are based on this calibration step likely have inherent error.
Furthermore, the fact that images can have many different exposure time and gain settings can potentially lead to two more sources of error.First, an additional mathematical step is required to relate the raw image parameters to the calibration image parameters before being converted from DN to reflectance.Second, the actual exposure and gain achieved at different settings could be inconsistent and different than the setting values.Another possible source of the error with autoexposure is the order of the data processing operations.Because each image can have a different exposure and gain, the images must be calibrated before the mosaic can be constructed.Laliberte et al. (2011) showed that even when using fixed exposure, images calibrated pre-mosaic resulted in higher radiometric error.
Fixed exposure, by contrast, has (in this research) three calibration targets of low, medium, and high reflectance that span the majority of the reflectance range of interest.Allowing interpolation between points to generate the relationship between reflectance and DN likely results in lesser error than extrapolation.Because the images all have the same parameter settings, the data processing order allows the mosaic to be constructed before the calibration step, again reducing error, as found by Laliberte et al. (2011).Once the mosaicking step has been completed, the calibration equation can be applied to all areas of the image mosaic with a single calibration equation for each band.

CONCLUSIONS
Analyzing and comparing two different UAV camera exposure settings and subsequent calibration methodsautoexposure with manufacturer-recommended calibration procedure versus fixed exposure with field-based targets and ELM calibration-showed that fixed exposure had consistently smaller reflectance error across all five spectral bands.For the RGB bands, fixed exposure typically produced approximately half the error of autoexposure.The difference in error was statistically significant at a confidence level of 0.01.The higher accuracy of fixed exposure was evident with image data from AOIs with different land cover types and with spectrometer data from calibration tarps in the field.When the RGB bands were used to calculate the VARI, the error in the VARI was again roughly twice as large with autoexposure as with fixed exposure.The impact of this error on cotton farm management decisions was illustrated with a simulation that suggested a meaningful improvement in revenue with fixed exposure versus autoexposure.The source of the greater error associated with autoexposure is not completely clear from this work, but it does not appear to be attributable to digitization error.More likely it results mainly from errors introduced in the conversion of DNs to reflectance.Autoexposure changes the image parameters often, requiring a different calibration equation for each unique set of image parameters.Because the image parameters are different, the autoexposure images must be calibrated pre-mosaic, which has been shown in other work to increase error.Autoexposure also generally calls for calibration with a single panel (requiring extrapolation) in lighting conditions that differ from those when the camera is at altitude.Some combination of these error sources likely drives the larger error and bias found in this study.By comparison, fixedexposure images are calibrated post-mosaic, requiring a single calibration equation for all images in each band, and it commonly uses interpolation between a high-and low-calibration point, simplifying the process and reducing the error and the bias.

A C K N O W L E D G M E N T S
The authors have nothing to acknowledge.

AU T H O R C O N T R I B U T I O N S
G. Cody Bagnall: Conceptualization; data curation; formal analysis; investigation; methodology; software; validation; visualization; writing-original draft.John Alex Thomasson: Conceptualization; funding acquisition; project administration; resources; writing-review and editing.Chenghai Yang: Resources; writing-review and editing.Tianyi Wang: Methodology; software; visualization; writingreview and editing.Xiongzhe Han: Data curation; methodology; resources; writing-review and editing.Chao Sima: Investigation; methodology; software; writing-review and editing.Anjin Chang: Data curation; software; writingreview and editing.

C O N F L I C T O F I N T E R E S T S T A T E M E N T
The authors declare no conflicts of interest.

F
The fields surveyed in this study.(A) Field 1 was long and narrow covering 12.1 ha with mixed land cover of crops, soil, and weed research plots.(B) Field 2 was 31.5 ha cotton field and had approximately one third of a center pivot irrigation system.WGS84 standard coordinate reference system was used to create these images.F I G U R E 2The uncrewed aerial vehicle (UAV) and sensor used to conduct these experiments.(A) The Tuffwing UAV.The orange panel can be removed to access the sensor payload.(B) The Micasense RedEdge camera was used for both UAV and crewed aerial vehicle flights.

F
Aerial calibration tarps used in radiometric calibration of crewed aircraft images.2017; and November 30, 2017.Flights performed on dates 2 through 4 were on Field 1, while the flight on the first date was conducted over Field 2. Each day had a clear sky, and all flights were conducted in a time window of ±1.0 h of solar noon.On each flight day, two UAV flights with the two different exposure methods were conducted along with one CAV flight.The UAV was flown at 120 m above ground level (AGL), producing a spatial resolution of approximately 7.6 cm.The CAV was flown at 1310 m AGL, giving 0.90 m spatial resolution, allowing the entire field to be captured in a single image.
Solar zenith angle measured in degrees for each of the four flight dates.

F
I G U R E 5 Thirty-one area of interest (AOI) locations across the field used for comparing uncrewed aerial vehicle and crewed aerial vehicle images.The AOIs are laid out using coordinates across images.(A) Field 1 used for the flights on October 6, 2017, October 25, 2017, and November 30, 2017.(B) Field 2 was used for the flight on August 18, 2017.WGS84 standard is the coordinate reference system used for these maps.

F
Raw image showing Area of Interest #A used to investigate the digital numbers of autoexposure.The white box shows the area of interest from which the digital numbers were collected.

F
The uncrewed aerial vehicle reflectance versus the crewed aerial vehicle reflectance with each column representing a flight day, while each row represents a specific spectral band.The color of the data points indicates the area of interest land cover classification and the data point shape indicates the exposure system used.T A B L E 7 The root mean squared error (RMSE) of the uncrewed aerial vehicle (UAV) exposure systems to the 1:1 line.The R 2 values are for a linear regression of the UAV onto the crewed aerial vehicle for each band.The paired T-test compares the autoexposure and fixed-exposure systems on an area of interest basis.

F
The visible atmospheric resistance index (VARI) data for the uncrewed aerial vehicle systems plotted against the VARI data from the crewed aerial vehicle system.(A) The autoexposure system.(B) The fixed-exposure system.

F
The bubble plot shows a distribution of exposure time and gain settings for the autoexposure system.The larger bubbles indicate more images sharing the same gain/exposure settings.(A) The bubble plot for the green band on October 6, 2017.(B) The bubble plot for the green band on November 30, 2017.

R 2 Paired T-test Band Auto Fixed Auto Fixed p-value Mean of the difference
Abbreviation: NIR, near-infrared radiation.

gain value change for the autoexposure
Abbreviation: NIR, near-infrared radiation.

Autoexposure system % of dynamic range used Fixed-exposure systems % of dynamic range used
Abbreviation: NIR, near-infrared radiation.