Development of a sweet pepper harvesting robot

This paper presents the development, testing and validation of SWEEPER, a robot for harvesting sweet pepper fruit in greenhouses. The robotic system includes a six degrees of freedom industrial arm equipped with a specially designed end effector, RGB‐D camera, high‐end computer with graphics processing unit, programmable logic controllers, other electronic equipment, and a small container to store harvested fruit. All is mounted on a cart that autonomously drives on pipe rails and concrete floor in the end‐user environment. The overall operation of the harvesting robot is described along with details of the algorithms for fruit detection and localization, grasp pose estimation, and motion control. The main contributions of this paper are the integrated system design and its validation and extensive field testing in a commercial greenhouse for different varieties and growing conditions. A total of 262 fruits were involved in a 4‐week long testing period. The average cycle time to harvest a fruit was 24 s. Logistics took approximately 50% of this time (7.8 s for discharge of fruit and 4.7 s for platform movements). Laboratory experiments have proven that the cycle time can be reduced to 15 s by running the robot manipulator at a higher speed. The harvest success rates were 61% for the best fit crop conditions and 18% in current crop conditions. This reveals the importance of finding the best fit crop conditions and crop varieties for successful robotic harvesting. The SWEEPER robot is the first sweet pepper harvesting robot to demonstrate this kind of performance in a commercial greenhouse.

harvesting success of prototypes has been around 66% (values ranging between 40% and 86%), with cycle times of 33 s per fruit (values ranging between 1 and 227 s). This performance is far below what is commercially viable, under the assumption that the market demands a machine that is highly accurate and fast. However, near-perfect harvest success and high speed are not necessarily prerequisites for economic viability, and also robots with lower performance can be of supportive value. The major problem is rather the gap between the required and achieved technological readiness level. In particular, earlier presented solutions most often lack practical robustness, failure control, and required postharvest logistics.
The goal of the European Union-funded research project SWEEPER 1 was to develop, test and validate a practical robotic harvesting solution for sweet peppers in real-world conditions. This paper describes various aspects of the project, from the scientific and technical challenges and solutions to the results of extensive field tests (262 fruits along 4 weeks) in a commercial greenhouse with a large variety of conditions (different growing rows, as well as different fruit varieties/crop conditions). The specific hypotheses tested were whether harvesting performance depend on the sweet pepper variety and crop modifications. Furthermore, the robot was integrated into the logistics of the greenhouse (equipped with a mobile platform which autonomously traversed along the rows).

| STATE OF THE ART
Agricultural robots are being developed for many agricultural tasks such as field cultivation, planting, harvesting, pruning, and spraying (Baxter, Cielniak, Hanheide, & From, 2018;Bechar & Vigneault, 2017;Blackmore, 2016;Duckett, Pearson, Blackmore, Grieve, & Smith, 2018;Edan, Han, & Kondo, 2009;van Henten, Bac, Hemming, & Edan, 2013). In practice, current working agricultural robotics systems are limited despite intensive R&D and many feasible technical solutions that have proven successful (Blackmore, 2016;Duckett et al., 2018). Fully robotized operations are not yet available, mostly due to the unstructured, dynamically changing, and undefined agricultural environment that demands a high level of sophistication and complicates the development process (Bac et al., 2014). Furthermore, each crop and task requires tailored designed developments (Bac et al., 2014). However, a number of companies have recently announced that they are close to a commercial product launch. Examples are robots for harvesting strawberries (Agrobot 2 , Octinion 3 ) and harvesting and deleafing of tomatoes (Privia 4 , Panasonic 5 ).
Given the complexity of harvesting that is highly variable in the environment and operating conditions (Kurtser & Edan, 2018) based on such few fruit samples can represent the robot performance in the complex crop environment (Bac et al., 2014).
Moreover, these test samples were probably evaluated at a specific hour on a day and did not include the weather variations throughout the day. Furthermore, even when a full system is tested, it usually does not include the full integration into the crop logistics and details on actual experimental conditions are not always fully reported.
Due to the large variation throughout and between seasons, it is important to conduct long-term tests in a variety of conditions and for different crop varieties (Bac et al., 2014;Edan & Miles, 1994) and mostly report testing conditions.

| ENVIRONMENT AND REQUIREMENTS
In Figure 1, a typical growing system of a commercial sweet pepper crop in a modern Dutch greenhouse is illustrated. Figure  Conditions in the greenhouse vary a lot during the day (e.g., illumination, temperature) and during the season (e.g., crop reaches a height of 4.5 m toward the end of the season).
The robot was designed to operate in a commercial greenhouse with the following requirements: • Ability to harvest ripe (yellow colored) fruits in the greenhouse.
• Should not damage the harvested fruit body or fruit stem (peduncle).
• Should not damage other fruits, plants, or guide wires.
• Must interface with automatic guided vehicle logistics.

| SYSTEM DESIGN AND OPERATION
On the basis of the requirements outlined in the previous section, an initial robot design was proposed and developed. Field tests of partial solutions called for alternative solutions, which were then developed and tested in an iterative manner. In the remaining part of this section, the final design is described. The overall operation of the robot and algorithms for fruit detection, grasp pose estimation, and motion control are detailed.

| Overall design
An overview of the robotic system is shown in Figure 3. The system was composed of a standard six degrees of freedom (DOF) industrial robot arm (Fanuc LR Mate 200iD), a custom designed end effector (patent No PCT/NL2019/050396-pending approval), a high-end main computer with graphics processing unit (GPU), programmable logic controllers (PLCs), sensors, other electronic equipment, and a small container to store harvested fruit. One Arduino-based PLC controlled the cart operations (motion along the row and cart elevation); another PLC controlled the low-level functions of the end effector. All equipment was mounted on a cart that could drive on the pipe rail and also on the concrete floor. The cart is a modification of the commercially available Qii-Drive Pepper AGV pepper harvest cart (Bogaerts Greenhouse Logistics, Belgium). This cart has a scissor lift that can elevate the platform with the robot manipulator to the harvest zone. Both cart propulsion and scissor lift as well as the camera system and manipulator were controlled by the system's main computer. The whole system was powered by 24 V rechargeable batteries.

| End effector design
The end effector is the tool mounted at the last link of the robot arm.
The purpose of the end effector in this robot was to facilitate safe removal of the fruit from the plant and also to hold it during transportation to a container.
F I G U R E 1 Schematic top and side view of one aisle in a sweet pepper greenhouse. In this situation, each plant in a row of plants has three stems. The growing architecture in the "Y" shape results in a double row of stems per plant row as seen from the aisle. This is also known as the "double row" system. In the "single row system" each plant in the row would have only one single stem [Color figure can be viewed at wileyonlinelibrary.com] As detailed in Figure 4, the end effector comprises a housing for the RGB-D camera and is fitted with custom made LED lighting fixtures, described in Section 4.2.3. The last joint of the robot arm is connected to this housing by a robot arm mount. Furthermore, on top of the housing, a plant stem fixation mechanism is placed with a vibrating knife configured to cut the fruit peduncle driven by an electrical motor. In the first contact position with the plant stem, the cutting edge of the knife is positioned right above the peduncle of a fruit while it is shielded by the stem fixation mechanism to prevent plant damage by the knife. When the cutting action is initiated, by moving the end effector downwards, the fixation mechanism is lifted, thereby providing the knife room to cut downwards through the peduncle. After the peduncle is cut, while the end effector is approaching the target. Figure 5 shows a photo of the final end effector, including the fingers.

| Software design
The software developed in the project was crucial to achieve the required functionality. Most software was written in C++ and Python, using ROS Indigo running on Ubuntu 14.04. Most programs were installed on the main computer mounted on the platform. Some functionality, such as image-based obstacle detection, was run on the main computer's GPU to speed up operation.

| Overall operation
The autonomous operation of the robot was defined in a flowchart, describing how sequencing of image-based fruit localization and motion control of platform, robot arm, and end effector was combined to achieve successful harvesting of fruit (see Figure 6).
For better readability, some details and most of the error handling are left out in the figure. The basic operation was as follows: The platform autonomously moves along the plant row and raises itself to the desired working height. The manipulator then moves in a predefined pattern to scan for fruits. As soon as a fruit is detected (see Section 4.2.3), the manipulator moves closer to the fruit, and the location of the stem relative to the fruit is determined (see Section 4.2.4). The end effector is then rotated such that the stem is behind the fruit and then moves towards the fruit using visual servoing. When the fruit is reached, the vibrating knife on the end effector is actuated, and the end effector is moved down to cut the peduncle of the fruit. The fruit is caught in the fingers mounted on the end effector, and the manipulator is moved to the fruit container, where the fruit is dropped. The manipulator is then brought back to where it first saw the fruit, and the scanning procedure and harvesting continue until the scanning pattern is completed. The platform is then lowered to the second harvesting level where harvesting proceeds. When all fruits have been harvested from the current platform position, the platform moves along the rails to the next plant, and the harvesting procedure is repeated. If a fruit harvesting operation fails, a maximum of two additional attempts are made (not shown in flowchart).
If the harvesting operation still fails, the fruit is left. When the platform has reached the end of the plant row, it automatically moves back to the starting point of the row.

| High-level control
To execute the flowchart described in the previous section, a state machine-based framework for agricultural and forestry robotics (Hellström & Ringdahl, 2013) was implemented. Each state is usually connected to one or several ROS nodes that perform specific actions such as moving the manipulator or detecting fruits in an image.

| Fruit detection
A central function in the system is detection of fruits to be harvested.
For successful operation, the three-dimensional (3D) location of each fruit must be determined with a high accuracy. The chosen solution is based on an RGB-D camera that simultaneously reports color and depth information. The camera (Fotonic F80; Fotonic Inc., Sweden; Figure 5) employs a time-of-flight technology for depth measurements and uses a single sensor for measuring both RGB and depth, thus allowing fully registered channels. It was proven very robust to varying outdoor illumination conditions with a signal-to-noise ratio well suited for agricultural applications in greenhouses . Using this camera and a custom built LED-based illumination system, RGB images of the plant were acquired from both overview distance and close range. Since detectability was dependent on the imaging acquisition distance and the sensing system tilt (Hemming, Ruizendaal, Hofstee, & van Henten, 2014;Kurtser & Edan, 2018), a constant tilt upwards of the sensors and illumination rig was applied.
To facilitate high frame-rate operation, a shape and color-based detection algorithm was implemented (Arad et al., 2019). Color constancy is a key factor in the performance of such algorithms and was achieved by applying a filter to obtain homogeneous light distribution in the image. Before operation, the algorithm was calibrated over several sample images (2-3 images), and the user was prompted to select the target fruit. Using this input, color thresholds for detection were automatically set on the basis of the statistics of the selected areas. Calibration needed to be performed only once for each target pepper variety. Several other approaches to automatically determine thresholds were developed and evaluated, see for instance Ostovar, Ringdahl, and Hellström (2018). Once calibrated, the algorithm scanned each acquired image for regions matching the target color thresholds. Detected regions were then further refined by removing detections exceeding predefined minimum/maximum sizes. A Pepperness measure P was also calculated, for each of the detected peppers according to Equation (1) where A P is the detected area of the pepper in pixels, C is the camera's pixel to cm coefficient, Z is the distance of the pepper (cm), P W and P H are standard pepper width and height (controllable parameters in cm). "Holes" in the detected regions were filled using mathematical morphology operations. Finally, depth information from the camera was used to compute the volume of the detected regions. Detected regions were compared with the volume of an average pepper, and the relative size was reported as part of the detection (see Figure 7). This information was then used to further prune false detections, avoid nonharvestable fruit clusters, and define harvest priorities. The exact 3D location of the point of mass was calculated using the depth information extracted from the detected region and a standard procedure of pixel-to-world transformation of the region (e.g., Wei & Ma, 1994;Zhang & Pless, 2004). Given the subsets of regions that were classified as peppers to be harvested, a methodology for harvesting sequencing was defined.
Several options for harvesting sequencing were reviewed, including optimization of harvesting distance (Edan, Flash, Peiper, Shmulevich, & Sarig, 1991;Zion et al., 2014). The heuristic that was implemented was harvesting in a top-down sequence to facilitate harvesting of clusters, followed by closest to the center to facilitate proper visual servoing that keeps the target in the middle of the image.

| Stem detection
As described in Section 4.2.1, the manipulator rotates around the stem such that the stem is behind the fruit as seen from the camera.
To enable this, a method to estimate at what angle the fruit was positioned around the stem was developed. First the images were semantically segmented using a deep learning approach (Barth, IJsselmuiden, Hemming, & Van Henten, 2017.

| Motion control
Once a fruit had been detected as described above, the robot arm should move the end effector to a position from which the fruit could be harvested while avoiding obstacles. This is a nontrivial task for several reasons. In general, the robot arm is moved by changing the angles of the six motor-controlled joints in the arm. Finding joint angles for a given goal position is known as the inverse kinematics problem, which in general has several possible solutions. The problem is further complicated by the fact that not only the position, but also the orientation of the end effector must be controlled. To move the end effector along a given path toward the fruit, a sequence of solutions of the inverse kinematics problem has to be computed.
Mostly, motion control of the robot arm was implemented using

| GREENHOUSE TESTING AND PERFORMANCE EVALUATION
The finalized robotic system 6 was evaluated in a commercial sweet pepper production greenhouse in IJsselmuiden, the Netherlands, during June 2018. The procedure for testing and evaluation of results is described in the remaining of this section.

| Testing procedure
The greenhouse was set up as described in Section 3. Two rows were selected for robotic harvesting experiments, with each comprising 100 stems of the sweet pepper variety Sardinero, followed by 100 stems of the variety Gialte. In this way, the hypothesis that Sardinero would be more suitable for robotic harvesting than Gialte could be tested. The motivation for this hypothesis is that a longer peduncle (fruit stem) should allow more free space to position the end effector

| 1033
and that a long peduncle should reduce the risk of damaging the fruit by cutting into the fruit body.
As sweet pepper commercially is harvested before the complete fruit body turns from green to yellow, each fruit with at least a 5 cm 2 big yellow colored patch was considered ripe and was manually labeled with a fruit number. Unripe fruits were not labeled and not selected to be harvested by the robot.
Half of the plant stems of both varieties were modified to simulate idealized circumstances, with low occlusion and no clusters interfering with the end effector's reachability. The proposed modifications were introduced after consulting with commercial growers and agronomy experts. The modification was done by removing leafs that occluded a labeled fruit from a frontal perspective. Neighboring fruits that might interfere with a harvest attempt (e.g., clusters) were also removed. Figure 9 illustrates this modification. It was hypothesized that performance for the modified crop would be significantly higher than for the commercial (unmodified) crop.
The current double row growing architecture (as shown in Figure 1) does not allow robotic harvesting from both sides of the stem (front and back) since fruits may be visually and physically occluded by the row of stems. To analyze the potential of the robot in a best fit cropping system (where the robot can reach the stem from both sides), we also calculated the robot performance by only considering fruits that were in front of the row. In the following, this cropping system is referred to as the single row system. For this, only fruits positioned in front of the row (positioned over −90°or +90°) were taken into account. The resulting percentages are with regard to only these fruits.
The actual testing procedure started by positioning the robot at the beginning of the plant row with the selected plant stems. The robot was then started to perform the autonomous harvest cycle as described in Section 4.2.1. After the robot finished harvesting fruits from all stems, the remaining ripe fruits that the robot was not able to harvest were manually harvested. In this period of the harvesting season, 4-5 days were the typical period between two consecutive manual harvest actions. Therefore, the robotic harvesting experiment was then paused for 5 days to let the next fruit setting ripen, before starting a new harvest trial at the same plant stems.

| Performance measures
The harvest cycle consists of a sequence of substeps that all have to work for a fruit to be successfully harvested. To improve understanding and analysis of the test results, success of the following five substeps were logged: fruit detected, fruit reached, fruit cut, fruit caught, and fruit in container. Results for all combinations of growing conditions (pepper variety, single/double row, modified/unmodified) are presented in Table 1, with success rates after each one of the five substeps of the harvesting cycle. Each column contains the fraction of fruits successfully completing the corresponding substep. Hence, success rates for the entire harvesting cycle can be found in the right-most column.  A center error in this phase caused the next phase of the obstacle detection to fail and abort the harvesting attempt. Sometimes, the distance reading of the fruit center was incorrect due to an occluding leaf coming into view during the centering of the robot, causing the robot to abort the current attempt.
When the fruit was properly centered, the obstacle detection took a first overview image to segment the stem and the fruit.
Sometimes this failed because the situation was so unique, and it was not covered in the training data of the machine learning algorithm.
Sometimes only one of the two classes was detected, causing a failure of this action.
Most failures during visual servo control were due to losing sight of the fruit, notably from leaves coming into the view when the end effector moved closer by. This most often happened in the last 5-10 cm to the fruit. For the modified condition, failures during visual servo control were rare. Visual servo control could also fail when the end effector pushed away the fruit (getting in contact with the stem or a neighboring fruit). In the current mechanical design, the fingers to catch fruit sometimes push the plant away while the end effector is approaching the fruit. This is because the fingers, even constructed to do so, do not flex away easy enough when touching leaves or other fruits in the way. The target fruit then never comes closer to the tool center point, aborting the procedure when the manipulator reached the boundary of its workspace.
Detachment of the fruit could fail in multiple ways. First, the end effector might have not been positioned accurately regarding the angle of the end effector and the fruit and stem (i.e., not in line with each other), but also regarding the Euclidean distance of the knife and the peduncle, which could be too much after visual servo control.
In such cases, the knife failed to intersect with the peduncle fully, causing only a partial or no cut of the fruit. Second, it was observed that the knife was sometimes slipping off the peduncle, especially with the Sardinero variety with its longer and more vertical peduncles. Third, the motion downward of the end effector was not always possible because the manipulator could already be near the end of its workspace when the knife was was positioned on top of the peduncle after visual servo control.
When the fruit was cut, it was required to be caught by the catching device. On a rare occasion, this failed because the fruit bounced just on the edge of the device. In some other rare occasions, the fruit got entangled in side branches or leaves while falling down after cutting, causing not to end up in the catching device.
The total average cycle time was 24 s (ranging between 18 and 25 s). It is important to mention that, due to safety reasons, the robot was not operated at the maximum speed technically possible. This was to protect the researchers working close to the system to collect data and also to protect the mechanical components of the robot.  Neighboring fruit can also affect the fruit localization process.
The results indicate that the performance of the robot could be increased by addressing occlusions and clusters of fruit. Approaches like plant breeding, mechanical solutions, or growing architecture, and crop maintained strategies could help to achieve this.
The hypothesis that the Sardinero crop variety would be more suitable for robotic harvesting compared with the Gialte variety could not be confirmed. It turned out that the longer and more vertical peduncles of Sardinero resulted in a less stable contact point for the knife such that sometimes the knife did not cut through but slipped off the peduncle. Furthermore, the longer peduncles increased the uncertainty about the exact location of the cutting point as the developed system only locates the fruit body and not the cutting point itself. This is also reflected by the observed higher end effector positioning errors for Sardinero, as shown in Figure 11. The hypothesis that a longer peduncle should reduce the risk of cutting into fruit body was not confirmed. Additional evaluation should also be conducted in different seasons and greenhouses to ensure robustness.

| SUGGESTED IMPROVEMENTS
The field tests provided valuable insights in the performance of the integrated system and how its modules functioned together. The following bottlenecks and possible improvements have been identified: F I G U R E 1 0 Average time needed for the different robot subtasks F I G U R E 1 1 Average positioning errors of the end effector after visual servoing control and before the cutting operation for the different crop varieties and cropping systems Ways to increase performance of the submodules for detection, reaching, cutting, and catching should be further investigated.
The performance results and failure analysis indicate that a different type of robot arm with an increased workspace and/or different mounting pose may improve the success rate. Compared with the overall size of the platform, the robotic arm (and its workspace) is rather small. However, an arm with longer joints will most likely also have more difficulties to plan and execute a collision free motion in this dense environment. An arm with more DOF will increase the computational efforts to plan the trajectory.
A smaller end effector would increase reachability and would decrease the risk of collisions with the crop. The end effector size can be reduced by using a smaller camera, light unit, or catching device.
As described above, the fingers of the catching device do not flex away enough when touching leaves or other fruits, thereby preventing in some occasions fruits from being reached and harvested. A redesign of the catching device is therefore recommended.
Transportation of the fruit to the storage bin currently consumes a lot of time compared with the other actions. The speed of the manipulator should be increased or alternative ways of storing the fruit should be considered.
Another insight concerns error handling and uncertainty control.
Currently, when the robot loses sight of the fruit or cannot reach a certain position, harvesting is simply aborted. Instead, the system should try to recover by backtracking to a point from which the operation can be resumed.

| CONCLUSIONS
The developed prototype was able to fully autonomously harvest sweet pepper fruits in a commercial greenhouse. For the current confirmed. This shows that breeding or other ways to enhance fruit visibility or decrease fruit clustering have great potential in the future of robotic harvesting.