Hyper‐Versatile Gripping: Synergizing Mechanical and Machine Intelligence of a Hybrid Robotic Gripper

Integration of robotic solutions in manufacturing sector is growing. However, it is still concentrated in certain industries (i.e., electronics and automotive) where standardization of product physical form is high. Current state‐of‐the‐art gripping solutions fall short when they need to accommodate items with high variability in physical form. This challenging scenario for automation can be found in a few industries (i.e., e‐commerce). Automation of pick‐and‐place processes in this area requires a more versatile gripping solution. To resolve this challenge, this article proposes a novel way to improve grip‐versatility by synergizing the mechanical and machine intelligence of a hybrid robotic gripper (HRG). Comparative analysis with commercial grippers shows that HRG can pick a more diverse range of items with success rate 94.78%. Visual perception‐based picking strategy is developed to automate the reconfiguration of HRG into a stable grasp pose for different objects. Using the proposed reconfigurable picking strategy, the efficacy of HRG in pick‐and‐place tasks is evaluated using three parameters—mean pick per hour (MPPH), successful execution over total attempts (SETA), and average cycle time (AVGCT). HRG can effectively pick items in cluttered workspace with MPPH of 98.54 ± 15.49, SETA of 0.93 ± 0.11, and AVGCT of 34.76 ± 3.31 s.


Introduction
Automation of pick-and-place processes with industrial and collaborative robots has been integrated in manufacturing lines to promote high-volume production.Pick-and-place tasks are commonly applied in production lines across various industries (i.e., logistics and automotive). [1]Although integration of robotic solutions in manufacturing sector is growing, it is still concentrated in a few industries where there is a high standardization of product physical form. [2]ess irregularities in geometric forms allow easier automation of pick-and-place tasks.In domains such as e-commerce industries, occurrence of large variation in morphology of items is inevitable. [3]In these areas, commercial grippers are not adequate for automation as they have been designed to handle items with specific physical characteristics.Thus, manual mode is still the default choice of operation. [4]obotic grippers are designed to automate mundane and manual processes such as pick-and-place tasks to promote productivity.Generally, commercial gripping solutions are categorized based on the number of fingers, gripping method, and actuation mechanism. [5,6]These grippers can have two to five [7] fingers.Gripping method can be impactive, astrictive, or contigutive. [8]The grippers can be actuated by means of vacuum, magnetic, pneumatic, hydraulic, or electric. [6,9]inger grippers can be further classified based on its material like metallic alloy [10,11] or silicone rubber. [12]Vacuum grippers are generally used to pick nonporous and structured items. [5]inger grippers made of metal alloys or soft rubber are more appropriate to handle items with high porosity and high moisture. [13][16][17][18][19][20] However, commercial soft grippers still lack grip-versatility in handling geometric uncertainties in objects. [21]Hence, automation of pick-and-place tasks will require installation of different grippers, which is cost-ineffective and increases operation complexity.
Grip-versatility is defined as the ability of the gripper to pick up different types of objects.Different gripping elements have their own advantages and disadvantages as discussed previously.Grip-versatility can be improved by integrating different gripping elements.[24][25] Parallel jaw gripper has been developed with suction cups at the finger tips.However, the grip-versatility is limited because the presence of a suction cup at the finger tips hindered effective grasping of items. [22]Kang et al. proposed a finger gripper that is fitted with suction cup at the center of the gripper.Center placement of suction cup improved the effectiveness of dual-gripping mode.However, the parallel fingers increased the risk of grasp instability with irregular shaped objects. [23]A soft gripper with a variable structure was proposed by Huang et al. to pick objects of various shapes and sizes. [24]owever, the presence of a protruding slider at the base of the gripper restricted its movement in cluttered and constrained environment.Liu et al. proposed soft fingered gripper with central suction cup.Dual-gripping mode improves grip-versatility.However, due to the fixed gripping width, it limits the number of items that can be grasped by the gripper. [25]o resolve these issues, this article proposes a reconfigurable HRG.Reconfigurability is defined as having the freedom to vary the configuration parameters of gripping components (i.e., finger gripping width and activating vacuum suction as alternative or synergistic gripping modes).Reconfigurability allows the gripper to adapt to items of different shapes and sizes.It is important to note that manual labor has been the default choice in production lines due to dexterous manipulation and cognitive capabilities of human workers.Hence, automation of these processes requires robotic solutions with both mechanical intelligence and machine intelligence.
28][29][30][31][32] Single depth camera, [33][34][35] depth sensors, [36] and 3D sensors [37] have been used to obtain visual data about the objects and their environment.40] Various learning-based grasping or grasp pose detection methods have been reported which include segmentation of target objects, [41] analysis RGB-D image data, [42] template-based method, [43,44] bounding box-based method, [45,46] and convolutional neural networks for detection of objects in cluttered environment. [47]ost existing data-driven grasping techniques have been conducted using either multifingered gripper or vacuum gripper.However, the hyper-versatile gripping solution in this article incorporates two gripping modes in one gripper.In addition, more gripping parameters (i.e., finger gripping width and finger orientation) can be adjusted based on the shape and size of the objects.Due to larger degree of variation in gripping parameters, existing data-driven grasping techniques could not be readily applied.Therefore, the article will also discuss the design and development of a visual perception-based picking strategy for the reconfigurable hybrid gripper.
In summary, this article proposes integration of machine intelligence and mechanical intelligence of the gripper whereby gripping parameters can auto-reconfigure based on the change in shape, size, and fill capacity of the detected objects.
The contributions of this article are as follows: 1) the design and realization of a versatile robotic gripping solution with reconfigurable grip poses and dual-gripping modes to effectively handle items of various shapes, sizes, and packaging materials; and 2) the development of visual perception-based autoreconfigurable pick-and-place system which synergizes machine intelligence and mechanical intelligence of the HRG.

Design and Realization of HRG
This section presents a novel HRG, as illustrated in Figure 1.
Hereafter, the HRG is referred to as the HRG.HRG is designed with the reconfigurable capability to improve its adaptability to a wider variety of items with enhanced grip-versatility.HRG is made possible by: 1) including reconfigurable soft fingers (i.e., they are reconfigurable in terms of position, orientation, and bending angle), and 2) synergizing two gripping components (i.e., a suction cup and soft fingers).
The expandable base of HRG uses four rotating linkages, each of which houses a soft finger at the end of the linkage.The rotating expansion mechanism allows the gripper to expand and contract based on the size of items that it needs to pick.Each soft finger is linked to an individual stepper motor.This allows the gripper to orientate the fingers into parallel and radial configurations based on the shape of the items.
At the center of HRG, it is equipped with the extendable palm actuator integrated with a suction cup.Upon pneumatic pressurization (with a maximum 50 kPa), the palm actuator can extend up to 110.07 mm from its fully contracted state (at À70 kPa).If it is not feasible for the gripper to pick an item with the soft fingers, the palm actuator that is fitted with a suction cup can be an alternate viable means.
These two gripping components can be deployed simultaneously to enhance grip stability.This is practical in use-cases whereby heavy sachets (such as rice packets) need to be transported.In such a scenario, the suction cup acts as the primary gripping mode for supporting the weight of the item, and the soft fingers act as the secondary gripping mode to stabilize the item during transportation.The fingers are also fitted with fabric sleeves that are integrated with silicone antislip skin to enhance the grip stability.
The configuration of HRG is represented by the following parameters: 1) gripping width, r; 2) orientation of soft fingers, θ; 3) the supplied pneumatic pressure to the soft fingers, p finger ; 4) the supplied pneumatic pressure to the extendable palm actuator, p palm ; and 5) the supplied pneumatic pressure to the suction cup, p vacuum .
The gripping width (r) ranges from 100 to 300 mm.The four soft fingers can be configured into a radial or a parallel configuration with the orientation of (θ = 0°) or (θ = 45°or À45°), respectively.Figure 1B-I demonstrates the different configurations of HRG and their corresponding parameter values.

Bending Angle-Pressure Relationship of Soft Fingers
Detailed experimental setup is covered in Appendix A1.During the depressurized state, the bending angle of the soft finger was denoted to be 0°(baseline) (Figure 2A).With the increase in the pneumatic pressure (p finger ) (from 0 kPa to 50 kPa), an approximate linear increase in the bending angles of the fingers was observed.The maximum bending angle of the soft finger, as shown in Figure 2G, is 85.43°AE 0.08°at 50 kPa.With the increase in pneumatic pressure, continuous expansion of silicone bellows caused the finger to consistently bend toward its thicker bottom layer.This resulted in an approximate linear increase in the bending angle of the fingers.

Displacement-Pressure Relationship of Extendable Palm Actuator
Detailed experimental setup is covered in Appendix A2.During the depressurized state (with no pressure supplied to the palm actuator), the baseline of the displacement (d) was measured to be 41.35 mm AE 0.28 mm (Figure 2E).With the increase in the pneumatic pressure ðp palm Þ, an approximate linear increase in the displacement of the palm actuator was observed.The maximum displacement, as shown in Figure 2H, was 110.07 mm AE 1.17 mm at 50 kPa.The uniform wall thickness and symmetrical tubular structure of the palm actuator allow equal distribution of compression force between adjacent bellows.This, in turn, enables the actuator to exhibit approximately linear displacement.

Performance Evaluation with Commercial Grippers
The grip-versatility of HRG was evaluated with the commercial gripping solutions.Detailed experimental setup is covered in Appendix A3.The average pick-and-place success rates of five grippers (Figure 3B-E) for 23 consumer goods of different shapes, sizes, and packaging materials (Figure 3A,F) are analyzed.The results (Figure 3G) show that the HRG has the highest success rate of 94.78%, followed by the ROBOTIQ 2F-140 gripper with 73.91%.The suction cup and the SRG_165 have the lowest success rate of 47.83%.
It is to be highlighted that the SRG was manually configured with two gripping widths, i.e., 100 mm (SRG_100) and 165 mm (SRG_165).With the SRG_100 configuration, the gripper is incapable of picking large items, such as items 20 and 23.However, with the SRG_165 configuration, the gripper is unable to pick smaller items, such as items 12, 13, and 14.If SRG can auto-reconfigure its gripping width, the success rates of SRG_100 and the SRG_165 can be combined.This could potentially increase the success rate of SRG to 82.61% which highlights the importance of having a reconfigurable gripping-width capability.Such a feature will improve the grip-versatility and enables the gripper to pick-and-place a larger variety of items.
However, the combined success rate of SRG is still notably lower than HRG.It was observed that certain items, i.e., item 2, can only be picked-and-placed by HRG.Out of the 23 items, item 2 is the largest and heaviest item with a dimension of 230 mm Â 170 mm Â y 50 mm and a weight of 2.5 kg.Compared to the commercial grippers, HRG was the only solution that was able to pick-and-place item 2 successfully.This success is attributed to its dual-gripping mode, where 1) the central suction cup was used as the primary gripping mode to support the heavy weight of the item, and 2) the soft fingers were deployed as the secondary gripping mode to reinforce the grip stability to prevent slippage during robot movement.
The results from this comparative analysis have emphasized the two features that are essential to enhance grip-versatility of a gripper, namely, 1) the reconfigurable gripping width and 2) the dual-gripping mode.

Reconfigurable Picking Framework
In this section, a visual perception-based framework is proposed to determine a stable grasp pose for HRG.A block diagram is illustrated in Figure 4A that provides the outline of the framework.This framework is based on image-processing and machine-learning techniques 1) to extract relevant geometric features of a targeted object and 2) to produce corresponding configuration of HRG for a stable grasp pose.
First, the framework functions by acquiring real-time RGB and depth images of the picking workspace (i.e., tote bin).Second, it utilizes a neural network YOLOv5 [48] to detect the objects in the RGB image.Third, a depth image processing algorithm extracts the geometric features of all the localized objects (i.e., shape, size, and fill capacity) and their corresponding object poses (i.e., orientation).Finally, based on the extracted object information, the framework deduces a stable grasp pose (a suitable configuration) for HRG to pick up the objects.
Depth image processing algorithm analyzes the areas of the bounding boxes of the detected objects generated by YOLOv5 on the acquired depth images (Figure 4C-E).The area of a bounding box of a detected object on a depth image is described as where B is the index of the detected object and M and N are the total number of pixels occupying the area of the bounding box of the detected object.The major and minor radius of the detected object (including the robot-arm grasp pose (x, y, z, r x, r y, r z )) are approximated based on a contour-detection algorithm that is implemented on the depth image (Figure 4).
The shape of the detected object is deduced using the eccentricity of the object (e approx ) and a preset threshold (e thres ).The preset threshold is provided to determine if the shape of the object is circular or noncircular.
The deduction of the shape of the object is given as follows: The fill capacity of the detected object is determined by comparing the height of the object at its center, i.e., h , with respect to the rest of its body.
The determination of the fill capacity of the object is provided as follows: where h min is the minimum height that can be measured at the area of the bounding box of the detected object.
Based on the extracted object information from the depth image processing algorithm, the framework determines the following: 1) the gripping mode, i.e., finger-gripping mode or dual-gripping mode; 2) the gripping width, r; and 3) the orientation of the soft fingers, θ.
First, with information of fill capacity of the object, the gripping mode of HRG is determined by the following condition.If the object is hollow, the finger-gripping mode of HRG would be deployed to pick it up.Otherwise, the dual-gripping mode of HRG would be chosen.It is to be highlighted that to ensure load stability during the pick-and-place tasks, the vacuum-gripping mode of HRG would only be deployed if the object can be grasped at its centroid which is commonly the center-of-mass.However, further considerations for deployment of the vacuum-gripping mode need to be addressed.This is because Second, with the minor radius of the object, the gripping width of HRG is described as r ¼ 2r minor (5)   Third, based on the shape of the object, HRG is configured into either a parallel or a radial configuration with the following equation: It is noted that for the framework to work effectively, the soft fingers need to be properly aligned with the gripping surface of the detected object.The gripping surface is defined as the surface of the object that is parallel to the major axis of the object, as illustrated in Figure 4E.As the gripper expands or contracts with this mechanism, it changes the positions of the soft fingers, as depicted in Figure 5A-C.The following elaborates a methodology to properly align the soft fingers with the gripping surface of the detected object.
The joint angles of the robot arm are adjusted to ensure the palmar surface of the fingers are aligned to the gripping surface of the object.There are three possible scenarios where the realignment of the robotic fingers is needed and they are as follows: 1) Scenario 1-Reconfiguration of the gripping width (r) and the gripper configuration from parallel θ ¼ AE45°ð Þto radial θ ¼ 0°ð Þ: 2) Scenario 2-Reconfiguration of the gripping width (r) and the gripper configuration from radial θ ¼ 0°ð Þto parallel θ ¼ AE45°ð Þ .3) Scenario 3-Reconfiguration of the gripping width (r) only.
The transformations of a soft finger (in terms of position and orientation) that are induced by the above scenarios are illustrated in Figure 5C and they are computed as follows: where α denotes the angle of the finger with respect to the centerof-gripper, c denotes the radius of the gripping width, subscripts i and j refer to the current and the previous configurations of HRG, respectively, a is the distance of the finger to the finger joint, b is the distance of the finger joint to the center-of-gripper, R denotes the radial configuration, and P refers to the parallel configuration of HRG.
Based on the known transformations of the fingers, the robot arm offsets its joint angles with q offset to align the palmar surface of the fingers to the gripping surface of the detected object for better grip stability.
The parameter q offset is described as follows: The stable grasp pose (q target ) for the robot arm to pick up the detected object is given by where q current is the current robot-arm grasp pose for picking up the detected object.

Performance Evaluation of Reconfigurable Picking Framework
Detailed setup for performance evaluation is covered in Appendix A4.The performance of the proposed framework was evaluated with the following metrics-mean pick per hour (MPPH), successful execution over total attempts (SETA), and average cycle time (AVGCT).
MPPH computes the average number of items that can be successfully picked and placed by the gripper and is expressed with following equation: MPPH ¼ n trials p avg (16)   where n trials denotes the number of pick-and-place trials per hour and p avg is the average probability of the robot picking-andplacing the items successfully.SETA estimates the average probability that a pick-and-place task is executed successfully and is described as follows: where N P denotes the number of items picked and placed by the robot successfully and p i refers to the probability of the robot picking-and-placing item i from the tote bin successfully.AVGCT computes the average cycle time of the successful pick-and-place task and is expressed with following equation: where T i refers to the number of grasp attempts for item i and the maximum number of attempts is capped at three.p i refers to the probability of the robot picking-and-placing item i from the tote bin successfully.Δ i t refers to the duration taken by the robot to successfully pick the item i from the tote bin and place it in the carton box.
The results for the three metrics based on the experimental data are summarized in Table 1.The highest and lowest average MPPH were recorded to be 107.9AE 0.00 and 82.35 AE 22.13 when the clutter scenario comprised two and three items, respectively.The lowest average MPPH was incurred by the repeated attempts to pick up the bag of chips (object ID 21) (Figure 3A).
Even though the bag of chips was classified as a solid object, it cannot be grasped successfully using the suction cup because the material of the suction cup was incompatible to the surface material of the bag.Hence, the suction cup could not form an airtight boundary on the surface of the object to lift it up.Nonetheless, such drawbacks can be eradicated by changing the suction cup to one that suits the application.
The results for SETA were observed to be stable throughout all the four clutter scenarios, and they were recorded to have a range of 0.90-0.95.The high SETA results were attributed by the two significant features of HRG for achieving great grip stability, namely, 1) the reconfigurable soft fingers and 2) the dualgripping mode.
The lowest and highest AVGCT were recorded to be 33.34AE 0.00 and 40.71AE 6.33 s when the clutter scenario comprised two and three items, respectively.The computation of AVGCT matrix considers the failed attempts to pick an item.Therefore, the highest AVGCT was incurred by the repeated attempts to pick up the bag of chips, i.e., object ID 1.The results for the three metrics have indicated that the proposed framework is an effective picking strategy for HRG, as the MPPH and AVGCT did not deteriorate with the high clutter scenarios.

Nesting Strategy
An optimal nesting strategy is devised here to compactly pack items with diverse physical features into a designated workspace (e.g., tote bin, corrugated carton box, and so on) (Figure 6A).This strategy requires two inputs: 1) the extracted object information from the depth image processing algorithm and 2) the geometric parameters of the corrugated carton box.
The optimal nesting strategy is formulated as follows: Minimize where j ∈ f0, : : : , N À 1g, i ∈ f1, : : : , Ng and  x ∈ fp subject to V i ≤ V j , where by The above notations are detailed below: 1) Superscripts i and j denote the current and the previous index of an object for the packing task, respectively.2) C x , C y , and C z refer to the 3D coordinates of the center of an object in the packing workspace.3) w, l, h, V refer to the width, length, height, and volume of item in millimeters (mm).4) N is the total number of items to be packed.5) h min denotes the minimum available height within the packing workspace.H is the height of the packing workspace.6) x and y denote the positions of the extreme corners of the object bounding box at the x-and y-direction.These two corners are chosen as possible placement positions for the next object because they offer minimal possible separation distance between the centroids of the two adjacent objects.7) P 0 denotes the designated packing workspace, i.e., the corrugated carton box.p 0 x , p 0 y , p 0 z refer to the 3D coordinates of any given point in the packing workspace (i.e., P 0 ).8) P i denotes item i, i.e., the current item for the packing task.p i x , p i y , p i z refer to the 3D coordinates of item i occupying P 0 .9) p x 0 , p y 0 , p z 0 refer to the 3D coordinates of the origin of an object in packing workspace.The origin of an object is at the bottom-most left-most corner of its bounding box derived by the depth image processing algorithm.v refers to the vector of translation for object i in the packing workspace, the unit of which is in millimeters.
The objective function, i.e., Equation (19), minimizes the volume usage of the designated packing workspace to compactly pack all the objects (Figure 6B).The following assumptions were made for the nesting strategy: 1) The first item will always be packed at the bottom-most left-most corner of the packing workspace.2) The total weight of the objects is within the load bearing limit of the designated packing workspace (i.e., corrugated carton box).3) All objects are stackable.4) The packing sequence of the objects is based on the fragility and the size of the objects.Fragility of the objects is defined by the end user prior to the execution of the pick-and-place task.5) The unused volume of the packing workspace will be filled with packaging foam to secure and protect the items in their respective positions.6) The object is not rotated when it is being placed in the packing workspace.
The constraints of the optimization problem are elaborated as follows: 1) Constraint (27) ensures that all objects are packed inside the packing workspace.2) Constraint ( 28) ensures that objects do not collide with each other.int P ð Þ represents the physical space occupied by the object.3) Constraint ( 29) enforces vertical stability of packed objects to minimize the probability of the items from toppling over.4) Constraint (30) ensures that the objects are packed according to the packing sequence that is derived based on the fragility and the size of the objects.

Effectiveness of Pick-and-Place System with HRG
Here, a robotic system with HRG for the pick-and-place tasks in packaging processes is developed.A prototype system framework with visual feedback is established (Figure 7).It consists of four subsystems for execution, perception, planning, and control.
The execution subsystem comprises a UR5e robot arm and a reconfigurable multimodal SRG-HRG.The perception subsystem consists of an Intel Realsense D435i camera to capture realtime RGB and depth images of the workspace.This subsystem also encompasses an algorithm that uses YOLOv5 [48] and depth image processing framework to provide real-time object localization and recognition.The planning subsystem includes the reconfigurable picking framework and the optimal nesting strategy.The control subsystem receives the feedbacks from the perception and planning modules and instructs the robotic system to execute the pick-and-place task.A set of experiments were performed to demonstrate the effectiveness of the proposed prototype system in the pick-and-place task with HRG.Five items with diverse physical features (Figure 5J) were picked-and-packed from a tote bin into a corrugated carton box (with a dimension of 350 mm Â 310 mm Â 200 mm).Figure 8 shows a series of recorded images depicting the representative experimental results of the proposed system in executing the pick-and-place task.
Figure 81A-5A depicts the items that are localized by the perception subsystem.Items that could be picked using vacuum suction as the primary gripping mode and that are of larger dimensions are picked first.Figure 81B-5B shows the sequence of items that are being picked from the tote bin and Figure 81C-5C shows the sequence of items that are brought to the carton box to be packed.The arrangement of items in the carton box is of the same layout as it was proposed by the nesting strategy (Figure 6C).However, it is to be highlighted that there will be narrow interspaces between items if the soft fingers were deployed to pack them; these empty spaces are needed to ensure that there is no collision during the packing process.

Conclusion
We propose a hyper-versatile gripping solution that synergizes the mechanical intelligence of a gripper and the machine intelligence that leverages on visual perception.Comparative analysis with commercial grippers shows that HRG can pick a larger variety of items with a success rate 94.78%.This is only possible due to enhanced versatility of the gripper which is made possible by 1) including grip pose reconfiguration mechanism and 2) dual gripping modes (vacuum suction and finger gripping).However, although reconfigurability improves the adaptability of the gripper, it also brings about operational complexity.To efficiently handle an item, more than one grasp pose can be available.Manual search for a stable grasp pose through various permutations of gripping parameters will be time-consuming.Hereinafter, a more efficient way to make use of HRG is needed.
Hence, computational framework that makes use of visual perception (machine intelligence) is developed to strategize the selection of a stable grasp pose.Using the proposed reconfigurable picking strategy, HRG can reconfigure and effectively pick mixed items in cluttered workspace with average MPPH of 98.54 AE 15.49, average SETA of 0.93 AE 0.11, and AVGCT of 34.76 AE 3.31 s.
Finally, a robotic system with visual feedback and reconfigurable hybrid gripper for the pick-and-place tasks in packaging processes is developed.To the best of our knowledge, this is the first study that integrates the proposed reconfigurable picking strategy with nesting strategy and effectively demonstrates picking-and-packing task with a reconfigurable hybrid soft gripper.The versatile gripping solution that is presented in this article is a step toward automation of challenging pick-and-place processes in the industries.

Appendixes A1. Characterizing Bending Angle of Soft Fingers
Three soft fingers were fabricated and evaluated for their bending angles at different pneumatic pressures.Preliminary tests show that each finger can withstand a pneumatic pressure (p finger ) of up to 50 kPa, beyond which leakage would occur at the interface between the finger and its base.Therefore, the pneumatic pressure (p finger ) was varied from 0 to 50 kPa-with an incremental step of 10 kPa.As shown in Figure 2A-C, the soft finger was clamped on a retort stand and a camera was used to capture the actuation process.The bending angle of the inflated finger was analyzed with an image-processing program, ImageJ (National Institutes of Health, Bethesda, MD).The bending angle was measured by obtaining the angle between the vertical y-axis and the fingertip of the actuator.Subsequently, the average bending angle at each pneumatic pressure was calculated.

A2. Characterizing Displacement of Extendable Palm Actuator
An extendable palm actuator was fabricated and evaluated for its displacement (d) at different pneumatic pressures.The displacement of the extendable palm actuator (d) is denoted as the change in height of the free-end of the palm actuator from its reference state.The reference state is regarded as the contracted state of the actuator with a negative pressure (p palm ) of À70 kPa being supplied.The preliminary tests have shown that the palm actuator could withstand pneumatic pressure ðp palm Þ of up to 50 kPa, beyond which leakage would occur.For the experiment, the pneumatic pressure ðp palm Þ was varied from 0 to 50 kPa-with an incremental step of 10 kPa, as shown in Figure 2D-F.This experiment was repeated 3 times.
Two yellow markers were labeled on the fixed-end and the free-end of the palm actuator, respectively.A camera was used to capture the actuation process and displacement was analyzed with an image-processing program, ImageJ (National Institutes of Health, Bethesda, MD).The height of the actuator was measured by obtaining the height difference between the two yellow markers.The displacement (d) was measured as the change in height of the actuator at a pressurized state with respect to its reference state.

A3. Comparative Analysis of HRG
The grip-versatility of HRG was evaluated with the commercial gripping solutions.The benchmarking test was conducted by tasking HRG and the commercial grippers to pick up 23 consumer goods of various shapes, sizes, and surface finishes/ materials (Figure 3A,F).
These grippers were mounted on the tool flange of the 6-DOF Universal Robot (UR5e) arm (Universal Robots, Odense, Denmark), and the robotic UR5e arm was fitted with a pneumatic WWR50 tool-changer (Zimmer Group, Rheinau, Germany).This tool-changer allows the robotic UR5e arm to readily switch between different grippers throughout the benchmarking experiment.All grippers were tasked to pick-and-place the item from the center of a tote bin to a corrugated carton box.This experiment was repeated 3 times to yield statistically meaningful results.
For the benchmarking experiment, the grippers were configured as follows: 1) HRG-The four soft fingers and the extendable palm actuator of HRG were supplied with 50 kPa of pressurized air.The suction cup of HRG was supplied with 70 kPa of vacuum pressure.Based on the size of the item, a suitable gripping width was chosen to pick the item.For the deformable items, the soft fingers act as the primary gripping mode to complete the task.For the remaining items, a combination of the two gripping components was deployed if there is a sufficient space available for the suction cup.2) SRG-The soft fingers of SRG were supplied with 50 kPa of pressurized air and they were arranged in a radial configuration.It is to be highlighted that the orientation of the fingers and the gripping width are not reconfigurable.However, it is noted that the gripping width can be manually adjusted.For the benchmarking experiment, two gripping widths, 100 mm (SRG_100) and 165 mm (SRG_165), were evaluated.3) Suction cup-The suction cup (Piab, Taby, Sweden) was attached to the ROBOTIQ E-Pick system, VAC-ES-UR-CB-kit1 (ROBOTIQ, Quebec, Canada) for the supply of the vacuum pressure for the pick-and-place tasks.4) ROBOTIQ 2F-140 gripper-The gripper was fully closed to grip the items using the ROBOTIQ's grasp controller with 50% of maximum speed and 20% of maximum force.

A4. Performance Evaluation of Reconfigurable Picking Framework
Here, experiment was conducted to evaluate the performance of the proposed picking framework for HRG.
HRG was mounted on the tool flange of a robot arm, i.e., a UR5e robot arm, and an Intel Realsense D435i camera was utilized to acquire real-time RGB and depth images of the content of a tote bin.The experimental setup is depicted in Figure 5F.For the experiments, five consumer goods with diverse physical features in terms of shape, size, and surface material were randomly placed in the tote bin to simulate a highly mixed packaging line of an e-commerce platform.In the clutter scenarios as shown in Figure 5G-J, the number of items was varied from two (i.e., low clutter scenario) to five (i.e., high clutter scenario).For each clutter scenario, the picking experiment was repeated 5 times to yield statistically meaningful results.

Figure 1 .
Figure 1.A) Illustration of HRG mounted on UR5e.B) Schematic of expansion mechanism/rotatable finger base.C-J) Reconfiguration of HRG.The box at the top left indicates the state of parameters for the specific reconfiguration.B-I) The reconfiguration sequence and the modified parameter from previous state are indicated in red.

Figure 2 .
Figure 2. Characterization experiment setup for HRG: A-C) bending angle of soft fingers and D-F) displacement of palm actuator.Characterization experiment result for HRG: G) average bending angle of soft fingers from three trials and H) average displacement of palm actuator from three trials.

Figure 3 .
Figure 3. A) Photos of 23 items used in comparative analysis experiment, B) suction cup attached to ROBOTIQ E-Pick, C) soft robotic gripper (SRG), D) ROBOTIQ 2F-140 gripper, E) HRG, F) graph summarizing weight, gripping width and aspect ratio (width/length) of 23 items used in benchmarking experiment, and G) average picking success rate of 23 items using different gripper from three trials.

Figure 4 .
Figure 4. A) Computational framework for reconfigurable picking strategy.It reconfigures HRG based on object localization and recognition using visual perception; input to perception module: B) RGB image and C) depth image; output of perception module: D) annotated RGB image.Three annotations are indicated and they are 1) arrows pointing toward the side of the object indicating parallel or radial finger gripping direction, 2) direction of arrow on the object indicating the object orientation detected by the perception module,and 3) the presence of circular ring in the center of the object indicates that vacuum suction can be used.Similarly, the absence of circular ring in the object center indicates that the object is hollow and hence vacuum suction will not be used.E) Illustration to depict how analysis of localized object (gray depth image) is done to extract its geometrical features.

Figure 5 .
Figure 5. Shifted position of soft fingers during expansion and contraction of gripper base: A) 100 mm gripping width, B) 200 mm gripping width, and C) 300 mm gripping width; finger orientation: D) radial configuration and E) parallel configuration, whereby individual finger (1-4) rotates by 45 degrees in specified direction as indicated by the black arrow; F) experimental setup for pick-and-place tests; and G-J) four clutter scenarios with labeled ID of objects in (J).

Figure 6 .
Figure 6.A) Proposed nesting strategy and B) illustrations of packing arrangement.Black color box refers to the packing workspace (i.e., carton box).Green color box refers to the first object that has been placed in the box.Blue color box refers to second object that is currently being placed in the box.The black dot refers to 3D coordinates of origin of the first object.The two red dots refer to corners of the box which could be the 3D coordinates of the origin of the second object.C) Visualization of item arrangement in packing workspace using proposed method.Object ID corresponds to items indicated in (A).

Figure 8 .
Figure 8. Demonstrator for robotic pick-and-place system.1A-5A) Live sequence of items detected by the camera.1B-5B) Live sequence of robot picking items from picking workspace.1C-5C) Live sequence of robot packing items in the packing workspace.

Table 1 .
Performance evaluation of HRG picking strategy from five trials.