PolyRec Gesture Design Tool: A tool for fast prototyping of gesture-based mobile applications

In this article, we present PolyRec Gesture Design Tool (PolyRec GDT), a tool that allows the definition of gesture sets for mobile applications. It supports the entire process of designing gesture sets, from collecting gestures to analyzing and exporting the set created for use in a mobile application. It is equipped with the PolyRec gesture recognizer, a unistroke recognizer based on template matching with good accuracy and performances while still allowing support for other recognizers. The main features of PolyRec GDT include the ability to collect gestures directly on mobile devices, to detect similarities between gestures to help the designer to detect ambiguities in the design phase, and the ability to automatically select the most representative gestures for each gesture class by using a clustering technique to increase recognition speed without affecting accuracy. Experimental results show that this latter feature allows an increase in accuracy compared with when the same operation is performed manually by developers or randomly by software. Finally, a user study was carried out to evaluate PolyRec GDT, in which participants were asked to use it to add gesture functionality to a mobile application. The results showed that with the support of PolyRec GDT this operation took only a few minutes for participants with mobile development experience, whereas it would have taken much longer without the support of PolyRec GDT.

The characteristics for which the different hardware differ are essentially accuracy, pressure detection, and the number of simultaneously detectable points (multitouch). These characteristics define the types of gestures that the device is able to accept and therefore must be taken into account during the design of the gesture set.
Mobile operating systems such as iOS and Android include features for recognizing the most common gestures, and the HCI community has produced different solutions to the problem of recognizing gestures belonging to a specific gesture sets, both multistroke and unistroke. In particular, the latter is often preferred for their usability and for the ease with which they can be interpreted by the computer.
Designing a good set of gestures can be complicated, especially when the number of gestures increases. A good set of gestures must balance two often conflicting features: be composed of gestures easily classifiable by an algorithm but at the same time easy to perform and remember for human beings. Moreover, for a software designer, it can be difficult to evaluate the characteristics of a gesture set with respect to these features.
A good tool for the gesture design, in addition to allowing to collect and organize the gesture set, must, therefore, provide the developer feedback about these features. Designing good gesture sets can be essential to take full advantage of gesture-based interfaces.
The goal of this work is therefore to present PolyRec Gesture Design Tool * (or PolyRec GDT), a tool for the design and definition of gesture sets for mobile applications (touch devices), with particular attention to the definition of gesture sets that allow a high recognition rate.
PolyRec GDT is equipped with the PolyRec 8 unistroke gesture recognizer based on template matching and supports the whole process of gesture set design, from the collection of gestures, to the analysis, to the export of the created set for use in a mobile application together with the PolyRec recognizer. Moreover, the tool is fairly generic and thus allows supporting additional gesture recognizers.
To simplify the analysis of the gesture set, the tool provides functionalities for the detection of similarity between gestures, in order to help the designer in identifying problems that could generate ambiguity in the recognition phase, so that he can redesign them to limit conflicts. In addition, the tool allows the developer to automatically select the most representative templates of a gesture class, so as to reduce the computational cost while reducing the impact on recognition accuracy. These are the key distinguishing features of PolyRec GDT with respect to other gesture management tools.
The latter features and the ability to export easy-to-use gesture recognition source code are the main design goals of the tool.
Another issue in the area of gesture recognition is how to find the right compromise between execution speed and accuracy of recognition. To this end, PolyRec GDT uses a clustering technique to automatically select the most representative templates for each gesture class. The usefulness of this feature was evaluated both on datasets of standard gestures and in the real case of a mobile application in which the developers, not having PolyRec GDT, had manually chosen the gestures for the recognizer. The results show that the algorithm selection allows to increase the accuracy of the recognition, especially when only one sample per class is available.
Finally, to evaluate PolyRec GDT, an experiment has been performed in which participants were asked to use it to support them during the development of a mobile application. Results show that adding gesture functionalities to an existing application took only a few minutes for a participant with mobile development experience, which would have taken a much longer time without PolyRec GDT.
This article is organized as follows: Section 2 describes some works related to ours; Section 3 describes the PolyRec gesture recognizer and analyzes its accuracy; Section 4 describes the PolyRec GDT user interface and features; Section 5 describes an empirical case study on the automatic templates selection functionality; Section 6 describes the PolyRec GDT experimental evaluation. A discussion on the results of this study and some comments on future work concludes the article.

RELATED WORK
In this section, we discuss previous researches related to our work. It is divided into two sections: the first one describes works on gesture design and supporting tools and techniques, while the second one describes algorithms for gesture recognition on touch interfaces. sets and recognizers, both for single and multistroke gestures. Its main feature is that it allows the developer to analyze 12 new relative accuracy measures for stroke gesture articulation that characterize geometric, kinematic, and articulation accuracy of gestures. The measures have been evaluated on public gesture datasets in order to better understand users' stroke gesture articulation behavior and to obtain useful information for developers on this topic. The GEsture Clustering toolKit (GECKo) 26 is a tool whose aim is to facilitate the study of how users articulate stroke gestures (both unistroke and multistroke) and help to better understand how users make gestures. It allows to visualize and playback gestures, and, in order to analyze user consistency in gesture production, to cluster them based on stroke number, order, and direction. It allows to audit the clustering results and to see within-and between-subject agreement rates after clustering. The methodology has been used on four gesture datasets and implications from results can be used to help designers create better gesture sets. So far we reported tools that support touchscreen operations; however, there are also systems that deal with different types of gestures, such as midair or hand-based gestures. We report them here only for completeness since these types of gestures are out of the scope of our current proposal. Among these systems we can mention MAGIC (Multiple Action Gesture Interface Creation), 27 a tool that allows to design motion gestures without specific knowledge of pattern matching, and also test them in order to ensure that they are not triggered unintentionally by the user (false positives); GRDT (Gesture Recognition Design Toolkit), 28 a set of tools to support gesture creations; Mogeste, 29 a tool that allows designers to define, train, and test new motion gestures just captured through the inertial sensors in a commodity device (e.g., smartphone); EventHurdle, 30 a visual design tool that support designers in exploratory prototyping, provides functionality to integrate the defined gestures into a prototype for their automatic recognition, and supports handled gestures through physical sensors, remote gestures through a camera, and also touchscreen gestures; ProGesture, 31 that supports rapid prototyping of full body gestures by combining three design matters: gesture, presentation, and dialog; KIND-DAMA, 32 a modular middleware for developing interactive applications based on gestures on Kinect-like devices; KinectAnalysis, 33 a system for elicitation studies on the Microsoft Kinect, with support for recording, analysis and sharing; GestureAnalyzer, 34 a tool that allows gesture analysis by applying clustering and visualization techniques to the gesture data captured by motion tracking; CUBOD, 35 which allows users to design and customize their own gestures, rather than relying on those defined by designers, and offers feedback to guide users through the design process in order to avoid gestures that are difficult to distinguish from others, difficult to execute with consistency, or too similar to unintended movements; and HotSpotizer, 36 another system that allows users to define their own custom gestures, in this case to map them into keyboard commands, in order to use them in arbitrary applications.
In addition to tools, techniques, and methods to aid in the design of gesture sets have also been presented in the literature. Henrique et al. 37 proposed a methodology, with focus on pen-gesture interfaces, to automate the design of a gesture vocabulary. Using a clustering algorithm, the gesture space can be explored for a set of gestures easily recognizable by the recognizer, and then allowing the gesture vocabulary designer to select the most appropriate gestures from the self-organized gesture lists. When gestures are designed for a specific system or platform, it is often difficult, if not impossible, to use the same gestures on systems with different characteristics. Lao et al. 38 attempted to make a complete understanding of all possible touch types for touch-sensitive devices, and then create a design model for touch interaction. The model consists of three levels: action level, motivation level and computing level. This allows reusing the design work performed in one level, for example, when designing an interactive interface it is possible to design the touch at the action level once and reuse it for other applications and platforms. In Reference 39 the authors propose a technique for gesture analysis called gesture heatmaps. This technique allows showing, through a colored visualization, how the values of some descriptors (e.g., speed) change along the gesture path. The authors also introduce chromatic confusion matrices, which extend the expressiveness of classical confusion matrices in the analysis of gesture classification performance. These visualizations can be helpful for people doing gesture analysis, for example, to understand in a simpler way the causes of recognition errors. Table 1 summarizes the main attributes of the tools and techniques for gesture design that we have discussed in this section. It is worth noting that most of the work focuses on using a single gesture recognizer. PolyRec GDT supports multiple recognizers and is easily extensible in that respect. Various tools, on the other hand, offer functionality to help the user design the gesture set, but none of them offer a technique that is exactly equal to the one of PolyRec GDT. The one that comes closest is the one proposed in Reference 37. Although various systems offer functionality to simply generate a gesture recognizer, none offer a dedicated feature for Android developers like that of PolyRec GDT, which allows to have a working recognizer in an Android touch application by just pasting the source code generated by the tool.

Gesture recognition methods
Most of the applications for the latest mobile devices support a gesture-based interaction. In the last years, the HCI scientific has proposed many gesture recognizers (in this context we refer to a "gesture" as the path traced on a touch-sensitive surface by a finger or a stylus). Among the first works, we can remember those based on dynamic time warping (DTW) 40,41 and Rubine, 42 the first widely used gesture recognizer, a feature-based linear classifier. The various techniques used for gesture recognition include those based on Hidden Markov Models, 43 neural networks, [44][45][46] feature-based statistical classifiers, 47 or a combination of them. 12 More recently, among the template-based matching algorithms, the "$-family" of gesture recognition algorithm has emerged. It started with $1, 21 an algorithm for unistroke gesture recognition, with a focus on simplicity and fast implementation, which compares points in the sampled strokes for each gesture and is scale and (optionally) rotation invariant. Although more accurate than Rubine, $1 has some problems, for example, a problem with uniform sampling has been reported: "misalignment" in one-to-one correspondence and "unwanted flexibility" in elastic matching. $1 has since been extended several times, for example, for multistroke gesture recognition ($N 48 ), for increased speed and accuracy as in Protractor 49 and $N-Protractor, 50 for articulation-invariant recognition ($P 22 and the faster and more accurate $Q 51 ), and for people with low vision ($P+ 52 ).
Methods of other authors inspired by the same philosophy include: Quick$, 53 which uses hierarchical clustering and branch and bound to increase efficiency and avoid making comparisons with all training templates; 1¢ 54 which improves speed compared with $1; Penny Pincher 55 which uses only addition and multiplication for recognition and improves the recognition rate and speed of previous $-family recognizers; Jackknife, 56 a set of techniques for pen and touch, Wii Remote, Kinect, Leap Motion, and sound-sensed gesture recognition; $3, 57 Protractor3D 57 and 3 cent 58 which allow 3D gesture recognition. 1 59 and 1F 60 are filters to improve the performance of recognizers in specific conditions.
Other recognizers based on geometric 3,23 functions have also been proposed.

THE POLYREC RECOGNIZER
PolyRec 8 is scale and (optionally) rotation invariant and uses a nearest neighbor approach to recognize an unknown gesture, that is, the distance between the unknown gesture and each template is calculated, then the gesture is classified into the class of the template at shortest distance. The main feature of PolyRec is to represent a gesture through a polylineidentifying its main movements. An advantage of this representation is to only use a small sequence of dominant points to calculate the distance from another gesture, improving recognition efficiency. Summarizing, the distance between two gestures is calculated through the following three steps: 1. Preprocessing: the two gestures are approximated to a polyline, in order to extract their dominant points. A polyline can be composed of one or more segments; 2. Alignment: the two polylines are aligned, that is, their segments are possibly split until the polylines are composed of an equal number of segments; 3. Calculation: the distance is found by summing the contribution of the distances between each pair of corresponding segments.
To find the distance between two gestures, the segments of each polyline are treated as vectors, characterized by magnitude and direction, and compared in pairs with those of the other polyline. The distance is obtained by summing the contribution of each pair of segments.

3.1
Step 1: Preprocessing An input gesture contains a variable number of points sampled from the device. Our preprocessing uses the Douglas-Peucker reducer 61 (DP, in the following) to extract the dominant points, where the gesture shows a greater curvature. The result is a broken line, which we call a polyline. DP recursively works as follows: initially the polyline is formed by the segment connecting the endpoints of the gesture; then the furthest point of the gesture is added to the polyline, and DP is recursively executed on the two parts of the gesture. The point is added only if it is at a distance greater than a tolerancethreshold.
In order to obtain a smoother polyline, with segments representing the main movements intended by the drawer, an iterative LineFusion procedure (see Reference 62) is executed. At each iteration, the point between the segments having the smallest direction change is removed. The process ends when the smallest direction change exceeds a minimum threshold . Figure 1 shows the result of preprocessing on an arrow sample gesture. As shown in the figure, after the DP step, LineFusion refines the polyline in order to better identify the main movements of the gesture.
The complexity of preprocessing is O(N 2 ), with N being the number of points in the input gesture. It is worth noting that preprocessing is executed only once on each template.

3.2
Step 2: Alignment After preprocessing, a polyline is composed of a variable number of points. In step 2, we equalize the number of points of the polylines to compare. To do this, we firstly establish a mapping between the existing points. If a point is unmapped, a corresponding point is added to the other polyline. The mapping is based on the similarity between the points.
To perform the mapping we use a modified version of Needleman-Wunsch 63 (NW, in the sequel), an algorithm used for the alignment of strings. NW calculates an optimal alignment between two strings based on a scoring scheme. Here we use the same idea to map points. In particular, the effect is to match two points of the polylines only when they are quite similar. Similarity is determined using two features: • pos(p): the position of the point within the gesture, calculated as the ratio between the length up to p and the total length; • angle(p): the difference of direction between the two segments of the polyline that meet at p.
Given two polylines P with points p 1 , … , p n and Q with points q 1 , … , q m , the alignment can result in: • Match(p i , q j ): points p i from P and q j from Q are matched, that is, the pair (p i , q j ) is added to the alignment; • Insert(P, q j ): a new point mapped to q j is added in P, that is, the pair (−, q j ) is added to the alignment; The insertion of a point in a polyline occurs between the closest points having a correspondence in the other polyline. For instance, from the alignment … (p m ′ , q m ′ )(−, q j )(p m ′′ , q m ′′ ) … , we determine that a new point p mapped to q j must be added in P between p m ′ and p m ′′ . The ideal position of the new sampled point p in the subgesture delimited by p m ′ and p m ′′ is established proportionally to the position of q j in the subgesture delimited by q m ′ and q m ′′ .
An example of polyline alignment is shown in Figure 2. The figure shows the result of the application of alignment on two gestures reduced to two polylines P and Q. Before alignment (see Figure 2A), Pand Q contain 8 and 9 points, respectively. The alignment returned by NW is the following: . As a result, two vertices corresponding to q 3 and q 7 are inserted in P and one corresponding to p 2 in Q (see Figure 2B), thus obtaining two polylines composed of 10 points each.

3.3
Step 3: Distance In this step, a polyline is considered as a sequence of vectors. Each segment of the polyline is transformed into a vector having magnitude equal to its relative length and direction equal to its slope. The distance between two polylines is calculated by summing the magnitude of the vectors obtained from the difference of each pair of vectors in the corresponding sequences of vectors. A penalty increasing the distance proportional to the number of mismatches is also used. Our recognizer can be used both in a rotation sensitive and in a rotation invariant context. In the latter case, the indicative angle 21 of the gesture is determined. In both cases, we adopt golden section search (GSS) to determine the distance at the best angle.
The three steps require the setting of parameters. We performed a tuning on a dataset composed of 400 curves. 64

Recognition accuracy
We evaluated our recognizer in a rotation-invariant context by testing it on two different unistroke datasets. In addition to Protractor 49 as in Reference 8, we compared its performance also with $P, 22 $P+, 52 $Q, 51 and Jackknife. 56 The main features of the datasets are summarized in Table 2. The first dataset is the one introduced to evaluate the $1 recognizer. 21 The second dataset is the one presented in Reference 65.

Methods
We performed both a writer-dependent test and a writer-independent test similar to the ones used in Reference 21 and 22.
For the writer-dependent test, the recognition rates were computed individually for each participant and each articulation speed. For each gesture class and speed, a random sample was selected for testing, and T different samples were randomly selected for training. For each value of T, the recognition was performed 100 times for each method, and results were averaged into a recognition rate per participant. For the $1 dataset the number of training samples per gesture class T varied from 1 to 9, this resulted in a total of 10 participants × 3 speeds × 9 values for T × 100 repetitions for each T × 16 gesture classes × 6 recognizers = 2,592,000 recognition tests.
For the ILGDB dataset the number of training samples per gesture class T varied from 1 to 4 (since there are writer/class combination with only five samples), this resulted in a total of 11 participants × 1 speed × 4 values for T × 100 repetitions × 21 gesture classes × 6 recognizers = 554,400 recognition tests.
In the writer-independent test, for each articulation speed, a random participant was selected for testing and P different participants for training. For each class, a random gesture was selected for the testing participant, while T samples were randomly selected for each gesture class from each training participant. For each value of P and T the recognition was performed 100 times for each method and results were averaged into a recognition rate for each P value.
For the $1 dataset the value of P varied from 1 to 9, and the value of T varied from 1 to 9, this resulted in a total of 9 values of P × 3 speeds × 9 values for T × 100 repetitions × 16 gesture classes × 6 recognizers = 2,332,800 recognition tests.
For the ILGDB dataset the value of P varied from 1 to 5, and the value of T varied from 1 to 9, this resulted in a total of 5 values of P × 1 speed × 9 values for T × 100 repetitions × 21 gesture classes × 6 recognizers = 567,000 recognition tests.
Furthermore, as a comparison, we have also measured the accuracy of PolyRec when the samples, rather than randomly, were selected using the k-medoids 66 clustering algorithm (see Section 4.3.3 for more details on the corresponding functionality of PolyRec GDT).
For all recognizers, we used the default parameters defined by the authors in their published implementations † . The experiment was conducted on a HPE server equipped with an Intel Xeon Gold 5118 CPU at 2.30-GHz running Ubuntu 20.04 operating system and the Java Run-Time Environment 11.

Results
The results of the recognition tests are shown in Figures 3 and 4, for the two datasets, $1 and ILGDB, respectively. In the case of the $1 dataset, PolyRec is the method with the highest accuracy in the writer-independent test, while it is the second in the writer-dependent case, since $P+ has higher performance, although at the cost of more than doubled recognition times. In the case of the ILGDB dataset PolyRec is also the method with the highest accuracy in the writer-independent test, while in writer-dependent case Jackknife is the method with the best accuracy when only one training sample is used, whereas PolyRec is the most accurate when two or more training samples are used.
As regards the recognition times, it can be noted that Protractor, although the least accurate, is clearly the fastest method, while $P and $P+ are the slowest methods. PolyRec and Jackknife have very similar performance and are most often the fastest methods (excluding Protractor), while $Q is faster when there are many training samples. This is because the optimization of $Q has a high initial cost which, however, allows an almost linear trend with increasing training samples. Furthermore, the results show that when, instead of using a random selection, the training samples are selected with the k-medoids algorithm (see Section 4.3.3), the accuracy is much higher for both datasets in the case of a single training sample. When the number of training samples increases this difference decreases until it approaches zero.

polyrec gesture design tool
PolyRec Gesture Design Tool (or PolyRec GDT in short) is a software designed to help mobile application developers in the creation of the gesture sets used in their applications. The goal is to allow designers, even those who have no experience with the PolyRec recognition technology (or other recognizers), to create good sets of gestures.
The system allows the developer to define the gesture classes that he intends to use and to add the corresponding gestures, both directly on the PC or by drawing them on a smartphone. In the following, we will call template a gesture that belongs to a class.
In addition to this basic functionality, PolyRec GDT provides functionality to help to understand if there are critical points in the gesture set that is being built. A study on the gesture design process 67 has revealed that designers find it difficult to discover and solve problems in the gesture sets they create: PolyRec GDT tries to help in this by providing feedback on gesture set quality and by showing possible problems.
The tool provides warnings and statistics to inform the designer and allows him to make changes if he deems it necessary. Some analysis on the gesture set is made when a template is added, while others can be launched by the developer when he deems appropriate. The data provided by the software are both of high level (e.g., messages as "too similar classes") and of low level (e.g., distances and score tables), so as to support users with varying degrees of knowledge of the PolyRec recognizer.
An issue in designing a gesture design tool is whether it should be dependent (or independent) on the gesture recognizer. The advantages of recognizing independence are obvious: the tool can be used with any recognition technology during both the design phase and in the deployment phase. On the other hand, by using features that are dependent on the recognizer, the tool can offer better advice in the design phase, but at the cost of nonportability. PolyRec GDT is equipped with PolyRec, but in the settings it is possible to choose whether to use PolyRec, $P, $P+ or $Q, and it is relatively simple to add support for other recognizers. However, the provided suggestions are dependent on the selected recognizer, so in order to exploit them correctly, it is necessary to use the same recognizer both in the design and in the application performing recognition. In addition, the effectiveness of these suggestions may vary depending on the characteristics of the recognizer and its calculated scores, and some features may not be supported, for instance, the choice per template of recognition mode (rotation sensitive or invariant). Moreover, PolyRec GDT offers some functionalities, such as the possibility to create and modify training templates and to display tables with the geometric features of the gesture, which can be used with any recognizer (even those not integrated into the tool).
Even though PolyRec is a unistroke recognizer, it allows simple multifinger gestures. In this case, the number of fingers used is recorded and the comparison is made, on templates with the same number, by using the track of a single finger (so this is only feasible for gestures in which the fingers remain united).
The tool has been implemented in Java and allows easy integration of the recognizer in Android applications. PolyRec GDT consists of two main display modes: the dashboard view and the template view. The dashboard mode represents the main screen (see Section 4.3) that gives a complete view of the gesture classes and contains the main commands to operate on it. The template view (see Section 4.4), that can be accessed from the dashboard, is the one in which it is possible to add or perform operations on a single template, also through companion apps for android devices (see Section 4.1). In both displays modes a menu to access common functionalities (e.g., save, load) and a title bar that provides information on the current location and allows to return to the dashboard are shown (see the top of Figure 7). The following sections describe these components in detail.

PolyRec companion apps
In addition to allowing gestures to be entered via the pc cursor that is typically controlled by mouse or touchpad, PolyRec GDT provides two Android apps that allow the user to enter gestures directly on a smartphone touchscreen, in order to allow the designer to enter gestures performed in a natural way. Both apps have similar functionalities and can be paired and used from the template view. Figure 5 shows the drawing interface of the Bluetooth app. The WiFi app has a similar interface.
The two apps differ for the method with which they communicate with PolyRec GDT: one uses Bluetooth and the other one uses WiFi (TCP/IP). PolyRec GDT can also send the Bluetooth app installation file directly to the Android device, while the WiFi app can be only downloaded from the PolyRec GDT site.

Menu
The menu, shown in Figure 6, can be called independently of the current application view. From top to bottom it allows sending the companion app via Bluetooth to an Android smartphone, to export the Java code that allows to use the recognizer, to import sample gestures from predefined datasets, to create a new gesture set, to open an existing gesture set (also by adding its gestures to the current set), to save the gesture set, to change application options (used gesture recognizer and warning thresholds) or to exit it.

Export recognizer
This feature allows the user to export the Java code that allows the gesture recognition, using the selected gesture recognizer, for the current gesture set. The user can simply copy the generated code to the clipboard and paste it into the Java class where he needs to perform gesture recognition. In order to simplify the recognizer usage and sped-up operations, the generated Java code directly contains the templates (codified as strings) and a handleGesture() method stub.

Dashboard view
The dashboard view is composed of several components, as shown in Figure 7. In particular from top to bottom: • Message bar: this bar shows messages about the last operation performed on the gesture set. For some operation, the system also uses popup windows to display longer messages. • Operations bar: this bar contains buttons that allow the execution of commands on the gesture set. In particular, it allows, from left to right, to add a new gesture class, to merge two or more gesture classes, to delete all classes, to check the similarity between them (see Section 4.3.1), to display the gesture set features -In each of the following columns, one of the class templates is shown, and by clicking on it, it is possible to switch to the template view. There are also buttons that allow to: move a template to another class, delete the template, display if the template is rotation invariant or rotation sensitive and possibly change this property. -The last column contains a ⊕ button that allows adding a new template to the class (by opening the template view).
• Status bar: this bar contains summary information on the number of classes, the number of templates, and the average number of templates per class.

Gesture set classes similarity analysis
Long et al. 67 carried out an experiment in which the participants were asked to design a set of gestures and try to get the best possible recognition accuracy for the Rubine 42 recognition algorithm. No participant managed to exceed the recognition rate of 95.4%. A typical problem that was observed was that the participants tended to add strokes that were too similar to those already defined. If two gestures are similar to the recognizer, they are more likely to be confused. In order to help designers find this potential problem, PolyRec GDT provides a feature that allows the user to check for similarity issues between the gesture set classes.
By pressing the "verify similarity" button in the dashboard, a window is shown (see Figure 8) that displays the list of "conflicts" between templates belonging to different classes: a conflict is found if two templates have a similarity that exceeds a certain threshold. The system compares each template with all the templates belonging to the other classes, if the comparison score exceeds the threshold (by default 88%) then a warning is displayed, which indicates that there is an anomaly that may require actions.
Within the window, it is possible to change the similarity threshold and immediately recheck the conflicts with the new threshold.
Besides that, it is also shown what in PolyRec GDT has been defined as the similarity set score of the gesture set. It is equal to the maximum similarity score found when performing the comparisons. As it is defined, the goal is to keep this score as low as possible: having a low similarity score indicates that the classes are properly distinct. Therefore, if the similarity score of the set exceeds the default threshold, the system shows a warning.

4.3.2
Gesture set features table The system allows the user to view the values of 13 (mainly geometric) features for each of the templates of a class or of the whole gesture set. As shown in Figure 9 the values are presented in a table that contains a row for each template and that allows to export the data as a .csv file or to show it in a chart.

F I G U R E 9 Features table
These 13 features can be useful for understanding the complexity and differences of classes and templates and are calculated after normalizing the gestures (i.e., scaling them to the same size).
Some of the features (indicated with fn in the list) are those proposed by Rubine 42 and are commonly used in the study of gestures.
1. number of segments of the polyline generated by the PolyRec recognizer; 2. total sum of angles or total curvature (f10); 3. length of the gesture path (f8); 4. distance between the first and last point (f5); 5. area of the bounding box; 6. length of the bounding box diagonal (f3); 7. angle of the bounding box diagonal (f4) 8. cosine of the initial angle (f1); 9. sine of the initial angle (f2); 10. cosine of the final angle; 11. angle of the line that joins the first and the last point of the gesture; 12. cosine of the angle of the line that joins the first and the last point of the gesture (f6); 13. sine of the angle of the line that joins the first and the last point of the gesture (f7).
It is also possible to show these values only for a single template by calling this feature from the template view.

Automatic templates selection
The performance of a recognizer is measured primarily by the recognition rate, which is an estimate of the percentage of gestures that are expected to be correctly recognized. In addition to the recognition rate, there are other variables, such as the time required for classification and the amount of system memory used to store the gesture set. Their importance is particularly relevant for mobile computing where resources are limited. Some studies have demonstrated the benefits of clustering for restructuring the gesture set and improving recognition accuracy and performance. For example, Ouyang and Li 68 used clustering to merge similar gesture patterns provided by many contributors in an attempt to build a large, constantly evolving dictionary of gestures for touchscreen mobile devices. Keskin et al. 69 used it as a preprocessing step to reconfigure the number and structure of gesture classes according to the similarity between templates in the training set. Inspired by the success of these approaches that use clustering algorithms to improve recognition performance, we decided to use clustering techniques also in PolyRec GDT. If a recognition algorithm based on template matching is used for real-time applications, it is often necessary to limit the number of templates with which the user input is compared in order to limit the required computational resources. It is then necessary to choose how many and which templates to select for each class. This is because different classes require a different number of templates: for example, some handwritten digits have more variations than others (typically more variations in 4 than in 0). For this reason, PolyRec GDT provides an automatic templates selection feature which allows the user to reduce the number of templates in a gesture class by automatically selecting the most representative templates. Given the desired number of templates k, the system calculates, using the selected gesture recognizer, the distance between each template and the others within the class, then applies the k-medoids 66 clustering algorithm to split the templates into k clusters and for each of them selects a template (medoid) to be kept. Alternatively, the system also offers the possibility of dividing the existing class into new k classes corresponding to the found clusters.
The desired number of templates k can be selected by the user. Also, since a developer often does not know which k number is the most suitable, the system proposes to the user a k value. This k is calculated by using the silhouette 70 analysis on different k values in order to select the "better" number of clusters.
The issue of selecting the optimal subset of gestures has already been discussed in the literature. In Reference 71 the authors present an intelligent template selection for the $-family recognizers that makes use of a random mutation hill climb algorithm and a genetic algorithm to search for a reduced sets of templates that minimize recognition error and shows that the error rate can be reduced compared with random template selection. In our opinion, however, techniques based on error rate reduction are reliable when applied to a dataset with a substantial number of gestures (by different users). Since PolyRec GDT is aimed directly at application developers (rather than at gesture recognizer designers or gesture dataset creators), we think it is likely that it will deal with a small number of templates drawn directly by the developer. In this case techniques based on error rate reduction could cause undesired side effects, for example, a developer may intentionally add to a gesture class a single template very different from the others in order to support a different drawing style, but a methodology based on error rate reduction could remove it, since it does not help in recognizing the other (many) gestures in the set. The technique we used, instead, being based on k-medoids, has a better behavior in this case.
The usefulness of this function for PolyRec has been analyzed in Section 5.

Template view
The template view can be accessed from the dashboard to: 1. add a new template; 2. perform recognition tests; 3. view an existing template ad its details.
The central part of the view is always used to show a gesture trace, either a real-time one drawn by the user (1, 2) or a template (3).

Add a new template and recognition tests
In this mode, the user can add a new template to a class, either by drawing it with the system cursor or through the companion app (by clicking the corresponding buttons at the top of the view). After the gesture has been performed (see Figure 10) the system compares it with the templates that are already part of the gesture set and displays the result in the message bar at the top, and shows a warning if the recognized class is different from the class in which the gesture would be added. Moreover, on the right side, there is a button that allows the user to view the score table. This table shows the distance and the score obtained by comparing the just inputted gesture with those already present in the gesture set, limiting it to the most similar one for each class. This allows the user to detect, for example, possible recognition problems. The table can also be used to generate a chart as shown in Figure 11. If the user is satisfied, he can add the gesture to the current class using the corresponding button at the bottom of the view. He can also change the number of fingers/pointers associated with the gesture and whether it should be rotation invariant/sensitive by using the corresponding buttons. Moreover, on the left side, the previews of the templates belonging to the same class are displayed, and it is possible to click on them to view them in detail (see next section).

F I G U R E 10
Template view-Add template (after a gesture has been performed) F I G U R E 11 An example of a score table and of the corresponding chart When the user chooses to perform a recognition test from the dashboard, an interface similar to the one described above is shown, with the only difference that the left and bottom bars are missing.

Existing template view
The application enters this mode when the user saves a new template or when the user clicks on the preview of a template.

F I G U R E 12 Template view-Existing template
As shown in Figure 12, the gesture is displayed at the center in the dimensions in which it was drawn. At the top of the view, buttons show and allow to modify the number of fingers/pointers of the gesture and whether the gesture is rotation invariant. By using the toolbar at the bottom the user can change how the gesture is visualized by changing zoom, rotation, translation. On the left side, the other templates belonging to the same class of the current template are displayed. On the right side, there are buttons to: • enable/disable the visualization of the gesture path; • enable/disable the visualization of the polyline created, from the original gesture, by the PolyRec recognizer; • enable/disable the visualization of the vertices of the polyline created by the PolyRec recognizer; • visualize a timed animation of the gesture, shown at the same speed with which it was entered; • show the gesture features table, as described in Section 4.3.2; • show the score table, as described in the next section.

EVALUATION OF THE AUTOMATIC TEMPLATES SELECTION
In order to evaluate the usefulness of the automatic templates selection method in a real-life case, the work done by the developers of a gestural keyboard for entering programming code on mobile devices 72 has been analyzed.
The developers have defined a set of gestures consisting of 14 classes (each one represents a programming construct), and have then collected the gestures from five people. Each of them performed the gesture of each class four times, for a total of 280 samples. The developers, not having a tool like PolyRec GDT, based the choice of templates to be used in their application on their perception (3 templates for each class).
We have therefore decided to evaluate the quality of the templates selected with the PolyRec GDT automatic templates selection method by comparing them to the ones selected by the developers.

F I G U R E 13 Error rate comparison between template selection methods
The accuracy of the recognition on the gesture set collected by the developers was therefore measured for three different conditions regarding the choice of the templates: three random templates for each class, the templates chosen by the developers, three templates for each class selected using automatic templates selection method. The accuracy was calculated by averaging the one obtained individually for each of the five people that performed the gestures in the dataset. For each person, the recognition test was repeated 100 times by selecting a random gesture and performing the recognition for the three conditions (selecting the templates from those of the other four participants).
The results are shown in Figure 13. In the case of the randomly selected templates, the error rate is 2.69%, while it is 2.50% in the case of the templates selected by the developers, with only a small accuracy increase. In the case of the templates selected with the automatic templates selection method, instead, the error rate dropped to 1.66%, thus showing that the use of PolyRec GDT may provide useful help to developers.

USABILITY EVALUATION
In order to evaluate PolyRec GDT, a user study has been performed in which participants were asked to use it to support them in the addition of gesture-based functionalities to an existing mobile application.

Participants
For the experiment, 12 participants (four females) between 21 and 29 years old (mean 24.25, standard deviation 2.53) were recruited. All of them were university students in computer science who agreed to participate for free. All of them already had experience with touchscreen devices, using their smartphones every day. They were divided into two groups, the first one consisting of the six participants who declared they had a good knowledge of Android programming, while the other composed of the other six participants who declared no/low knowledge of Android programming (but a good knowledge of Java).

Apparatus
Each participant performed the experiment on their notebook, appropriately configured for the development of Android applications. They also used their Android smartphone to draw gestures with PolyRec GDT and as a development device. An alternative device was provided to the few participants without a personal Android smartphone.

Procedure
In the experiment invitation mail, the participants were asked to appropriately configure their Notebook for Android applications development by installing Android Studio and by compiling/running two stub example applications, in order to download the required dependencies and check the proper functioning of the development environment.
Moreover, they were also asked to read and try to comprehend the code that handles the GestureOverlayView component in the first example. Before starting the experiment the participants were asked to fill out a form with the following information: age, gender, English knowledge level, experience level with smartphones, Java programming experience, and Android programming experience.
Then, they received a brief tutorial showing the functionalities PolyRec GDT. They were also asked to install PolyRec GDT on their notebook and the PolyRec GDT WiFi app on their smartphone and try PolyRec GDT functionalities. They were then asked to download and run the PolyRec Android Example from GitHub, open it in Android studio, run it on their device, and read the main parts of its source code. They had to perform the same operations for the "Pairs Game" Android application, a simple game in which the user has to match two pairs of cards depicting geometric figures.
Then the objectives of the experiment and its procedure were described to them. They were also given an instruction sheet, containing the description of the three tasks to perform to complete the experiment. The experiment started after the experimenter ascertained that the participant had well understood the procedure.
Task 1 consisted of using the PolyRec GDT tool to create a gesture set for the videogame "Pairs Game." The gestures must be used to perform the following actions: • up: moves the cursor up; • down: moves the cursor down; • left: moves the cursor left; • right: moves the cursor right; • guess circle: bet that in the selected position there is a circle; • guess square: bet that in the selected position there is a square; • guess triangle: bet that in the selected position there is a triangle; • guess check: bet that in the selected position there is a check symbol.
When the participant is satisfied with the gesture design he must save the gesture set in a file called SurnameName.pgs. The time limit for this task is 15 min.
Task 2 consisted of modifying the "Pairs Game" videogame by: hiding the buttons for the eight operations (up, down, left, right, guess circle, guess square, guess triangle, guess check), adding a GestureOverlayView to the layout of the videogame to allow the user to perform a gesture in the application view area and see its trace, make the application log/print, when the user lifts the finger, the coordinates of all the points of the performed gesture. When the modification is completed without errors the participant has to commit its changes to the application. The time limit for this task is 25 min. Since task 3 depends on this task, in case the participant was not able to complete it successfully, an operator helped him complete it before starting task 3.
Finally, task 3 consisted of continuing the editing of "Pairs Game" by recognizing the gesture performed by the user using the gesture set defined in task 1 and performing the corresponding game operation (up, down, left, right, guess circle, guess square, guess triangle, guess check). If need the previously defined gesture set may be changed. When the modification is completed without errors the participant has to commit its changes to the application (and the gesture set, if modified). The time limit for this task is 25 min.
After the end of the experiment, participants were asked to fill out a System Usability Scale (SUS) questionnaire. 73 The SUS questionnaire consists of 10 statements to which the user assigns a score on a scale from 1 (strongly disagree) to 5 (strongly agree). The final score of the SUS ranges from 0 to 100. A higher score indicates greater user usability. Moreover, after the experiment, user free-form comments and suggestions were also collected. In particular, at the end of the questionnaire, there was a blank space where each participant could write his/her comments and suggestions about PolyRec GDT.

Design
The experiment was a one-way with one between-subjects factor design. The one factor was the group (nonexpert and expert). For each task, the experiment-dependent variables were the task completion time and the task success/failures. To evaluate user satisfaction the SUS was used.

F I G U R E 14
Mean completion times and error rate for the three tasks by group. Error bars show the standard deviation

Results
All participants completed the experiment. The experiment took each of them, excluding the introductory phase, up to two hours. We tested for significance using a repeated measures analysis of variance (ANOVA). 74 The overall task completion times per task and group are shown in Figure 14. As it can be seen, all participants completed task 1 successfully and the two groups used similar times, 7:54 min (SD = 2:44) and 7:50 min (SD = 1:06) respectively for the nonexpert and expert groups. From the ANOVA resulted that the difference between the groups on completion time was not statistically significant (F 1,10 = 0.004, p > .05). This provides a good sign on the usability of PolyRec GDT.
As for task 2, in the nonexpert group, two-thirds of the participants failed to complete the two tasks in the 25 min allocated time, with an average time of 22:35 min (SD = 3:47), while in the expert group all participants completed the task with an average time of 11:43 min (SD = 2:00). From the ANOVA resulted that the difference between the groups on completion time was statistically significant (F 1,10 = 38.817, p = .0001). The difference on task failures was also statistically significant (F 1,10 = 10.000, p = .0101).
As for task 3, in the nonexpert group, two-thirds of the participants failed to complete the two tasks in the 25 min allocated time, with an average time of 21:45 min (SD = 6:13), while in the expert group all participants completed the task with an average time of 8:54 min (SD = 5:06). From the ANOVA resulted that the difference between the groups on completion time was statistically significant (F 1,10 = 15.345, p = .0029). The difference on task failures was also statistically significant (F 1,10 = 10.000, p = .0101).
As would be widely expected, there was a significant difference between expert and nonexpert participants, although the results of task 1 show that all participants were able to define their set of gestures. The results on tasks 2 and 3 show that despite the initial tutorial, modifying an Android application was too difficult for participants with little experience in the field, even though one third of them succeeded in the available time.
On the other hand, for experienced participants, adding gesture functionalities took only a few minutes, which would have been instead impossible in the allocated time without the support of PolyRec GDT.

User satisfaction and free-form comments
The average SUS score was 65.83 for the nonexpert group (SD = 25.33) and 76.25 (SD = 6.85) for the expert group with a grand mean of 71.04 (SD = 18.51). This can be considered a good result. 75 Regarding the free-form comments, most nonexperts participants highlighted the difficulty of completing tasks 2 and 3 due to their lack of experience with Android programming. Comments about PolyRec GDT were mostly positive, such as one participant who said "I found PolyRec GDT software extremely useful if you want to implement an application that uses gestures. If I were to implement such an application, I would use this software." Most of them liked the possibility to generate the Java code to recognize the gestures, stating that it greatly sped up the implementation. Some participants appreciated the possibility of making gestures, instead of using the mouse, directly on a smartphone thanks to the companion app. In this regard, however, some participants noted an instability on their hardware of the WiFi connection with the app, which did not compromise its use but made it necessary to repeat the connection multiple times. One user suggested offering the option of defining a gesture as "bidirectional," similar to how it is possible to define one as rotation invariant, so that when desired it will not be necessary to add a second gesture with the same shape but the opposite direction. Finally, one user suggested giving the possibility to sort the classes in the dashboard in a custom way (they are now in alphabetical order), while another one complained about the ambiguity of the ⊕ button when adding a new template.
When asked how they would perform the same tasks without having PolyRec GDT available, they gave two types of answers: a minority who stated that they would not know how to do it, and those who stated that they would instead modify an existing app (or create a new one) in order to capture their gestures. Most of them said they would modify an existing app, either the first stub app (which already has a GestureOverlayView) or the PolyRec Android Example available on GitHub, in order to be able to capture gestures made on the smartphone. In this respect, most said they would simply print on the log console (Logcat) the x and ycoordinates of the points composing the gesture, and then copy and paste that data into Java code. Instead, another party stated that they would offer the ability to save the gestures performed as files on the smartphone, which they would then copy to the computer. When asked, they all answered that, even after learning about it thanks to PolyRec GDT, they would not perform any particular operations to choose the best gestures for the recognition, considering it too difficult to perform it manually. This could also be due to their low experience about gesture recognition, as well as the difficulty to perform them without a support tool. In fact, on this subject, none of them answered that they would search online for a tool for capturing/managing gestures.

Control experiment
In order to assess the difficulty of performing the above tasks without PolyRec GDT, we programmed a control experiment. For this purpose we started a pilot study with three "expert" participants (two females) between 21 and 28 years old, university students in computer science who had taken the course on Android programming and who had not already participated in the previous study. They followed a procedure similar to the one for the experiment described above, with the difference that they were not instructed on how to use PolyRec GDT (but just PolyRec, using the examples and documentation on GitHub) and were asked to perform the tasks using the methodology they thought appropriate. We increased the time available to complete the tasks, setting it at 50 min for all of them. The time taken for the three tasks by the participants was respectively 47:40 (SD = 2.05), 15:20 (SD = 4:44), and 45:20 (SD = 8.05) min. One participant was unable to successfully complete task 1 and 3, while all successfully completed task 2.
As was to be expected, not having PolyRec GDT available caused a significant increase in the time required to complete tasks 1 and 3, in which in the previous experiment PolyRec GDT was respectively used to design the gesture set and to generate the Java code for gesture recognition, with an average time of about six or five times higher than in the expert group of the previous experiment. In task 2, which does not benefit from the use of PolyRec GDT, the difference is small. Even for those who successfully completed the tasks, we saw less attention paid to the quality of the created gesture set, with typically only one sample used for each gesture class.
Regarding the used methodology, the two participants who successfully completed all the tasks modified the PolyRec Android Example in order to capture their own gestures and print their points on the log console (Logcat). The participant who failed to complete tasks 1 and 3 instead tried to create a new app to capture the gestures, but failed to capture the points correctly (resulting in task failure).
In comments after the experiment, participants stated that since they had never built Adroid apps with gesture recognition, they had to spend time figuring out how to operate and searching for documentation/answers online. When asked, one of them stated that while searching online he also tried to see if he could find an app or website to capture the gestures on his smartphone, but desisted after a few attempts, while the other two participants stated that they did not think about it.
Considered the poor results of the above described study, we felt it was useless to continue with a full-scale experiment.

Discussion
From the results and comments of the participants, it can be seen that PolyRec GDT was appreciated and it allows to define the gesture set for a simple smartphone application in a short time. The functionality offered by PolyRec GDT to export the Java code needed to use the recognizer also allows users familiar with Android programming to add gesture capability to an existing application in a few minutes, while without a similar tool the time required would have been much longer. As expected, however, it was difficult for participants not familiar with Android programming to integrate gesture recognition into existing software. Nevertheless, the experiment showed that they too were able to define their gesture set. In addition, the control experiment confirmed that the use of PolyRec GDT allows for a better and faster workflow compared with a development without a gesture management tool. It would also be useful to increase developer awareness of such tools, for example, it might be useful to have a reference gesture management tool mentioned in the documentation (and/or code comments) of gesture recognizers.

CONCLUSIONS AND FURTHER WORKS
We presented PolyRec Gesture Design Tool (PolyRec GDT), a tool that allows the definition of gesture sets for mobile applications that uses the PolyRec gesture recognizer. PolyRec GDT supports the entire gesture set design process, from collecting gestures, to analyzing and exporting the set created for use in a mobile application together with the PolyRec recognizer. PolyRec GDT also allows to detect similarities between gestures in order to help the designer in detecting possible ambiguities, and thanks to a clustering technique it can automatically select the most representative gestures for each class of gestures, in order to allow the right compromise between execution speed and recognition accuracy. The usefulness of this function is supported by experimental results that have shown greater accuracy than the same operation done manually by developers or randomly by software.
In addition, an experiment was carried out to evaluate PolyRec GDT, in which participants were asked to use it to support them during the development of a mobile application. The results showed that adding gesture functionality to an existing application only took a few minutes for developers with mobile development experience, which would have taken much longer without PolyRec GDT.
Future work will include small improvements to user interaction with PolyRec GDT following feedback from its users, and support for the iOS system (in addition to Android) to support apps for both major mobile operating systems.

ACKNOWLEDGMENT
This work was partially supported by two grants from the University of Salerno:300392FRB19COSTA and 300392FRB20COSTA.