Rapid identification of chrysanthemum teas by computer vision and deep learning.

Seven commercial Chinese chrysanthemum tea products were classified by computer vision combined with machine learning algorithms. Without the need of building any specific hardware, the image acquisition was achieved in two computer vision approaches. In the first approach, a series of multivariate classification models were built after morphological feature extraction of the image. The best prediction accuracies when classifying flowering stages and tea types were respectively 90% and 63%. In comparison, the deep neural network was applied directly on the raw image, yielded 96% and 89% correct identifications when classifying flowering stage and tea type, respectively. The model can be applied for rapid and automatic quality determination of teas and other related foods. The result indicated that computer vision, especially when combined with deep learning or other machine learning techniques can be a convenient and versatile method in the evaluation of food quality.


| INTRODUC TI ON
The chrysanthemums are the flowering perennial plants with enormous horticultural varieties and cultivars. They are widely used as a raw ingredient of functional food products in China and other East Asian countries, because they have ornamental value as well as functional benefits after consumption. As a healthy functional food, some chrysanthemum plants such as Chrysanthemum morifolium Ramat. and Coreopsis tinctoria are traditionally consumed as teas. Chrysanthemum teas are dried capitate inflorescence of the chrysanthemums, and usually consumed in combination with hot or boiling water. They are considered to have many beneficial effects, most typically are anti-inflammation (Li, Yang, et al., 2019;Zhang, Shi, Zhao, Chai, & Tu, 2013) and antioxidant (Li, Yang, et al., 2019;Wang, Xi, Guo, Wang, & Shen, 2015;Yang et al., 2011) and have been applied in treating a series of symptoms such as blurred vision, dermatitis, and eye itch for more than 2,000 years. In our previous studies, it is indicated that different chrysanthemums have different phytochemical compositions together with different health functions (Li, Hao, et al., 2019;Li, Yang, et al., 2019). Therefore, differentiation of chrysanthemum teas can be useful in the quality assurance of its related products.
It is generally accepted that the cultivar, growing environment, and cultivation technique correlates to the chemical compositions, biological properties and final product quality of a botanical (Lu, Jiang, et al., 2014;Lu, Gao, Chen, Charles, & Yu, 2014). A unique cultivation characteristic of chrysanthemum tea products involves in their harvest time. Two different types of chrysanthemum tea products, that is, the fetal and the common chrysanthemum teas, were marketed according to different harvest time. The fetal and common chrysanthemum teas are respectively collected before and in full bloom. Variations in compositional, quality and clinical efficacy existed between the two types of chrysanthemum teas according to a series of researches (Tou, Sun, Zhang, Si, & Liu, 2013;Yuan et al., 2015;Zhou, Yu, Ren, & Wang, 2009). For instance, Zhou et al. demonstrated that the harvest time influences the content of chlorogenic acid in chrysanthemum tea products and suggested that the fetal chrysanthemum harvested before mid-November yielded better quality (Zhou et al., 2009). In addition, the mineral contents also vary with different flowering stages (Tou et al., 2013). The research of Yuan et al. implied that the fetal chrysanthemum possesses better functional effects than a common chrysanthemum tea "Xiaobaiju" (Yuan et al., 2015). On the other hand, fetal chrysanthemum teas are usually smaller in size compared with common chrysanthemum teas due to different extent of pedal stretching. Furthermore, other cultivation characteristics such as the geographical origination and species were closely related to quality and may also influence the size and shape of teas. Mislabeling and improper use of teas harms the consumers' and producers' mutual interest. Therefore, to better evaluate the quality and promote the utility of chrysanthemums, the classification and quality assessment of chrysanthemum teas are needed.
The quality assessment of chrysanthemum teas was usually performed by instrumental analysis techniques, such as by the gas chromatography-mass spectrometry and olfactometry (Luo, Chen, Gao, Liu, & Wu, 2017). Although precise chemical information can be obtained through this approach, it may not be suitable for rapid assay and high-throughput, on-line monitoring due to its relative high cost and long sample pretreatment and analysis time. A computer vision system (CVS) that mimics human vision may be an attractive alternative. By a specially designed image-recording device combined with a series of image processing methods, CVS not only acquire and process visual information but also make intelligent decisions without any human intervention. The CVS has been already widely used in massive product quality inspection and grading, where repeated and monotonous processing of visual information is needed (Vithu & Moses, 2016). Compared with manual operation, it can achieve a fast, reliable and nondestructive analysis. In addition, with the help of artificial intelligence, automated decisions can be made with complex but mathematically assured models, such that objective, accurate, convenient, and rapid quality detections and identifications can be achieved. Such approach is suitable for large-scale, on-line, or atline manufacturing of food products.
The CVS is widely applied as a quality assurance technique for food products nowadays. More specifically, the CVS has been extensively studied over the decades in rapidly examining a series of interior and exterior quality metrics such as the varieties, defects and maturities of fruits in grapes (Xia, Wu, Nie, & He, 2016), bananas (Mendoza & Aguilera, 2004), watermelons (Koc, 2007), and rapeseeds (Kurtulmus & Unal, 2015). It is foreseeable that the CVS can be applied in larger fields of industrial applications. However, due to that there is intensive shape deformation during food processing such as the drying process, it is also interesting to examine whether the CVS could predict the quality of food products such as dried foods.
The deep learning is one of the significant advancements in the field of machine learning algorithms in recent years (LeCun, Bengio, & Hinton, 2015). Inspired by the structure of visual cortex, the deep neural network (DNN) is a successful example of complex artificial neural networks that typically bring high classification accuracy on the unstructured data such as image data. Contrary to the previous procedures with the calculation of a shape, color, and texture data set as intermediate feature set for model training, the DNN introduces the concept of convolution core and pool layer, and implements the classification through series of layered network using the entire picture as input. Due that a relatively large-scaled neural network is applied to simulate human decision process, DNN has already achieved promising results over a wide range of applications involving complicated tasks, such as image classification and speech recognition. Recently, the application of DNN has also been reported in the food industry, such as to evaluate the quality of the fresh-cut lettuce (Cavallo, Cefola, Pace, Logrieco, & Attolico, 2018), dry-cured ham slices (Muñoz, Gou, & Fulladosa, 2019), salmon fillets (Xu & Sun, 2018), and the commercially prepared pureed food (Pfisterer, Amelard, Chung, & Wong, 2018).
In this study, two rapid identification approaches for chrysanthemum tea quality were evaluated. The first approach utilized a regular gel imager as hardware workbench, in association with morphological feature extraction and multivariate classification. The second approach applied DNN to the raw image directly. The resulted model was aimed to achieve the automatic discrimination of different types and characteristics of chrysanthemum teas. This approach may help to understand the relationships between appearance and functional components of chrysanthemum teas in the future.  Table 1.

| Sample collection
The samples were originated in ZheJiang, AnHui, HeNan, and HuBei provinces and Xinjiang Uygur Autonomous Region. These samples were completely dried flowers sealed in plastic bags and stored at ambient temperature. All samples were tested without further processing. In this study, each individual flower bud was treated as an independent instance from the sample. A total of 2,343 and 1,581 instances were collected for shape feature and DNN modeling, respectively.

| Instrumentation
The monochrome contour images were collected from a gel imager.
The gel imager consists of a camera, its accompanying illumination devices, a computer and the corresponding software. The camera and illumination devices were applied to capture an image for each sample. A ChemiDoc XRS + gel imager (Bio-Rad Laboratories) was used as the integrated instrument for camera and illumination in the CVS. The gel imager is a popular instrument used in molecular biology experiments to take snapshots of the gel after electrophoresis. Similar to other specialized CVS hardware, the gel imager is equipped with a high-resolution, high-sensitivity charge-coupled device (CCD) camera as detector. The gel imager was operated at the transmittance mode as demonstrated in Figure 1a

F I G U R E 1 Diagram of image acquisition using a gel imager (a) and arrangement of flower buds (b)
to extract the area. Afterward, the same image segmentation routine was applied to extract the images of individual buds for further DNN classification.

| Image processing and data analysis
The routines for CVS image processing include a sequence of functionalities including image pre-processing, segmentation, contour extraction, and morphological features calculation in a fully automated manner. The image preprocessing was applied to obtain proper contour image of chrysanthemum teas. A median filter with a template of 3 × 3 pixel was used to de-noise and enhance the image. Afterward, the image was mapped to binarized intensity using the Otsu's method (Otsu, 1979), followed by segmentation to subimages of individual flower buds. The image contour was then computed by the "findContours" function in OpenCV. The morphological features were calculated based on the image contour, and the morphological feature dataset were obtained. Fourteen contour features that describe representative properties of image contour were described in Table 2. Besides the commonly used features such as the perimeter, area, and roundness, three metrics were proposed to measure the irregularity. The irregularity metric was calculated by the variance of contour distances, where the contour distances are defined as the distances between all contour points on the outline and the center of mass. Additionally, the normalized irregularity with a similar calculation procedure to the irregularity feature, except that all the contour distances were divided by the mean contour distance. Similar to roundness, the irregularity metrics also describe the extent to which the object is similar to a circle. When the object is closer to a round shape, the irregularity will be lower, and vice versa. These shape descriptors were compared individually or taken together by chemometrics to study tea quality with specific quantitative models of interest.
The morphological feature dataset was projected to a two-di- The entire data are randomly divided by a 9:1 ratio into a training set (2,112 and 1,423 samples for shape factors and DNN, respectively) and an independent test set (231 and 158 samples for shape factors and DNN, respectively) to establish a classifier for flowering stage and tea type classification.
Three representative machine learning classifiers were evaluated in this study, including the k-nearest neighbor (KNN), multiple linear perceptron (MLP), and support vector machine (SVM).
Because appropriate selection of parameters is the key to achieve optimal performances of the classifier, cross-validation were applied for all classifiers. Specifically, a grid-search algorithm by a 10-fold cross-validation on solely the training set was used to determine the optimal parameters, and the optimal hyperparameters were se-

| Comparison of raw images and shape features
The raw images acquired from the gel imager were first visually evaluated. Figure 3 shows typical raw images of HB and HT subjected to CVS processing. Compared with the open-field or hand-held smartphone without fixed lighting, the gel imager can acquire better-quality images with less interference (Figure 3a and b). However, due to the limitation of hardware setup, all the color and texture information were discarded. The reason is that the light source, the sample, and the camera were aligned vertically to ensure the most effective extraction of morphological features ( Figure 1). On the other hand, further image analyses of shape factors were greatly simplified.
It would be interesting to characterize the morphological fea- For color image subjected to DNN modeling as shown in Figure 3c and d, it is demonstrated that there are visible variations of position, lighting, shadow, and color during image capture due to the random fluctuations caused by the operator and the environment. Such differences brought additional challenge to the correct identification of tea types. Although the CVS is popular for food quality assurance, it is suggested that the CVS hardware, especially lighting may greatly affect the image quality and lead to poor recognition (Jahr, 2006). As DNN is reported to be a robust method, it is interesting to examine the overall performance.

| Principal component analysis of chrysanthemum teas by their shape features
Since the univariate approach was insufficient to determine the type of chrysanthemum teas, a multivariate model may be a suitable alternative. The PCA was performed to provide initial analyses of the multivariate relationships according to the whole contour feature set. Figure  Differentiating seven individual tea types is more challenging since the number of classes is larger than classifying flowering stages, while the number of instances in each class lower. A limited extent of separation between different type of teas can also be observed (Figure 4b). Samples HJ and HG were located on the boundary of data clusters, indicating that they were the most characteristic samples classes for fetal and bloom teas, respectively. However, overlapping occurred in many classes of samples, especially KX and DT spanned also the same region in the scores plot, despite that KX and DT belongs to different flowering stage. The overlapping suggested that classification of individual tea type is more challenging that classification of flowering stage, due that the extent of overlapping between different types of teas. The result indicated PCA is also suitable for discovering the multivariate morphological regularities of teas according to their contour features, while conventional methods including direct comparisons may not achieve. Overall, the PCA did not led a conclusive and clear separation. Therefore, further multivariate modeling techniques were applied to achieve quantitative models.

| Classification by shape features of chrysanthemum teas
Since different tea samples overlapped from the PCA scores plots, further multivariate classification is necessary. Besides, the PCA

| DNN classification of chrysanthemum tea by raw images
A novel approach using the deep neural network with entire colored image approach was also used and compared with the conventional counterparts by shape factors. One of the most attractive features of DNN is that the images can be directly used as modeling inputs; therefore, minimal data-preprocessing is required. The network architecture of the DNN was first optimized. A series of networks with different architectures were trained and compared. All candidate structures and their respective performances classifying tea types were given in  Pfisterer et al., 2018;Xu & Sun, 2018). Therefore, the automated and high-throughput visual authentication and quality assurance of food and may be among one of the potential areas of application of DNN, where higher accuracy may be achieved than the traditional multivariate image classification approaches. Additionally, the recognition was achieved by manually taking images of the samples using a regular mobile smartphone without any external calibration, thus leading to great convenience in large-scale application. Although the computer vision is a valuable technique to provide efficient and objective detection for food manufacturing companies, specialized hardware is required to keep the quality of image consistent and reproducible. This probably is the reason why computer vision is also referred to as machine vision. On the contrary, a gel imager or a smartphone as the general-purpose image acquisition device can overcome the drawback that CVS requires specialized machines built with domain-specific electronic and mechanical parts. Generally, the more variations in these illumination factors, the poorer the classification performance, or vice versa (Jahr, 2006).

| CON CLUS ION
The utility of a hand-held device in this study with ambient lighting inevitably introduced variances in lighting-related factors, such as the intensity, direction, and daylight temperature, as can be observed from the differences of the raw images taken from the smartphone. However, DNN yielded a robust and consistent performance by 96% and 89% classification accuracies when differentiating flowering stages and tea types, respectively. The results were consistent with previous findings that DNN can be a useful alternative modeling method for computer vision applications in foods (Muñoz et al., 2019;Pfisterer et al., 2018;Xu & Sun, 2018). It showed promising results when the lighting, environmental background, and camera setup greatly affected the image quality. Both approaches in this study demonstrated that a common, commercially available gel imager or a smartphone can be used as rapid alternatives to the specialized imaging device for CVS, which brings convenience when building an initial CVS classification model at the trial phase as the prototype building of specific imaging hardware is avoided.
Therefore, the approaches may be suitable in the initial development of a CVS model so that the model of CVS can be built and validated prior to the hardware development.
This study was our preliminary attempt of applying DNN for food quality monitoring; therefore, the sample size was limited. It is expected that more samples could be incrementally included in the model in future applications. The DNN may be suitable for significantly larger amounts of data, which is advantageous in large-scale manufacturing. This study may promote the modern computer vision technique to be applied in processed food products such as other dried botanicals and roasted nuts. It is also foreseeable that deep learning can be useful in other hyperspectral imaging techniques.

CO N FLI C T O F I NTE R E S T
The authors declare no conflict of interest.

E TH I C A L S TATEM ENT
Neither human nor animal testing were used in this study.