The construction of the evaluation system of nurses' post‐training and the application of the system in 25 grade‐A general hospitals in China

Abstract Aims To make the evaluation more scientific, structured and systematic, this study aims to develop an evaluation index system for nurses training and to explore clinical effect of system. Design The evaluation index system of nurses’ post‐training was constructed using the Delphi method. Methods Introducing the system, we used the pre‐work training of new nurses as an example for discussing the specific implementation scheme of the system. Twenty‐five tertiary and first‐class general hospitals in 14 provinces and municipalities were evaluated on the spot, and the application effect of the system was evaluated comprehensively. Results The index system consisted of three first‐grade indexes, seven second‐grade indexes and 23 third‐grade indexes. There were three levels in teaching and training ability, and the distance had statistical significance.


| INTRODUC TI ON
Nurses account for approximately 0.005% of the total population in most countries of the world, while they count for only 0.001% in China. The shortage of nurses has become a common problem faced by hospitals all over China, especially large third-class general hospitals. For this reason, hospitals, clinics, community service stations and other major hospitals have vigorously introduced nursing talents to make up for the shortage of nurses. According to the National Nursing Development Plan of 2016, the number of Registered Nurses in China will reach 4.45 million by 2020 and the comprehensive quality and professional and technical ability of nurses will directly affect the quality of clinical nursing and patient safety. With the transformation of the medical model and the continuous advancement of the medical and health system reforms, the people pose higher requirements for the quality and ability of nurses. Nurse post-training is an important way to maintain and expand nurses' abilities and to improve the professional quality of nurses. It provides nursing personnel a guarantee for maintaining and improving people's health (Lewinson et al., 2015;Okoli et al., 2015;Stirling, 2015). The question of how to train nurses effectively has become an urgent one among nursing managers in hospitals. Although nurses' on-the-job training is widely carried out in China -often with a great deal of money and energy invested -there is no evidence to support whether this training can bring enough benefits to patients, nurses and hospitals.
Training evaluation is of great significance for ensuring training effectiveness, measuring training performance and achieving training objectives (Janssens & Van Amerlsvoort, 2008;Mehralizadeh et al., 2007). Nemec (2018) pointed out that most health institutions do not evaluate training at all or that they limit their evaluation to simple satisfaction indicators in the United States. South Africa and other countries use only a self-report feedback scale and 360 degree to measure the training evaluation work (Mash et al., 2018). Scholars have pointed out that understanding the cost and benefit of training is very important to ensure the monetary value of hospital capital input; the economic evaluation of training is very limited and it is necessary to expand and standardize the process and to select an appropriate economic evaluation model to capture the cost and benefit of training to achieve comparability of training effectiveness in different environments (Banke-Thomas et al., 2017). However, the training evaluation of nurses in China also faces complex challenges.
Without a thorough understanding of the essence of the training system cycle, the whole process of training and the results of training at all levels have not been fully excavated in time and space. The nursing training system is inefficient. There is a lack of evaluation of training, a lagging behind of evaluation results, there are obstacles encountered in training migration, a lack of strategic orientation in evaluation activities, a single evaluation method and a lack of longterm follow-up evaluation mechanism, etc. Therefore, it is particularly urgent that we do a good job in evaluating the effect of nurses' on-the-job training. The purpose of this study is to construct a scientific and reasonable evaluation index system of nurse post-training and to explore the clinical effect of the system, so as to provide a theoretical basis for enriching the system of training and improving the evaluation of hospital nurse post-training.

| The establishment of a research group
A research group was established in November 2015. The research group consisted of eight members, including two nursing education management experts, four clinical managers and two nursing master's degree students. Among them, the nursing education management experts were responsible for supervising and inspecting the research process and providing manpower and material support; the nursing postgraduates were responsible for the whole research design, data processing, statistical analysis and carrying out the clinical practice of the index system; the clinical managers were responsible for forming semi-structured interview outlines, formulating interviewees, forming expert inquiry papers, selecting experts and completing three rounds of inquiry workers.

| The preliminary selection of evaluation indicators
According to the requirements of the State Council of the People's

| The establishment of the evaluation model
Some scholars have exploratorily introduced evaluation models to guide the evaluation (Moore et al., 2009;Smidt et al., 2009 (Kaufman & Kelley, 1994;Kirkpatrick, 1996Kirkpatrick, , 2009Phillips, 1995). CIPP and the Kirkpatrick model were the most widely used training theoretical models. CIPP includes four parts: context, input, process and product. Not only does it analyse the necessity and feasibility of training, but it also pays attention to the monitoring of the whole training process. The Kirkpatrick model includes four progressive levels: the reflection level, the learning level, the behaviour level and the result level. as "training," "index system" and "evaluation" to collect all the evaluation indexes of nurse training.
2. Expert interviews: In April 2016, 19 nurses, training managers and nursing educators were selected as interviewees to partake in semi-structured interviews. Before the interview, team members were trained in a unified face-to-face manner, clarifying the purpose of the interview, the topic of the interview, the inclusion criteria of the interviewees, the time of the interview, the matters needing attention in the process of the interview implementation, etc. These took the form of one-to-one, face-to-face interviews, each lasting approximately 30 min. The main themes were "Do you think it is necessary to evaluate the effect of training after it has taken place?" and "What suggestions do you have for setting up the evaluation index of nurses' on-the-job training?" Experts revised and perfected the index categories, names, definitions, connotations and calculation methods one by one and initially formulated the framework of evaluation index system for nurses' job training.
3. Pre-Survey: Based on semi-structured interviews and a literature review, a small-scale pre-survey was carried out according to the framework of the evaluation index system for nurse post-training.
Six experts of nursing management, nursing education, medical education and clinical nursing were invited to discuss and revise the structure level, classification and expression of indicators in the framework.
4. According to the preliminary investigation, an expert inquiry questionnaire was finally formed, which consisted of three parts: 1) completing the form; 2) including index name, index connotation, calculation formula, Likert grade 5 score and revision opinion column; and 3) Basic situation of experts: general information such as age, length of service, professional title, educational background and expert's familiarity with research issues and judgement basis. if they had a bachelor's degree or above, held a post of associate senior or associate professor or above, were responsible for nursing post-training management or nursing teaching for more than 15 years, were highly motivated and were willing to cooperate in the completion of multiple rounds of expert consultation. The first two rounds of questionnaires were distributed to 32 consultants, while the third round consisted of 31 consultants, as one withdrew due to illness. There were nine senior and 23 vice senior titles, 11 master's degrees and 21 undergraduate students, aged 38-57 (46.17 SD 2.12) with working years from 15-37 (19.28 SD 3.23).

| Constructing the index system
The establishment of the evaluation index using the Delphi method From July-October 2016, the inquiry volume was sent to experts by mail and the inquiry explanation and confirmation were provided via telephone with experts. In the first two rounds of expert consultation, 32 questionnaires were sent out and 32 questionnaires were recovered. The initial response rate of expert questionnaires was 100% and that of the third round was 96.88%. According to the results of three rounds of expert inquiries, the second and third evaluation indicators were determined.

Determining the weight of indicators using the analytic hierarchy process
In the study, the evaluation index system was composed of the first-, second-and third-level indicators, of which there were seven secondary indicators and there were several three-level indicators under each secondary indicator. First, the seven secondary indicators were compared and a grade was assessed according to their importance, using the pair comparison method and a 1-9 comparative scale to create a comparison matrix; the same method was then used to construct a comparison matrix for the three-level indicator under each secondary indicator.
For the second-and third-level indicators, the maximum eigenvalues and corresponding eigenvectors were calculated, respectively. At the same time, a consistency test was carried out. When the random consistency ratio is less than 0.1, it can be concluded that the weight obtained by the judgement matrix meets the consistency requirement (Ishizaka & Labib, 2004;Satty, 1980Satty, , 2006Taleai & Manaourian, 2008).

| Implementing the process of evaluation index system
To further validate the feasibility of the evaluation index system, the research group conducted in-depth analysis and discussion of the previous research results and formulated the evaluation criteria and rules of the system combined with the established evaluation index system of nurses' post-training. A total of 169 new nurses from a tertiary-A general hospital in August 2017 were selected as participants for a six-month pre-job training programme. The time of evaluation can be divided into three stages: pre-training evaluation, training process evaluation and post-training evaluation; this was to explore the specific implementation plan of the system. To enhance the standardization and effectiveness of the training and to avoid the influence of subjective factors, the whole evaluation was carried out by the members of this subject group. According to the evaluation criteria and methods of the system, the corresponding checklist or questionnaire or examination paper were designed and embedded in the Nursing Information System platform or 512 examination education network. The whole evaluation process was carried out in the information system, including data entry, extraction, statistical analysis and other functions.

| Pre-training assessment
Before the training of new nurses, first, we held a symposium for nursing managers to clarify whether each department had formulated training programmes and plans for new nurses, whether the title, educational background, teaching age and professional level of the teaching staff were reasonable, the division of training for the instructors and the training methods used (combination of scenario simulation and classroom teaching, PBL, group demonstration, discussion, network learning), etc. Second, the theoretical knowledge and relevant operational skills of new nurses were investigated using self-designed test papers and the training goal was established by investigating the training needs of new nurses with the questionnaire of pre-post training needs.

| The training process assessment
New nurses were trained for 3 months. The three-level index C10-C13 was used as the evaluation index of training process. The evaluation tools were as follows: (1) a self-designed pre-job training feedback questionnaire for new nurses was designed to understand the subjective feelings and evaluation of the whole training project of new nurses; and (2) a self-designed Clinical Nursing Teaching Training Plan Implementation Evaluation Table to extract the implementation rate, ensuring a timely understanding of the progress of training, so as to amend and improve the follow-up training projects.

| Post-training assessment
Post-training assessment is divided into immediate evaluation, shortterm evaluation and long-term evaluation: 1. Immediate evaluation: At the end of pre-job training for new nurses, it can be used to judge the degree of the realization of the training objectives, the satisfaction of the nurses and the improvement of nurses' knowledge and skills. The three-level indicator C14-C17 is the immediate evaluation index, and the evaluation tools were as follows: the feedback questionnaire of pre-job training for new nurses (self-designed); the theory and operation examination; and the comprehensive ability assessment form at bedside (self-designed).
2. Short-term effect assessment was conducted 3-6 months after the completion of pre-job training for new nurses. It was used to assess

| The application effect of the evaluation index system
To verify the application effect of the evaluation index system, from November 2016-September 2018, our team selected 25 general hospitals in 14 provinces and municipalities in China for evaluation, which were represented by the letters A~Y ( Figure 2). We strictly abided by the procedures and conditions required by our methodologies and controlled the admission criteria of hospitals as follows: all hospitals were grade three and grade-A general hospitals; we obtained the approval of the hospital ethics committee; we excluded hospitals that refused to participate in research; we selected hospitals where experts participated in three rounds of expert inquiries; and we distributed our study as far as possible across hospitals of all provinces and cities throughout the country.
Before conducting an on-site evaluation of 25 hospitals, hospital experts who were familiar with nurses' job training were selected. The members of the task group contacted them by telephone to explain the purpose, content and method of the investigation, to obtain their consent and to collect the basic information of investigators (name, phone number, gender, title, etc.). The basic information of the hospital is shown in Table

| Data collection and statistical analysis
SPSS19.0 was used to calculate experts' enthusiasm, authority and coordination coefficients, to test the reliability of experts and AHP9.0 statistical software was used to calculate the weight of indicators. Metrological data were expressed by mean and standard deviation, and counting data were expressed in examples and percentages. SPSS19.0 was used for cluster analysis and variance analysis of the comprehensive score of hospital training level; p < .05 was considered statistically significant.

| The enthusiasm of experts
The enthusiasm of experts was expressed by the response rate of the questionnaire. In the first round of expert inquiry, 32 questionnaires were sent out, 32 valid questionnaires were recovered, and the recovery rate was 100%. In the second round, 32 valid questionnaires were sent out and the recovery rate was 100%. In the third round, 32 valid questionnaires were sent out and 31 valid questionnaires were recovered, with an effective recovery rate of 96.88%.

| The authority of experts
The degree of authority of experts is the arithmetic average of the sum of the judgement basis and familiarity of the indicators in the expert self-evaluation table, which was expressed by Cr. In the study, the Cr of consultation experts was 0.82.

| Degree of coordination of expert opinions
The coordination degree of expert opinions was generally expressed by a coordination coefficient. The coordination degree of the first, second and third round of consultation expert opinions was 0.889, 0.847 and 0.869, respectively. Chi-square test showed that there was a statistical significance (p < .01).

| Revision, weight and consistency test of evaluation indicators at all levels
After expert inquiry and statistical analysis of the data, the experts agreed on the second-level indicators without any modification; in the third-level indicators, 3 items were merged, 7 items were modified, and 13 items were deleted. Taking the secondary index as an example, the judgement matrix was constructed and the weights of each index were normalized ( Table 2). The consistency test of the judgement matrix is carried out (Table 3) and Cr < 0.1. It could be concluded that the weights obtained by the judgement matrix meet the consistency requirements.

| Determining the evaluation index system
The evaluation index system was finally established, which consisted of three first-level indicators, seven second-level indicators and 23 thirdlevel indicators. At the same time, according to the experts' assignment of the importance of indicators at each level, the weight and coefficient of variation of indicators at each level were determined (Table 4).

F I G U R E 2
The Distribution of the 25 tertiary and first-class general hospitals across 14 provinces and municipalities

| Implementation results
According to different evaluation methods, each three-level index was scored one by one. The score of each index was 100 points. The weighted cumulative comprehensive scoring method was used to multiply the average score of each index by the weight of the index to get the weighted score of the index (Table 5). All weighted scores were accumulated to get the total score of the index. The comprehensive score calculated in the study was 90.28.

| Application results
Based on the results of on-site evaluation, the overall situation of teaching and training in 25 hospitals is shown in Table 6. A cluster analysis of 25 hospitals showed that there were four levels of differences in teaching and training ability: the best, the better, the general and the worse. Hospitals A, B, C, P, Q, R, S and T were in one category, and the teaching and training abilities of the eight hospitals were the best; hospitals J, K, L, M, N, O, V and X were in one category; hospitals D, E, F, G, H, I, U and W were in one category; the teaching and training abilities of the eight hospitals were general, while hospital Y was in one category and the teaching and training abilities were relatively poor. The results of variance analysis showed that the distance between the four groups had significant statistical significance (p < .05). It showed that the evaluation index system of nursing teaching training based on the cluster analysis had a strong distinction and could evaluate the teaching training situation of the 25 hospitals better. The cluster analysis is shown in Figure 3.

| Reconstructing the nurse training evaluation model
Choosing an appropriate evaluation model as the theoretical framework can effectively guarantee the comprehensiveness and pertinence of the index setting. Suhayda et al. (2006) point out that the evaluation model is accountability-oriented, emphasizing active evaluation and retrospective evaluation to guide the optimal design of training projects and ensure the quality of training evaluation.

| Scientific-ness and credibility of the evaluation index system
The effect of nurses' job training needs a scientific evaluation system to evaluate it comprehensively. This study combines the character-  given by experts to the study. The results showed that the expert authority coefficient was 0.82 and the higher the coefficient, the more authoritative and representative the expert inquiry results. The coordination coefficients were 0.889, 0.847 and 0.869, respectively, which showed that experts agree with each other, the trend was stable and the index weight structure had high reliability.

| A weight analysis of the evaluation index
The results of the study showed that the weights of post-training

TA B L E 5
The Comprehensive score of the evaluation index system of pre-job training effect for new nurses (score)

Tri-grade index Evaluation standard Evaluation method scores
Weight scores

A1-B1-C1 Policy and Planning
The hospital has a series of supporting policies, such as promotion, employment, tuition subsidies, implementation of the training plan for nurses and related documents for nurses' training.

| The practicability of the evaluation index system
The pre-training needs analysis of nurses is the first step in the whole process of training activities. It is the basis for formulating training plans, designing training programmes, implementing training activities and evaluating training effects. It is very important for nursing education and training. The training needs of ZK hospital were mainly based on the subjective judgement of the hospital management, focusing on the hospital needs and work needs, seldom considering the personal needs of employees, depriving employees of their autonomy in choosing training, thus reducing the enthusiasm of employees to participate in training. In the study, a pre-job training needs questionnaire for new nurses was set up to determine the training needs of new nurses and establish training objectives, so as to promote long-term planning and development of nurses, which is conducive to the overall human resources reserve and promotion of hospitals.
The index system constructed in the study regards pre-training evaluation as the primary index and highlights its importance. It will guide medical institutions to attach importance to pre-training evaluation, control the behaviour of evaluation objects and guide them to approach training objectives. During the training process, The results of Table 4 showed that the three-level indicators "training teachers," "training content" and "plan implementation" score are lower when the system is used to evaluate the training process. The reason is that most of the nurses training teachers were backbone nurses in hospitals and few of them employ teachers from outside schools or from outside hospitals, which inevitably affects the teaching attitudes, teaching methods and skills, teachers' pre-class preparation and the implementation of training plans of the training teachers. Therefore, the problem of nursing training teachers is a key issue related to the effectiveness of clinical nurses training and it is also an urgent problem to be solved at present. Zhou et al. (2016) point out that before each class, teachers must determine the teaching objectives, adjust the curriculum system timely and improve the implementation methods and evaluation strategies of nursing teaching. Hospitals should focus on "setting up platforms, gathering talents, stressing the introduction and training, creating mechanisms, increasing vitality and being the leader" and make every effort to promote the construction of a nursing talent team, so as to build a compound nursing team with both virtue and talent and excellent specialty. After transforming the index system into a questionnaire or a checklist, the study quantifies each three-level index, together with a rigorous evaluation method, which can objectively and accurately reflect the problems in the training and provides a quantitative standard for the evaluation of the training quality of nurses.
For the management department to use the evaluation system regularly, it can obtain a large amount of information about the training situation from multiple perspectives, grasp the situation dynamically and in a timely manner, clarify or adjust the training focus and input and provide a scientific basis for decision-making.
In applying the index system, we can select different medical institutions or clinical departments and carry out assessment, award evaluation and merit evaluation based on the evaluation results, which was conducive to improving the effectiveness of hospital training resources allocation.

| Information-based nursing training evaluation and the monitoring system
With the rapid development of global information technology, the development and application of hospital information systems have become one of the key aspects of current hospital development.
Based on the practice of advanced hospitals at home and abroad and supported by hospital information systems, the author's hos-

Understanding the lagging effect of the training effect in an all-round way
Not all training projects can achieve immediate results, which can only be reflected after a certain amount of practice, which requires training to adhere to the principle of long-term sustainability.
Research such as that conducted by Eid and Quinn (2017) reveals three factors influencing the transfer of training: trainee characteristics, training courses and working environment. Relevant trainees' characteristics include attitudes to change, motivation, psychological processing ability, interpersonal skills and personality characteristics: curiosity, humility, seriousness, flexibility, intelligence and enthusiasm. Training projects, team learning and lectures were identified as relevant aspects of training courses. Work culture, work relationships and resources were all sub-themes of the work environment category. In a systematic review, Beidas and Kendall (2010) suggested that the effectiveness of physician training was influ-

| CON CLUS ION
The evaluation index system of nurses' post-training constructed in the study is scientific, practical and operable. It is of great significance for nursing managers to understand the effects of training and to improve training quality. It also provides a reference for training supervision of relevant health administrative departments.
The index system is still in the initial stage of construction. Although its universality has been studied in some third-class and first-class general hospitals and has achieved remarkable results, it still needs to further expand the empirical research area.

ACK N OWLED G EM ENTS
Thank you to the 32 experts who participated in the study. Thank you for participating in the study of 25 tertiary and first-class general hospitals in 14 provinces and municipalities. Thank you, Professor Ma Jun, School of Public Health, Tianjin Medical University, for your statistical analysis.

CO N FLI C T O F I NTE R E S T
The authors declared no potential conflict of interest with respect to the research, authorship and/or publication of this article.

AUTH O R S' CO NTR I B UTI O N S
The first author and the corresponding author made substantive intellectual contributions to the article. Linlin JIAO and Yuanda SUI conceptualized and designed the research and writing the paper.
Guihua YANG collected and interpreted the data. Pulin WANG and Qiaofeng LI analysed the data. Junhong CHEN, Lili LIU and Chunling YANG interpreted the data and discussed core ideas. All authors had full access to the data and take responsibility for the integrity of the data and the accuracy of the data analyses.

DATA AVA I L A B I L I T Y S TAT E M E N T
Data are available on reasonable request from the authors. Full data set available from the corresponding author at suiyuanda@126.com.