Validity evaluation of the genetics and genomics in nursing practice survey

Abstract Aim To psychometrically test the Genetics and Genomics Nursing Practice Survey (GGNPS) for evidence of content, face and construct validity. Design This study was a secondary data analysis. Method Data collected from the Method for Introducing a New Competency into Nursing Practice (MINC) study were used to evaluate the GGNPS for evidence of construct validity via structural equation modelling and confirmatory factor analysis. Face validity was evaluated via feedback from practicing RNs without specific experience with or knowledge of genetics/genomics. Content validity was evaluated via content expert feedback and assessment of a content validity index. Results The thresholds for evidence of content and face validity were met. However, we found less evidence for construct validity of the GGNPS; an exploratory factor analysis, conducted to gain additional insight into the theorized latent constructs, determined that the variables were more interrelated that previously predicted.


| INTRODUC TI ON
Genetics and genomics play key roles in nursing practice. These roles include providing patient and family education, administering prescribed treatments, advocating disease prevention/health promotion, collecting and interpreting family history information, and collaborating with the healthcare team to facilitate genetic referrals if warranted. These roles and responsibilities demonstrate that Registered Nurses (RNs) in all specialties, clinical roles and practice settings must adopt genetics/genomics into their practice (Jenkins & Calzone, 2007;Starkweather et al., 2018).
Due to the importance of genetics/genomics in clinical practice, the Genetics and Genomics Nursing Practice Survey (GGNPS) was created to evaluate RNs' competency/knowledge, attitudes/receptivity, confidence, and decision/adoption of genetics and genomics into nursing practice; in addition, the GGNPS evaluates the effect of social systems in the healthcare setting on all of these domains.
The instrument was developed from a well-validated instrument assessing the adoption of genetics/genomics by family physicians (FPs) (Jenkins, Woolford, Stevens, Kahn, & McBride, 2010). Revising the original FP instrument to reflect nursing practice created the GGNPS. While the FP instrument was evaluated using content expert feedback and structural equation modelling, the GGNPS has not had same validity evaluation.

| BACKG ROU N D
Genetics and genomics have a significant impact on the leading causes of mortality and morbidity and must be a component of evidence-based, competent nursing practice aiming to improve patient outcomes (National Academies of Sciences, Engineering, and Medicine, 2016). Genetics/genomics affect aspects of care in all clinical roles and nursing specialties, making it a required competency of nursing practice (Calzone, Jenkins, Culp, Caskey, & Badzek, 2014; Consensus Panel on Genetic/Genomic Nursing Competencies, 2009). The GGNPS can play an important role as researchers assess nurses' knowledge and use of genetics/genomics in clinical practice.
A number of instruments relating to genetics/genomics are available, but focus predominantly on education, perceived (as opposed to objective) knowledge or do not focus specifically on nursing practice. For example, the Genetics and Genomics in Nursing State of the Science Conference Survey focuses on nurse educators and faculty. This instrument contains questions about genetics/genomics courses taught in nursing curricula, whether participants' certification examinations contained questions related to genetics/genomics, continuing education and current research (Thompson & Brooks, 2011).
The Genetics Needs Assessment Survey (Maradiegue, Edwards, Seibert, Macri, & Sitzer, 2005) is a 70-item instrument evaluating participants' perceived knowledge of and comfort with genetics as well as desired educational activities to increase knowledge (e.g. lectures, role play, problem sets). It does not assess objective genetics knowledge via questions that can be marked as correct/incorrect; it also does not assess associated concepts such as attitudes towards genetics/genomics, or use in clinical practice.
Additional instruments, such as the Genomic Nursing Concept Inventory (GNCI), measure understanding of genetic/genomic topics (Ward, 2011). The GNCI was developed by surveying baccalaureate nursing students. It focuses on evaluation of knowledge and has been predominantly used with students and faculty. Bottorff et al. (2005) developed an instrument to evaluate knowledge, confidence and professional roles in providing genetic services for adult-onset hereditary diseases. The instrument was created and tested on nurses and physicians in Canada. Lastly, researchers in the United Kingdom created an instrument evaluating nurses' confidence and knowledge about collecting family history information, as well as providing referrals (Bankhead et al., 2001). While a variety of genetics/genomics-based instruments are available, the GGNPS focuses on nursing practice. It also evaluates how factors such as confidence, social systems and attitudes interact with knowledge to ultimately lead to adoption or rejection of genetics/genomics in clinical practice.
The GGNPS is internationally relevant because the adoption benchmark is family history (FH) collection. Collecting FH information is not dependent on access to technology or funding. This benchmark was specifically selected by the developers of the GGNPS to allow it to be used in a variety of countries and areas-especially regions with variable resources (Calzone et al., 2012). Additionally, the Global Genomics Nursing Alliance (G2NA) conducted an international survey, including representatives from 18 countries, and it included FH taking as a key component of nursing roles in the delivery of genomic services (Calzone, Jenkins, Jenkins, Culp, & Badzek, 2018;Calzone, Kirk, et al., 2018). The aim of this study was to evaluate the GGNPS for evidence of content, face and construct validity.

| DE S I G N
This is a secondary analysis of baseline data previously collected during the MINC study.

| Survey instrument
The GGNPS was initially pilot tested for usability with a convenience sample (5 nurses), then tested again with a larger sample (Calzone et al., 2012). A second pilot test was conducted with 239 RNs employed by the National Institutes of Health (NIH). Following pilot testing, the GGNPS was revised based on participant feedback and a review by content experts; these revisions included the removal of questions that expert reviewers considered unclear. It also included the addition of two items about knowledge of genetics of common diseases (added with permission) from the Genetic Variation Knowledge Assessment Index (Bonham, Sellers, & Woolford, 2014).
In addition, the GGNPS underwent reliability testing using data from two clinical institutions (Calzone et al., 2016). The reliability testing study included baseline and completion assessments from a 1-year educational intervention. Participants completed the GGNPS twice, at the beginning and at the end of the intervention. The study found that Likert-scale items with five or more options had poor test-retest reliability; scales of those items were shortened to increase reliability. Calzone et al. (2016) reported that the mean agreement across all items in the GGNPS was "moderately" strong (mean Cohen's κ ≈ .41), as defined by Landis and Koch (1977, p. 165). Of the testretest κs, 39% (n = 36/95) were between .41 and .60. However, some of the items had substantial variability in reliability, ranging from .150 to 1. Calzone et al. (2016) also found that "Select-allthat-apply" items related to self-assessment of one's ability to discuss genetics of common diseases with clients performed the poorest. Items pertaining to attitude/receptivity also performed nearly as poorly. In addition, they found that Likert-scaled items with fewer response options, 3-4 options, performed better than Likert-scaled items with 7 response options, regardless of item content.
The version of the GGNPS used for this study did not include revisions made following reliability testing because reliability testing was completed after baseline data were collected in the Method for Introducing a New Competency into Nursing Practice (MINC) study (Calzone et al., 2016). Following an educational intervention, reliability testing was conducted to obtain a sample in which participants would answer questions based on understanding of genetics/genomics.
If the baseline data were used for reliability testing, reliability could also be affected by participants' lack of knowledge about genetics/ genomics. In simplest terms, reliability may be affected if participants were guessing while completing the GGNPS. The current study complements Calzone and colleagues' investigations into the reliability of the instrument with research into several aspects of its validity.

| Sample/Participants
The sample included 7,798 RNs from 23 Magnet ® hospitals in 17 states from all regions of the United States. Magnet ® hospitals were selected because they are believed to provide higher quality of nursing care; Kalisch and Xie (2014) reported increased patient teaching, additional staffing resources and improved communication in Magnet ® hospitals compared to comparable non-Magnet ® hospitals.
Due to these characteristics, Magnet ® hospitals have an increased capacity to support both innovation and pilot programs for new initiatives (Calzone et al., 2014); therefore Magnet ® hospitals were selected as part of purposive sampling.
This sample is representative of the target population because the GGNPS is intended to evaluate the use of genetics/genomics by nurses providing direct patient care (Tables 1 and 2).

| Conceptual framework
The conceptual framework was based on Rogers' (2003)   The study variables were derived from Rogers' model and work by Calzone et al., 2014Calzone et al., , 2012. The "competency/knowledge" variable addressed gaining an understanding of genetics/genomics and being able to use it in the scope of RNs' professional standards and responsibilities. The "attitude/receptivity" variable evaluated nurses' beliefs and perceptions about genetics/genomics. "Social systems" was the variable that explored the setting and environment where this innovation was being used. Lastly, "decision/adoption" addressed the decision to adopt or reject genetics/genomics and how this innovation was used in nursing practice. Each variable was operationally and conceptually defined by mapping the item to the associated variable for statistical modelling (Table 3).
Confidence was included as a variable based Calzone et al., 2014, 2012, Calzone, Jenkins, Culp, Bonham, and Badzek (2013) (Benner, Hughes, & Sutphen, 2008). This parallels Rogers's (2003) theory individuals who are more capable of coping with uncertainty and risk are more likely to adopt innovations. As a result, this study evaluated the direct effect of competency/knowledge on confidence, the direct effect of confidence on attitudes/ receptivity, as well as the indirect effect of confidence on decision/ adoption.
Decision/Adoption* Decision: activities that lead to a choice of either adopting or rejecting the innovation. Adoption: use of an innovation (Rogers, 2003). In the present study, decision/adoption was operationally defined as self-reported collection and assessment of a family history, as well as self-reported facilitation of referrals to genetic services.
Confidence "Level of certainty that knowledge about the innovation is accurate" (Calzone et al., 2012, p. 12).

Social system
Social system: the setting or environment where the innovation was introduced such as the clinical site where nurses are employed (Calzone et al., 2012).
*Decision was the portion of the DOI model where individuals engaged in activities that affected the choice of adopting or rejecting an innovation. However, the decision stage is difficult to observe and measure because decisions are often internal thoughts (Rogers, 2003). For this reason, Calzone and colleagues combined decision and adoption into one domain, facilitating empirical measurement. In this study, decision was considered part of decision/adoption and is not measured separately.
variables were used to create a proposed model, based on both Rogers' DOI model and on research related to adoption of genetics/ genomics to nursing practice.

| Hypothesized model
The

| Ethical considerations
Institutional review board (IRB) approval for this study was obtained from the primary investigator's institution, protocol number 2016-0067. The IRB allowed an exemption from full review because data were not collected with identifying information, ensuring confidentiality.

| Data analysis
Data were analysed using Mplus version 7.3 (Muthén & Muthén, 2012) and R version 3.1.2 (R Core Team, 2013). R packages used included nortest (Gross & Ligges, 2015), psych (Revelle, 2018), MVN (Korkmaz, Goksuluk, & Zararsiz, 2014)  Multiple imputation, the most statistically robust method (Carpenter, Kenward, & Vansteelandt, 2006), was used to account for missing items because not all data were MAR. Imputations were conducted using the mice (van Buuren & Groothuis-Oudshoorn, 2011) package in R, because the GGNPS contains a variety of items including dichotomous 'yes/no' items, Likert-scaled items and an interval-level knowledge score. By beginning every imputation with the variable that has the least amount of missing data, the mice package utilizes the most complete variables to fill in data on less complete variables. Lastly, the mice package also uses random draws of data to mimic natural variability in the data.
Four imputations, with 50 iterations for each imputation, were conducted. The multiple iterations allow the program to try multiple F I G U R E 1 Hypothesized model variations until a "best-fit" scenario is selected. ANOVAs compared means of the four imputations. Results demonstrated that there were no statistically significant differences between the imputations, demonstrating increased likelihood that the data were imputed based on the parameters of the existing data.

| Content validity
Evidence of content validity was investigated using a content validity ratio (CVR to "not relevant") (Lawshe, 1975). The CVR was calculated using Lawshe's formula: where N e is the number of panellists who classified the item as "very relevant" or "relevant" and N is the total number of content  7) is achieved if only one content expert rates an item as "somewhat relevant" or "not relevant". Therefore, both calculating the CVR using normal approximation to the binomial and consideration that all, except one, content experts consider the item "very relevant" or "relevant" are the basis for establishing support for content validity.

| Face validity
Seven reviewers, all nurses with a variety of backgrounds, provided feedback for face validity evaluation (Table 4). The reviewers were selected to provide the perspective of the majority of nurses in the workforce; therefore, none had specific education or expertise in genetics/genomics. Because the GGNPS was created for use in clinical nursing practice, reviewers whose primary role was patient care were selected. Face validity evaluated the ease of understanding and applying the instrument to nursing practice (Table 4). The threshold for acceptable support for face-related validity for the GGNPS was met.

| Construct validity
Approximately 12% of the data in the original data set were missing; although this amount of missingness is common, we used multiple imputation to ensure the most accurate and robust results possible. Additionally, relationships between items appear more complicated than the factor model could explain.

| Structural equation model
The CFAs, therefore, could neither adequately support nor refute the theoretical model. To further investigate construct-related validity of the GGNPS, we conducted a series of structural equation models (SEMs) evaluating the relation between factors and the associated evidence for construct-related validity.
Using the model, χ 2 from the SEM analyses as an initial fit test (Table 5) found that the model did not fit the data well (χ 2 = 451.979, p < .001). However, χ 2 statistics are sensitive to sample size (Hooper, Coughlan, & Mullen, 2008) and provide one, limited perspective on the SEMs. The root mean square error of approximation (RMSEA) was used to evaluate model fit, while also considering sample size.
The RMSEAs for the SEMs used the original data set, and the four variations of imputed data suggest the models still did not provide what Browne and Cudeck (1993) would consider a good fit (Table 5).
The comparative fit index (CFI) was used to gain further insight into the SEMs by comparing how well the model structure proposed in the SEM compares against a null model. An incremental fit index, a CFI ≥0.9 indicates a good model fit (Davis & Murrell, 1993). The CFI was not >0.9 in the original data or the four variations of imputed data (Table 5).
Lastly, we computed Bayesian information criteria (BICs) for each SEM. BIC corrects for both the sample size and number of parameters in a model and (Raftery, 1993). BIC is a comparative fit measure between models, with a lower BICs indicating better model fit.
Among the original data set and the four variations of imputed data, the BICs ranged from 79,507 to 139,625 (Table 5).
Although reported, the relationships among the variables in SEM must be interpreted cautiously because numerous fit analyses all indicated that the hypothesized model did not fit the data well (see Figure 2).   (Wolf, Harrington, Clark, & Miller, 2013).

| Ancillary analysis
Thresholds for evidence of content and face validity were met; however, we did not find strong evidence for construct validity.
Since we did not find sufficient evidence that the items on the GGPNS produce a theoretically sound model, we conducted an exploratory factor analysis (EFA) to investigate the factor structure. We used five factors to reflect the number used in the SEM; we also used oblique factor rotation to allow factors to be related to each other (Osborne, 2015). Criteria used for evaluation were factor loadings ≥0.30 (Costello & Osborne, 2005) and presence of Thurstone's (1947) simple structure. The EFA results demonstrate that the factors were more interrelated than anticipated; these results can be used to re-evaluate which items are related to respective DOI factors and used to inform future construct validity analyses.

| D ISCUSS I ON
Content and face validity for the GGNPS were well supported.
However, the threshold for construct validity was not met. The inability of the CFA to produce factor loading is related to the varied structure and relationships between the items. The response structure of the GGNPS, specifically the variety of questions (e.g. categorical and dichotomous items), created fluctuating amounts of variance among the items that affected the ability to evaluate the variance of the overall model. For example, items with more answer choices contained higher variance than dichotomous items.
Additionally, the relationships between the variables may be more complex than predicted in the hypothesized model. This complexity appears to prevent us from establishing construct validity: the GGNPS is a complex instrument attempting to measure multiple domains in different ways. The complexity was a barrier in attempting to establish construct validity. Additionally, because the EFA showed that many items designated in the GGNPS as one variable actually loaded on a different variable (Table 6), this can create a measurement error that would be compounded because the SEM evaluated direct and indirect relations amongst five variables (Table 7). Even if a small number of items in each variable actually measured a different variable, this would create non-negligible measurement error.
The previously conducted evaluation of reliability also provides insight into the weak relations in the SEM. Many attitude/receptivity items, especially "Select-all-that-apply" items or those that included large Likert scales, were the most poorly performing items during reliability testing (Calzone et al., 2016 and social systems and attitude receptivity (0.150). Relations that included attitudes/receptivity as either the dependent or the independent variable were all weaker than other relations in the hypothesized model.
Evaluation of construct validity also demonstrated key parallels between the GGNPS and the FP instrument. In the SEM analysis of the original instrument, used by FPs, Jenkins et al. (2010) also reported that adoption of genetics/genomics did not match the path in the DOI model. Researchers reported two paths leading to adoption of genomic-related innovations: one path based on "comfort" with using genetics/genomics in clinical practice and a second path based on "relevance" to clinical practice. The researchers hypothesized that the multiple paths may be related to the complexity of clinical practice and the complexity of the innovation, genetics/genomics.
This complexity was a contrast to the linear nature of Rogers' DOI model. These varied paths to adoption of genetics/genomics can provide insight into further evaluations of construct validity in the GGNPS. Together with the EFA results, this will be used to redefine the GGNPS variables and evaluate paths to adoption of genetics/ genomics in nursing practice.

| Limitations
Although a common limitation among research based in healthcare settings, survey burden and time constraints, resulting in data missingness, were notable limitations in this study. RNs may initially express interest in participating, but realize they do not have adequate time to complete the GGNPS or consider all of the items carefully. In fact, 679/7,798 participants (8.71%) opened the instrument to complete it, but left all items blank. Furthermore, all of the institutions used for data collection had conducted institution-wide nursing data collection in the previous 6 months (Calzone et al., 2014). Survey burden may affect not only the accuracy of responses, but may also contribute to measurement error.
Multiple imputation, a statistically rigorous method to address missing data, was used to decrease the limitations caused by partially completed surveys.

| CON CLUS ION
Analyses support GGNPS face and content validity. However, construct validity remains to be supported. The CFA and SEM models proposed in this study did not fit the data well. EFA results in the ancillary analysis provide both insight into the likely sources of measurement error and recommendations for revision and retesting of the GGNPS.

ACK N OWLED G EM ENTS
A big thank you to Dr. Eileen Gigliotti for her support during my doctoral dissertation.

CO N FLI C T O F I NTE R E S T
There are no conflicts of interest to disclose.