Factors Affecting Compliance with Reference Check Requests

Structured reference checks have been demonstrated to be a reliable and valid predictor of job performance. However, the reference check is a unique assessment method for personnel selection in that a third party, the reference provider, is the source of the critical information on the candidate. If a reference provider is unwilling to either partially or fully comply with a reference check request, the usefulness of the reference check is likely to be compromised. This study examined the factors affecting compliance of the potential reference provider with a reference check request. The sample consisted of 905 U.S. adults who were recruited through the Prolific online crowdsourcing platform (). We asked the participants a series of questions related to their actual experiences over the past year in responding to reference check requests. We also included a between&#8208;subjects scenario that examined whether job candidate performance, relationship to job candidate, and method of providing the employment reference would affect hypothetical compliance. The results of the study can be used to more deeply understand the factors that are related to the compliance of the reference provider, and as a result, more fully understand the value of the reference check for selection.

acceptable levels of reliability (internal consistency, inter-rater, and test-retest) and levels of criterion-related validity that rival that of other traditional noncognitive predictors such as personality tests, assessment centers, and biodata (Schmidt & Hunter, 1998). Muchinsky (1979) noted that the reference check is one of the most under-researched techniques used in personnel selection, perhaps due to its (at the time deserved) reputation of low inter-rater reliability and criterion-related validity. Similarly, as noted by Hedricks, Rupayana, Puchalski, and Robie (2018), many practitioners have a negative perception as to the utility of reference checks. 2 One of the most oft-heard complaints is that very little actionable information can be gleaned from reference checks. Reasons for this could include the notion that: (a) job candidates tend to solicit references from individuals they know will only say positive things about them; (b) reference providers are reluctant to say anything negative about the job candidate for fear of adverse social or legal consequences; or (c) job candidates could provide fraudulent references (e.g., a friend or family member) (Hedricks et al., 2018). These concerns, however, are primarily anecdotal in nature. More systematic research should be directed towards understanding what factors impact the free flow of information in the reference-checking process. We believe that examining the factors that impact compliance with a reference check request is a good place to start. This paper is organized in the following manner. First, we describe the practice of reference checking, and discuss the role of compliance by the potential reference provider as related to the success of this practice. Second, research questions and hypotheses related to reference compliance are presented. Third, we discuss the methodology used to address the research questions and hypotheses. Finally, we present and discuss the results of the research study with particular attention to implications for the practice of reference checking as related to job candidate selection.
The reference-checking process is depicted in a figure that can be found in the online supplementary file. As a first step, employers choose whether to ask candidates for the contact information of reference providers (e.g., former or current managers, teachers, or coworkers). If the employer does ask the candidates to provide this information, the candidate can then either provide the requested information or decline. The candidate may also either provide the reference contact information with or without asking the reference for permission. 3 The potential employer may then contact the reference provider to presumably ask work-related questions related to past employment. Many reasons could impact whether a reference provider is contacted by the potential employer, including (but not limited to) how much time is at the potential employer's disposal, the method used to gather the information, the perceived value of contacting a particular reference provider, whether the potential employer remains interested in the candidate, or whether the candidate remains interested in the position with the potential employer. The employer may request to speak to the reference provider over the phone, talk to them in person, or request that they provide information on the candidate via an online survey platform, letter of reference (i.e., recommendation) (LOR), or fax. The reference provider may then supply the requested the information; in some instances, a reference provider may supply only some of the information requested (e.g., provide job title and dates of employment but little else).
The compliance of potential reference providers is fundamental to the success of the reference-checking process, which is to gather and verify work-related behavior to aid in making hiring decisions. There are several points in the process that can affect the level of agreement, or compliance, of the reference provider: when candidates decide to provide a list of references; whom candidates choose to list as references; whether potential employers decide to contact a specific reference; and when reference providers decide how much information (if any) they will provide. In the exploratory, descriptive portion of our study, we propose the following research questions to ascertain the degree and nature of compliance by the potential reference provider: We attempted to answer these research questions by surveying a working population of U.S. adults who are likely to be in a position to be asked to serve as employment references.
We were also interested in answering a second set of complementary research questions using quasi-experimental methods designed to more deeply understand the factors that influence compliance with a request to provide a reference. In a large sample of 150,000 candidates who requested references from 769,822 reference providers, Hedricks and colleagues (2013) found that the compliance rate by those asked to provide a reference was 85.3%.
Given that the percent of reference providers who comply with the reference check request has been found to predict work outcomes (Hedricks et al., 2013), it is important to attempt to understand possible reasons for why the reference provider does not complyreasons that could indeed be related to the candidate's past work performance, their relationship to the candidate at work, or the method by which the reference was to be provided. We examined these three factors in a scenario study using the same participants as used for answering the research questions above; specifically, we examined the potential influences of the: (a) candidate's past work performance (below average, average, above average); (b) work relationship of the reference provider to the candidate (former supervisor vs. former coworker); and (c) the method by which they were asked to provide the employment reference (phone, online, LOR) on a range of dependent variables related to the compliance by the potential reference provider with the reference request.
Although we could find no studies that directly tie prior candidate job performance to the compliance of reference providers, student academic performance has been found to be related to the quality of their recommendations for graduate school (Stake, Walker, & Speno, 1981). Furthermore, prior research has found that, even when controlling for ratings on the survey items on a standardized employment reference check, the percentage of references who comply with the reference request is positively related to performance, and negatively related to involuntary turnover of those subsequently hired (Hedricks et al., 2013). These findings suggest that for lower performing individuals, potential reference providers are simply less likely to respond to the request to provide information. In addition, potential reference providers may not feel comfortable with providing information about the job performance of low performers because of the increased amount of sensitive (i.e., negative) information they might feel compelled to provide. It makes sense then that compliance of the reference provider would be linearly related to a candidate's past work performance. H1: Compliance by the potential reference provider will be highest for above average performing job candidates, followed by average performers, followed by below average performers.
It has been suggested that coworkers are more likely to observe a greater range of ratee (candidate) performance than are supervisors (Murphy & Cleveland, 1995). With this greater opportunity to observe ratee performance, coworkers could witness a wider range of both desirable and undesirable behaviors. Research suggests that potential reference providers are sometimes concerned with claims of defamation as a result of providing negative feedback (e.g., information on undesirable behaviors) even though they are likely covered by qualified privilege (Gatewood, Feild, & Barrick, 2016;Ryan & Laser, 1991).
It is possible then that given coworkers' increased potential to observe undesirable behaviors exhibited by the candidate, that coworkers would be less likely than supervisors to comply with a reference check request.
H2: Coworkers will be less likely to comply with an employment reference check request than managers. 4 Finally, LORs can take several hours to compose and are thus generally considered to be an onerous task (Rosen, 2017). A reference check conducted over the phone or via a standardized web-based form, on the other hand, may take less than 30 min to complete. Further, the confidentiality of a physical LOR is less assured as compared to other methods, and any negative comments about the candidate's behavior, even in an overall positive reference, may be perceived as opening the reference provider to legal repercussions. It is likely then that compliance will be lower for a reference check that is in the form of a LOR compared to the phone-or a web-based survey.
H3: Compliance will be lower for LORs than for phone-or web-based employment reference checks.
We recognize that the three factors we examined (candidate past work performance; relationship to candidate; method of reference check) may interact to affect compliance by the reference provider.
Considering the dearth of existing research on the topic, we preferred to consider evaluation of those interactions as exploratory in nature, and thus do not propose any hypotheses regarding them.

| Participants
We recruited a sample of 1,030 U.S. employed individuals using the Prolific Academic crowd-sourcing platform for a web-based survey (https://prolific.ac). Prolific is similar to the Amazon Mechanical Turk (MTurk) platform, which has seen widespread use in organizational research (Keith, Tay, & Harms, 2017). However, Prolific participants have been found to be more naïve, less dishonest, more diverse, and to provide data of similar quality compared to MTurk participants (Peer, Brandimarte, Sama, & Acquisti, 2017). Also, unlike MTurk, no credible accusations have been made of participants using automated scripts to complete surveys on Prolific (Stokel-Walker, 2018). To ensure that our sample was as representative as possible of the target population (i.e., those individuals most likely to have been recently asked to serve as an employment reference provider), participation was restricted to individuals between the ages of 30 and 66 who indicated that they were employed full-time. Participants were compensated $2.00 for their participation. A total of 125 participants were removed; 123 incorrectly answered the manipulation check questions (see Scenario section below), and two self-reported that their data were not useable.

| Descriptive
After reading a general description of the study and consenting to participate, all participants were presented with our definition of employment reference checking: We are conducting a study on the practice of employment reference checking. An employment reference check is focused on a job candidate's past work performance or reputation at work. As such, an employment reference check is not to be confused with a background check of a person's criminal, educational, or credit records. It is also not to be confused with information that people post about themselves on their social media websites, such as LinkedIn or Facebook. To recap: an employment reference check is focused on a job candidate's past performance or reputation at work.
Participants had to indicate that they had read the above statement and understood the definition of an employment reference check to be able to continue with the remainder of the survey.
In order to gather as much information as possible about factors that influence compliance with a reference request, our study included a series of questions asking participants to describe past experience with a reference request, as well as a scenario in which participants provided their level of compliance toward providing a reference in the future. Participants who indicated that they had been asked to provide a reference within the past 12 months participated in the descriptive portion of the survey which asked about their past experience, whereas participants who had not recently been asked to provide a reference moved immediately onto the final survey question and the scenario portion of the survey. For each of the descriptive survey questions we list in Table 1, participants were asked to refer back only to the most recent time that they had been asked to serve as an employment reference.

1
Within the past 12 months, how many times have you been asked to serve as an employment reference for a former or current boss, co-worker, subordinate, student, or client? 2 Which of the following best describes the job candidate? (applying for a job where I have never worked, applying for a job where I work now, applying for a job where I used to work, other) 3 Was the person already an existing employee where you work, and applying for a different position within your company? (this question is for those who noted in #2 above that the job candidate was applying for a job where I work now) 4 What was the method by which you were to provide the employment reference? (e.g., LOR, phone, web-based survey, face-to-face) 5 What was your work relationship to the job applicant? (e.g., manager/supervisor, coworker, teacher, subordinate, client, vendor) 6 How did you respond to the request to provide an employment reference? (declined; accepted and complied; or accepted but did not comply) 7 What was the primary reason that you provided the employment reference for the applicant? (for those participants who accepted and complied) a ; 8 What type(s) of information were you asked to provide, and what information did you actually provide in the employment reference? (for those participants who accepted and complied) 9 What was the primary reason that you did not provide an employment reference for the applicant? (for those participants who declined or accepted but did not comply)

10
In the future, if you are asked to provide an employment reference, under what conditions might you be most likely to do so?
Note. a Question #7 was open-ended. Participants often included more than one reason (instead of a single primary reason). We decided to code the first reason followed by the second reason recounted by the participant. We made the decision to report reasons only if they were listed by at least 2% of the participants.
TA B L E 1 Questions asked of participants in the descriptive part of the study

| Scenario
Regardless of whether participants had recently been asked to serve as an employment reference, all participants were randomly presented with a hypothetical scenario in which they were being asked to provide a reference for someone with whom they worked in the past. The scenario differed based upon job candidate performance (below average, average, above average); relationship to job candidate (participant acted as supervisor vs. a coworker); and method of providing the employment reference (phone, web-based online survey, LOR) (i.e., a 3 × 2 × 3 between-subjects experimental design). An example scenario in which the participant acted as the candidate's supervisor, the candidate's performance was above average, and the reference check method was an online survey, can be found in the Appendix.
As a manipulation check, participants were then asked three specific questions after being presented with the scenario to ascertain whether they could identify: (a) the candidate's job performance; (b) their relationship to the candidate; and (c) the method by which they were asked to provide the employment reference. Participants who failed the manipulation check questions twice were removed from analyses. 6 After the manipulation check questions, participants were asked to indicate their level of compliance, or agreement, with four statements on a 5-point Likert-type scale from 1 (strongly disagree) to 5 (strongly agree): (1) I would ignore the individual when they attempt to contact me to be an employment reference for them; (2) I would reply to the individual that I would rather not provide an employment reference for them; (3) I would reply to the individual that I will provide an employment reference for them, but then NOT (take the potential employer's phone call/complete the potential future employer's online survey/send a letter of recommendation to the potential future employer); (4) I would reply to the individual that I will provide an employment reference for them, and then (take the potential future employer's phone call/complete the potential future employer's online survey/send a letter of recommendation to the potential future employer).

| Descriptive
The results of the descriptive portion of the study can be found in Table 2. Almost half (45.2%) of the participants reported being asked to serve as an employment reference over the past 12 months, with most (76.8%) being asked only 1 or 2 times. (Question #1).
The majority of participants reported that the most recent candidate for whom they served as an employment reference was applying for a job where the survey participant had never worked (78.0%) (Question #2). Exactly half of those participants who reported that the candidate was applying for a job where they work now were asked by an existing employee who was applying for a different position within the company (Question #3). Over half (55.7%) of the participants reported that the method by which they were asked to complete the employment reference was by phone (Question #4).
Almost half (48.7%) of the participants reported that they were the candidate's manager (Question #5). The overwhelming majority (94.9%) of participants complied with the request by providing the reference when contacted by the candidate's potential employer (Question #6).
The top five types of information that participants were asked, and actually provided about the job candidate, in order of decreas- The primary reason (47.6%) that participants (n = 21, Question #9) did not provide an employment reference was that they were not able to provide a positive employment reference due to issues with the candidate's knowledge or expertise, soft skills, or quality of their work. The next most prevalent reason for not providing an employment reference was that their company forbids the practice of serving as an employment reference (28.6%). Interestingly, no participants reported that they did not provide an employment reference because it had been too long since they had worked with the candidate.

| Scenario
A series of 3 × 2 × 3 fully factorial analyses of variances (ANOVAs) were conducted to determine: differences between candidate job performance ("They were a below average performer," "They were an average performer," "They were an above average performer"); My company policy forbids the practice of serving as an employment reference I did not think I could provide a positive employment reference for the job candidate, as there were some issues with their knowledge or expertise for the job I did not think I could provide a positive employment reference for the job candidate, as there were some issues with their soft skills (e.g., poor attitude, lack of collaboration with others, poor communication, personality, absenteeism, tardiness, etc.) survey participants' relationship to the candidate ("I was their coworker," "I was their supervisor"); and method of providing reference ("Over the phone," "By following a link to a reference checking platform in an email," "By sending a letter of reference") on behavioral intentions toward each of the different responses to requests for an employment reference. We controlled for family-wise error rate by applying a Bonferroni correction to evaluating statistical significance (i.e., 0.05/4 dependent variables ≈ 0.01).
A summary of the ANOVA analyses can be found in Table 3means and standard deviations can be found in Table 4 TA B L E 3 Summary ANOVA results for scenario study participants reported that they would accept and also comply, the findings were reversed from the above such that mean scores were lowest for below average performers, followed by average performers, followed by above average performers. Again, post hoc LSD tests confirmed that these means were all statistically different from one another (p < 0.05).
Relationship to the candidate was nonsignificant across all dependent variables. Thus, Hypothesis 2, which predicted that coworkers would be less likely to comply with an employment reference request than managers, was not supported.
Method by which the reference was to be provided was the other source of statistically significant main effects in this study.
Hypothesis 3, which predicted that compliance would be lower for LORs than for phone-or web-based employment reference checks, was supported for two of the four dependent variables.
For the "decline" dependent variable, participants were more likely to say that they would decline to provide a reference if the method was a LOR compared to an online survey or by phone. Similarly, for the "accept and comply" dependent variable, participants who were asked to provide a LOR were less likely to indicate that they would accept the request and provide a reference as compared to participants who were asked to provide the reference by phone or online survey.
Our exploratory analyses uncovered three two-way interactions that were significant at the 0.05 level but not at the 0.01 level. Given their small effect size and lack of significance when taking into account family-wise error rate, we chose not to attempt to interpret them.

| G ENER AL D ISCUSS I ON
This study was designed to address a series of exploratory research questions and several specific hypotheses about compliance with a request to provide an employment reference. Namely, why an individual may choose to accept or decline the opportunity to provide an employment reference check, and what kinds of information are provided (or not), are fundamental to the practical implementation of employment reference checks as an employee selection method.
This information, however, is exceptionally difficult to gather in situ, given its often highly sensitive nature.
This study is, to our knowledge, the first to examine many of these aspects of employment reference checks in a large sample of employed adults from a diverse set of industries. As a result, our findings represent many aspects of employment reference

| Reference-checking method
Interestingly, just over half of all participants reported that the method by which they were asked to complete the employment reference was by phone, followed by a written LOR, a web-based (online) survey accessed through an email link, face-to-face in person, and finally "other." This raises an interesting point about the potential tradeoffs that hiring managers and potential reference providers may face in deciding how to conduct an employment ref- erence check, and how to respond to such a request, respectively.
In particular, a reference check provided over the phone requires the employer to schedule a time that works for both themselves and the potential reference provider, as well as a time commitment to take part in the phone call and record pertinent information or playing "phone tag." Reference checks conducted over the phone may also be difficult to aggregate across multiple candidates for the same position, further increasing the difficulty in making any meaningful comparisons between candidates. Finally, if phone reference checks are not structured and/or not properly documented, this could pose problems if an employer is challenged on a hiring decision.

| Reference provider relationship
The vast majority of participants who reported receiving a request for an employment reference acted as either the candidate's manager (48.7%) or coworker (39.1%), suggesting that employment references are often provided by individuals who would have had an opportunity to observe a candidate's behavior in a work context, which is good news for prospective employers who hope to gather information about exactly that. On the other hand, the plurality of reference providers acted as the candidate's manager. As noted above, managers may generally have less opportunity to observe a wide range of work behaviors compared to coworkers (although we could find no empirical evidence to support this conjecture).
Moreover, opportunity to observe has been linked to more reliable performance ratings (Rothstein, 1990) and more valid assessment center ratings (Moser, Schuler, & Funke, 1999). It stands to reason that if managers in general have less opportunity to observe performance than coworkers, then the information supplied by managers could be less valuable than what could be provided by coworkers.
On the other hand, in our experience, those involved in the hiring process often place more weight on manager feedback from reference checks. More research is certainly necessary to determine the truth behind these suppositions.

| Reference provider compliance
The overwhelming majority of participants (94.9%) reported that they accepted the request and provided the reference when contacted by the candidate's potential employer. This is slightly higher than previous research would tend to indicate. As noted previously, Hedricks and colleagues (2013) found that, in a sample of 769,822 reference requests made via an online survey platform, 85% of the individuals complied with the request for a reference.
One of the most practically compelling findings in our study was the fact that the most common reason for providing an employment reference was that the candidate was a good or competent worker. If reference checks are meant to be predictive of candidate job performance, it is imperative that responses to reference check requests are motivated by in-depth knowledge of past performance, which we have found to be largely the case, as opposed to a quid pro quo or laissez-faire attitude to the reference-checking process.
Similarly, assisting/helping was often given as a reason for compliance with the reference request; this reason occupied the second and third ranking for why a reference was provided and possibly indicates that the candidate was associated with positive LMX/TMX (i.e., good interpersonal abilities) (Banks et al., 2014). On the other hand, these findings also point to the possible selection bias that occurs in the compliance process-potential reference providers seem to be more motivated to provide references for good performers, but in this sample, no potential reference provider noted that they were motivated to provide a reference to ensure that an organization did not hire a poor performer. 9 The possible negative effects of this selection bias are tempered by the finding that if one can capture the percentage of reference noncompliance at the candidate level, it actually can be used as a predictor of subsequent involuntary termination of those hired (see Hedricks et al., 2013).  & Laser, 1991), it behooves the employer to inquire about a range of information, including information that may lead to uncovering less desirable workplace behaviors that a candidate has displayed in the past.
Of the relatively few participants (n = 21) who had been asked to provide an employment reference but indicated that they did not comply with the reference request, the primary reasoning was that they were not able to provide a positive employment reference due to issues with the candidate's knowledge or expertise, soft skills, or quality of their work. Practitioners should be especially concerned about this finding because, as noted previously from research in the applied setting, Hedricks et al. (2013) found that candidates with a high level of reference provider noncompliance were significantly more likely to be involuntary terminated after hire as compared to candidates with a low level of reference provider noncompliance. On the other hand, several other, relatively innocuous reasons for noncompliance were also reported.
Finally, and consistent with the findings we present above about reasons for compliance, opportunity to observe work behavior, candidate work competence, and recent experience working with the candidate (and thus, an accurate recollection of their behaviors), were three of the four primary reasons given for why participants might serve as an employment reference. Interestingly, being contacted ahead of time was also a strong motivator for acting as an employment reference. This may also be consistent with the positive LMX/TMX and good interpersonal skills we interpret from responses to previous questions: conscientious candidates with the forethought and interpersonal skills to contact their potential references ahead of time are likely to be viewed more positively and thus increase compliance to their reference requests.

| Scenario
The findings from the scenario portion of the study are particularly interesting in light of the responses from the descriptive questions about past experience. Consistent with the findings we present from the descriptive questions, candidate performance was the most consistent driver of the effects we found in the scenario portion of the study. For each of the responses for which the participants indicated their compliance intentions, compliance was most strongly associated with candidate performance, independent of their relationship to the candidate and the method of providing the reference, fully supporting Hypothesis 1. These findings further support our descriptive survey findings and suggest that a candidate's previous work performance is the strongest motivator toward an individual complying with a request to serve as a reference provider.
Participants who reviewed a scenario containing a candidate who was a below-average performer were significantly less likely to comply (either by ignoring the request, declining the request, or accepting the request but ultimately not providing the information), than participants who reviewed a scenario in which the candidate was an average performer. Compliance was thus most likely for those who reviewed a scenario in which the candidate was an above-average performer. This finding may appear troubling in that it reinforces practitioners' beliefs that reference checks provide only positive feedback on candidates. In contrast, a study by Hedricks et al. (2018) of the verbatim comments on confidential online surveys supplied by 20,000 reference providers on 5,000 job candidates found that 83% of the reference providers listed at least one work-related area for improvement and 89% listed at least one work-related strength.
Hypothesis 2, which suggested that coworkers, as compared to managers, would be less likely to comply with an employment reference check, was not supported. Future research should examine the possible mechanism more directly that managers comply more often than coworkers because they see providing references to current and former subordinates as part of their formal job responsibilities.
The method by which the reference was provided also had a significant impact on compliance with a reference request. Participants indicated that requests for LORs were more likely to be declined and suggested that they would be less likely to fully comply and provide a LOR if requested, partially supporting Hypothesis 3. These findings contrast with the prevalence of LORs reported by participants who had been asked to serve as a reference provider as reported in the descriptive survey results: despite being significantly less popular among reference providers, results from the descriptive part of the study revealed that LORs remain one of the most common methods by which employers request reference information. This suggests that practitioners who are concerned with compliance by the reference provider might want to consider alternatives to a LOR (Madera, Hebl, Dial, Martin, & Valian, in press;Nicklin & Roch, 2009).

| Practical implications
Although we have alluded to some practical implications in our discussion of the results thus far, it is worth directly restating them from the perspective of employers who are using or are considering the use of reference checks in their selection process. First, our study revealed that employers are not necessarily consistent when asking questions of reference providers that will help them make reasoned hiring decisions on candidates. For example, the majority of employers (73.7%) did not ask about a candidate's work-related areas for improvement (or weaknesses). Employers who ask such questions may thus benefit from competitive advantage in the war for talent. Second, a minority of participants reported being contacted through an email link to an online survey for a reference check. Online methods of reference checking have proven to be faster and more efficient than phone or written LORs (Hedricks et al., 2013). Using slower and less efficient methods of reference checking will necessarily prolong the selection process and likely lead to a loss of top prospects. Third, our study and previous data show that the rates of reference provider compliance are likely to range from 85% to 100% and if one captures a measure of this compliance, it can be used as a variable in and of itself in addition to the ref-

| Conclusion
To our knowledge, this is the first study that has examined the issue of compliance with a request to provide an employment reference. Our study has identified several issues for practitioners to consider closely, as well as several avenues for future academic research into reference provider compliance. We hope that the results of our study can be used to improve the reference-checking process and ultimately to help organizations make fair and well-informed decisions on job candidates in an efficient, scalable manner.

E N D N OTE S
1 Please note that the criterion-related validity of reference checks from Schmidt and Hunter (1998) is not updated from Hunter and Hunter (1984), only based on 10 samples that predate 1984 and not corrected for range restriction-unlike most of the other operational validities in the aforementioned meta-analyses in which range restriction corrections were applied. Hedricks et al. (2013), on a structured reference check, found internal consistency estimates ranging between 0.96 and 0.98 (N = 8,000); 2-week test-retest reliabilities averaging 0.74 (N = 1,298); and an uncorrected correlation of r = 0.35 (p < 0.001) between the overall score reference check and supervisory ratings (N = 223). Taylor et al. (2004) in a predictive, criterion-related validity study of 223 applicants found the overall score on a structured reference check to be related to supervisory ratings at a moderate level (r = 0.25, p < 0.01) but higher (r = 0.36) when corrected for range restriction and criterion unreliability.