Timely access to rheumatology consultation is fundamental to appropriate and effective management of patients with musculoskeletal and autoimmune diseases. Yet, for a variety of reasons, limited and delayed access is commonplace. Moreover, information exchange for referral is often inadequate or poorly communicated. The objective of this work was to improve referral from primary care to rheumatology by formulating and testing a clinically coherent, reliable, and non–diagnosis-dependent Priority Referral Score (PRS).
Using a deliberative process, a clinical panel of 10 primary care providers (PCPs) and rheumatology specialists reviewed clinical case scenarios and engaged in a highly iterative process to develop criteria, definitions, and weights for the PRS, a linear 100-point scale to rate the relative urgency of referral. Following tool formulation, clinicians uninvolved with the process tested the PRS against their clinical judgment.
The PRS comprises 8 criteria, with 2–4 levels for each criterion, and each having a weight generated through conjoint analysis, which forced choices around the comparative urgency of all of the criteria and levels. The PRS showed a strong correlation between clinical rankings of rheumatologists and PCPs in both the deliberative panel, and the physicians subsequently involved in the testing of the PRS.
No standardized priority-setting criteria are available for the full range of primary care referrals to rheumatologists. The PRS had face value with panelists and provided acceptable interrater and intrarater reliability when tested with other rheumatologists and PCPs. Pilot testing with other clinicians and in other settings is justified and prerequisite to use in clinical practice.
Limited and delayed access to care for patients with rheumatic diseases is a multifaceted and global problem. The burden of arthritis is increasing through population aging as well as increasing longevity (1–3). Musculoskeletal disorders rank first in prevalence as the causes of chronic health problems, long-term disabilities, and consultation with health professionals, and rank second for restricted activity days and use of both prescription and nonprescription drugs (4). The World Bank places arthritis among the top 10 causes of disability worldwide.
At the same time, the availability of rheumatologists in Canada (5), the US (6), and Europe (7, 8) is limited both in absolute numbers as well as geographic distribution (9). In Canada, a survey of rheumatology and postgraduate training in academic centers concluded that rheumatology physician resources within Canada were inadequate to meet clinical responsibilities and academic program requirements (5).
The American College of Rheumatology manpower task force report anticipates a growing rheumatology manpower shortage over the next 14 years. It calls for expanded rheumatology training programs and an increase in the number of fellowship positions, expanded use of allied health workers, and practice redesign, such as new scheduling systems and greater agreement with referring physicians as to how problems would be managed across diseases and levels of severity (6).
A rheumatology manpower review in 2005 by the British Society for Rheumatology and the Arthritis Research Campaign showed a slight improvement in the number of consultants in the UK as a whole compared to 1997, but there was inequality in optimal provision within some parts of the UK (7). Less is known about rheumatology manpower issues in other parts of Europe. A 1991 audit revealed that different centers see a different spectrum of rheumatic diseases. Notwithstanding, there have been wide variations noted in rheumatology manpower between countries. Currently in Europe, the European League Against Rheumatism, through the European Union of Medical Specialties section of rheumatology, is undertaking a survey on manpower and patterns of clinical practice (8).
Access to rheumatologists is an important limiting step in the provision of appropriate and timely care for patients with musculoskeletal and autoimmune diseases (10–13). In addition, assuming most patients with rheumatic diseases present to primary care providers (PCPs), the PCP must be able to recognize potential disease and initiate referral. Shortages of PCPs in many countries act as a barrier to accessible and appropriate care (14). Moreover, PCPs may not feel confident about their ability to diagnose arthritis and related problems, which may limit their willingness or enthusiasm to initiate referral to rheumatology (15–17). Compounding access challenges is the inadequacy of referral information. Major shortcomings are recognized in the quality and content of information offered to initiate the referral process from PCPs to rheumatologists (18).
The Western Canada Waiting List Project (WCWL), first formed in 1999, is a consortium of 15 investigators who have researched methods of improving access to scheduled health care services (19–23). In 2008, Alberta Health Services, the province's health care delivery system, collaborated with the WCWL on an initiative to improve access to and flow through medical specialties. Rheumatology was chosen as an exemplar, given the local single-entry model of access through central triage and the existence of clinics focused on early inflammatory arthritis.
Most guidelines by rheumatologists have focused on identifying early inflammatory arthritis or early rheumatoid arthritis. This may be of less relevance to PCPs, who may not be confident in detecting early inflammatory arthritis and may deal with a wide range of other conditions of relevance to them. Accordingly, the primary purpose of this work was to formulate and test a clinically coherent, reliable, and non–diagnosis-dependent Priority Referral Score (PRS) for PCP referral to rheumatologists. The secondary purpose was to standardize the necessary and relevant patient information for referral, offering a reproducible means of evaluating relative urgency for referral. The PRS had to require little time for completion by PCPs, while maintaining clinical coherence for the spectrum of clinical referrals from PCPs to rheumatology. The criteria within the PRS, and their relative weightings, had to combine clinical relevance and urgency for referral from both the perspectives of the consultant rheumatologists as well as the referring PCPs.
MATERIALS AND METHODS
Priority-setting tools are well-recognized waiting time–management strategies (19, 24). Other jurisdictions and previous work by the WCWL have produced priority-setting tools for selected interventions and, in some cases, referrals (25, 26). Following a comparable process for this work, a weighted linear score was formulated that was intended to reflect the relative urgency of the referral. Urgency was not only determined by the severity of the illness, but included the impact of the illness and potential to benefit from treatment.
Initial inputs to the PRS tool formulation included a synthesis of the existing literature and development of clinical case scenarios that were used as clinical exemplars throughout the work. A panel of 5 rheumatologists, 4 family physicians, and 1 nurse practitioner provided the clinical input and inference based on the case scenarios. The panel's deliberative process was a highly iterative, clinically-based dialogue centered on real-life scenarios and involving practitioners in active clinical practice. The end product was a linear scale (range 0–100) that reflected the relative urgency for referral of individual patients to rheumatologists.
The objective of the literature synthesis was to identify any preexisting priority-setting, triage, and referral tools/scales developed to guide referral from PCPs to rheumatologists. The approach used standard methods for systematic reviews (27–29). We did not, however, incorporate an assessment of the quality of the articles reviewed. Given our intent to identify any and all content on tools/scales for determination of relative urgency for referral, we reviewed studies and web sites that appeared relevant, regardless of quality. The literature was compiled and provided to the deliberative panel before their first meeting, acting as an input to thinking during the deliberations.
Unidentifiable case-based scenarios were developed by one rheumatologist from rheumatology referral letters. These scenarios were reviewed and refined for content, relevance, and clarity by one primary care physician and a second rheumatologist prior to presentation to the members of the clinical panel. It was believed that the resultant 32 cases portrayed a realistic spectrum of a clinical case mix that could be referred to a rheumatologist, including a variety of disease severities, management difficulties, and impacts of illness.
A panel of 5 rheumatologists, 4 family physicians, and 1 nurse practitioner actively involved in rheumatology referral and consultation was recruited to participate in the deliberative process to formulate the PRS criteria and weights. These clinicians were selected to reflect the breadth of clinical practice, experience, and practice settings. The rheumatologists came from 5 cities across Canada, with a mixture of academic and community-based practices. Of the family physicians, 1 was rural-based and 3 came from separate cities in 2 western Canadian provinces. The nurse practitioner was a PhD academic and faculty member at the University of Calgary. The facilitator was an internist from New Zealand who had expertise in formulating priority-setting tools.
The clinical panel met for a total of 4 days over a 3-month period. At the start of the deliberative process, each panelist was given the 32 rheumatology case scenarios and asked to rank the cases in order of urgency for referral, using their clinical judgment. Upon completion of this task, variations in the rankings of each case assigned by the panelists and the possible reasons for these variations were presented and explored in an anonymous fashion. With a consensus-building approach, a group ranking for each case arose out of the case deliberations. During the first 12 hours of clinical panel deliberations, prompted by the facilitator to consider any and all possibilities, an extensive list of factors believed to affect urgency of referral was promulgated, debated, rationalized and, with general agreement, distilled into 3 domains. Within these domains, and prompted by the facilitator to converge individual factors into more comprehensive ones, mutually exclusive individual criteria, as well as levels within each criterion, were formulated through repeated reconsideration and dialogue. The extensive list was a varied assortment of symptoms, signs, laboratory tests, imaging results, treatment information, sociobehavioral impact factors, etc. Substantial debate revolved particularly around laboratory tests, some of a disease-specific nature. Given the intention to develop a tool relevant to the full range of rheumatologic practice, the varied list was progressively telescoped into the 8 criteria.
To lend clarity to the PRS criteria, and to reduce user misinterpretation of the criteria, considerable time was spent on the development of clear and unambiguous definitions and wording for each criterion and levels within each criterion. The panel did this with the understanding that the resulting tool and its contents would act as the standardized clinical content for any and all referrals and would be the central component of referral documentation.
Pursuant to this, the panel assigned relative weights (and levels) to each criterion, using conjoint analysis (30, 31) and discrete choice software (1000 Minds) (32). The discrete choice approach provides pairwise comparisons between all of the criteria and levels within the criteria, and forces a choice as to which is more urgent. The series of pairwise choices across the criteria and sublevels of criteria were completed by each panelist. After populating all of the potential pairwise comparisons, the results of the choices and variabilities were discussed among the group, and origins of the variabilities were explored. Through open dialogue and compromise, efforts were made to reach an acceptable resolution of differences, such that all of the panelists could accept the weighting decisions of the majority of the panelists.
Through the deliberative process, consensus-derived criteria, levels, definitions, and weights were achieved over the 4 days. The maximum score of the weighted criteria (and levels within the criteria) was 100 points. The higher the score was, the greater the urgency for referral.
The PRS was subjected to testing and retesting by 14 rheumatologists and 10 family physicians. All of the participants were actively practicing clinicians, none of whom were involved with the PRS formulation. These clinicians individually reviewed 16 case scenarios that were derived and modified from the original 32 case scenarios used to formulate the PRS. These cases had been selected and refined to represent the spectrum of clinical case mix and variations in management difficulties, severity and impact of illness and, hence, PRS scores. Among the 16 scenarios, there were 3 cases that could be considered red flag or emergent conditions, which normally would represent exclusions for use of the PRS. However, the participants were asked to score all 16 cases for testing purposes. The participants ranked the 16 cases in order of urgency using clinical judgment, scored each case using the PRS, and also were asked to identify cases that they considered to be red flags. Moreover, they were asked to comment on the clarity and value of the criteria as the standard referral information that would be available to them.
Six weeks after the initial testing, rheumatologists were retested by presenting them with the same 16 cases but with altered identifiers. They were asked to rank the cases in order of urgency, using their clinical judgment, and subsequently to score the cases using the PRS. Family physicians were retested in a similar fashion 18 weeks after their initial testing. Intraclass correlation coefficients (ICCs) for interrater and intrarater reliability of clinical rankings and the PRS were analyzed for both rheumatologists and family physicians, and these were compared. The analysis combined the PRS scores for the rheumatologists and for the PCPs, providing an overall interrater correlation coefficient to assess the reliability of the PRS. The ICC is the estimated reliability based on a single assessment (i.e., one rater). It ranges from 0 to 1, with a higher ICC indicating better reliability. An acceptable reliability coefficient for scores used to make decisions about individuals is 0.80.
It was apparent during the tool development that, although the majority of clinical scenarios were acceptable for queuing in a waiting list, a limited number of cases required expedited referral. It was thought that these situations should be referred to rheumatologists through direct means, such as telephone contact, and should not be placed on any waiting list. These were designated “red flag” cases and are listed in Table 1.
Patients with clinical presentations that meet red flag criteria must not be scored with the Priority Referral Score. They are advised to contact their rheumatologist directly regarding their concerns and arrangements for more expedited referral.
Emergent clinical concerns
1. Acute monarthritis with specific concern for septic arthritis
2. Giant cell arteritis: significant concern for a diagnosis of giant cell arteritis
3. Acute systemic vasculitis: significant concern for acute development of a systematic vasculitis with major organ involvement, e.g., acute pulmonary, cardiac, renal, gastrointestinal, or neurologic features, cutaneous vasculitis, or digital infarcts
4. Acute connective tissue disease: significant concern for acute development of a connective tissue disease with major organ involvement, e.g., acute pulmonary, cardiac, renal, hematologic, or neurologic features, cutaneous vasculitis, or digital infarcts
5. Significant unexplained constitutional symptoms of either fever >38°C or weight loss >5% in the preceding 6 weeks, where there is a strong suspicion that it may relate to a connective tissue disease
Time-sensitive issues regarding receipt of a previously approved biologic or advanced therapeutic agent(s) for a specific patient
The literature review demonstrated that there is no reliable and valid measurement tool that rates the relative urgency for referral from primary care physicians to rheumatologists (33). Figure 1 shows the wide variability of clinical rankings at the outset of the clinical panel's deliberative process, with an example of patient 16, ranked by panelists in a range from fifth to twenty-first.
Through the deliberative process, as described, criteria were aligned to 3 domains: 1) current state: self-reliance/independence, limit to usual work/role, and pain; 2) threat of progression: currently receiving corticosteroids for the referred condition, complexity of management due to comorbidities, evidence of progressive major organ involvement, and presence of inflammatory markers; and 3) potential for benefit: evidence of active inflammatory arthritis.
Figure 2 shows the 8 criteria, levels, and relative weights. Within each criterion there are 2–4 levels that affect the relative weights assigned (Table 2). The criteria assigned the greatest weights were “evidence of progressive major organ involvement” (20.5 points) and “conditions that threaten patient self-reliance/independence” (20.2 points). Criteria with the least weights assigned included “pain” (6.1 points) and “inflammatory markers” (5.3 points).
Table 2. Definitions of criteria and sublevels within the criteria*
The ability to perform their usual activities of self-care, e.g., eating, bathing, dressing, grooming, toileting, transfers. Activities such as cooking and cleaning may be relevant to self-reliance, depending on the contextual environment, i.e., has other assistance
No significant or meaningful impact
The patient has their normal capability of performing their usual activities of self-care
Significant or meaningful impact but not requiring assistance
Due to the reason for referral, the patient has adjusted activities and is continuing to perform their usual activities of self-care
Requiring increased assistance in the current environment
Due to the reason for referral, the patient has obtained assistance, which allows the patient to continue to perform their usual activities of self-care within their current usual environment
At risk of recently requiring increased level of care in a new environment
Due to the reason for referral, the patient is currently not able to perform their usual activities of self-care, even with assistance in their current environment, and will be required to seek increased level of care in their new environment within the next 30 days
2. Limitation to this patient's usual role/work likely related to reason for referral
The ability of this patient to fulfill their usual roles and function in areas of work/employment, childcare and care giving, homemaking, school, or other social role participation important to this person
No significant or meaningful limitation to their usual role/work
Able to fulfill and participate in these roles(s) normally
Limited or unable to fulfill the patient's usual role/work for >6 months
Due to the reason for referral, the patient has not been able to fulfill and participate in these role(s) for a period of >6 months
Some recent limitation to the patient's usual role/work
Due to the reason for referral, the patient has significant difficulty in fulfilling these roles(s) for a period of ≤6 months
Recently unable to perform the patient's usual role/work
Due to the reason for referral, the patient has been totally unable to fulfill these role(s) for a period of ≤6 months
Directly related to the current medical problem
No significant pain or pain controlled effectively
Pain is not effectively controlled
4. Receiving corticosteroid for rheumatic condition
Currently receiving oral or parenteral prednisone or prednisolone specifically for a diagnosed or suspected rheumatic condition
The patient is not receiving corticosteroid therapy at present for a diagnosed or suspected rheumatic condition
Yes, <25 mg of prednisone (or equivalent) per day
The patient is currently receiving corticosteroid therapy at present for diagnosed or suspected rheumatologic condition at a dose of <25 mg of prednisone (or equivalent) per day
Yes, ≥25 mg of prednisone (or equivalent) per day
The patient is currently receiving corticosteroid therapy at present for a diagnosed or suspected rheumatologic condition at a dose of ≥25 mg of prednisone (or equivalent) per day
5. Complexity of management or risk due to preexisting comorbidities
Patients in whom management is complicated by other significant comorbidities that directly contribute to the complexity of management of the condition for which the patient is being referred, e.g., diabetes mellitus, renal dysfunction, osteoporosis, heart disease, peptic ulcer disease
6. Evidence of progressive major organ involvement probably related to the reason for referral
Laboratory, diagnostic imaging, or clinical evidence of progressive significant secondary major organ involvement (renal, pulmonary, neurologic, vasculitis, hematologic) due to the condition for which the patient is being referred
No evidence of progressive significant secondary major organ involvement
Secondary major organ involvement where there is concern it is unstable or progressive
7. Inflammatory markers: ESR >30 mm/hour (Westergren) OR positive CRP level OR platelet count >400 × 109/liter
Laboratory tests performed to date show evidence of significant inflammation by one or more of the following tests: ESR >30 mm/hour (Westergren), positive CRP level, platelet count >400 × 109/liter
No or not assessed
8. Active inflammatory arthritis features: joint swelling OR morning stiffness of the joints or spine (prolonged for ≥60 minutes) OR MCP/MTP joint involvement
Evidence of active inflammatory arthritis by one or more of the following: a) presence of clinically detectable joint swelling (does not = chronic bony enlargement), b) presence of morning stiffness of a duration of >60 minutes, or c) tenderness on lateral compression across MCP or MTP joints
No evidence of a, b, or c
Yes, without rheumatoid factor
Presence of one or more of the clinical inflammatory features above without a positive rheumatoid factor
Yes, with rheumatoid factor
Presence of one or more of the clinical inflammatory features above with a positive rheumatoid factor
While a substantial number of laboratory tests were considered, only a few were retained as inflammatory markers (criterion 7). Rheumatoid factor was used rather than anti–cyclic citrullinated peptide (anti-CCP) antibodies because of less familiarity with the test by PCPs and, in some cases, restricted availability (criterion 8). Moreover, and arising from repeated debate, the use of several pharmaceutical agents was considered. Unanimous concerns focused on the use of corticosteroids as a critically important factor in determining urgency for referral. While a strong evidence base could not be cited to support a specific dose, which should herald concern, the clinical judgment and experience of the clinicians uniformly supported a threshold dose of 25 mg of prednisone as one of substantial importance. This is supported by the difference in weights (8 points) for patients above and below 25 mg, derived from the discrete choice analysis.
Tool testing results.
With initial testing, the interrater ICCs for the PRS were 0.80 and 0.81 for the rheumatologists and PCPs, respectively (Table 3). For individual raters, correlations between their clinical ranking and the PRS score for the 13 non–red flag cases were in the expected direction and ranged from 0.35 to 0.92 (mean 0.71) for the rheumatologists and from 0.20 to 0.82 (mean 0.58) for the PCPs. With retesting, the average intrarater ICC for the PRS was 0.83 for the rheumatologists and 0.82 for the PCPs. Combining the interrater correlation coefficients for time 1 (24 raters) and for time 2 (22 raters), the results were similar to ICCs for the individual groups, at 0.79 and 0.78, respectively, for time 1 and time 2.
Table 3. Summary of interrater and intrarater reliability for independent rheumatologists and PCPs*
Interrater, time 1 (n = 14)
Interrater, time 2 (n = 14)
Intrarater (test–retest), range
Intrarater (test–retest), average
Interrater, time 1 (n = 10)
Interrater, time 2 (n = 8)
Intrarater (test–retest), range
Intrarater (test–retest), average
Values are the intraclass correlation coefficient (ICC). The ICC model is 2-way random (assumes raters and measurement time are randomly selected) and assesses systematic differences between raters and measurement time. PCP = primary care provider; PRS = Priority Referral Score.
Feedback on wording, definitions, and presentation of the tool was highly favorable and minor suggestions were incorporated. There was a uniform suggestion that the referral form should include a section for “reason for referral” and a section for free-form notes for any additional information deemed useful by the referring PCP.
Of the 3 red flag cases, 1 was correctly identified by 14 of 14 rheumatologists and 10 of 10 PCPs on initial testing. The other 2 red flag cases were identified by 10 and 12 rheumatologists and 7 and 8 PCPs. The PRS scores of the red flag cases were generally in the upper one-third of the PRS scores (Figure 3).
Timely access to medical specialists following referral by PCPs is a challenge for many specialties, particularly those dealing with chronic diseases, such as rheumatology. Access to rheumatologists is a growing issue due to the increasing burden of musculoskeletal conditions, the impact of arthritis, and limited numbers of primary care and specialist providers. While the focus of this work was in Canada, rheumatology access and manpower is an issue in the US, the UK, and Europe. Managing access is made more difficult by the absence of a standardized, priority-setting tool that can rate the relative urgency of individual patients for referral by primary care physicians. To respond to this need and fulfill our primary purpose, a PRS for rheumatology has been formulated that is not diagnosis dependent. Currently, no other priority-setting criteria are available for the full range of rheumatology referrals and the determination of relative urgency for referral.
A secondary purpose was to standardize the information exchange for referral between clinicians in a format acceptable to rheumatologists and meaningful for PCPs. The structure of the 8 criteria that constitute the PRS serves as a template for standardized and relevant patient information that the PCP should gather and provide at the time of referral. This content may be inserted in a standardized referral template, with provision for demographic information and other contents as deemed necessary in local area or international contexts.
The PRS had face validity for the clinical panelists involved in its development. It has provided acceptable interrater and intrarater reliability when tested with other rheumatologists and PCPs uninvolved with the formulation of the tool.
Several limitations of this work are recognized. The work was limited to Canada, using case scenarios drawn from practice and referred to Canadian rheumatologists. Moreover, it has not been evaluated in other countries. This may limit international generalizability. However, the work may have value to others, if not in content, then in approach, with necessary adaptation to suit local context and practice realities.
Arising from the Canadian focus of this work, the PCPs comprising the clinical panel were limited to family physicians (general practitioners). Although not a PCP, we included a nurse practitioner in active clinical practice and directly involved with both referral and consultation of rheumatology patients. It is acknowledged that primary care is organized differently in other countries. For instance, PCPs in the US may include general internists, general pediatricians, and others besides family physicians. Differences in training, referral patterns, and reimbursement, among other things, might well influence the perspectives of differing clinicians as to factors important in urgency for referral. The nature and membership of the clinical panel may have emphasized criteria or weights that would differ from rheumatologists and the wider spectrum of professionals who provide primary care in other regions or countries. Every attempt was made to minimize local-area nuances in practice by selection of rheumatologists and PCPs from several regions from across Canada, representing academic and community-based practices.
During the deliberative process, the views of strongly opinionated specialists or PCPs may have diminished the perspective of other clinical panel members, although the facilitation process attempted to offer all of the participants meaningful opportunities to express their perspectives and discuss their strongly held opinions. Also, it is possible that the case examples may not have reflected the full practice characteristics of most rheumatologists, or those from other geographic areas or countries. Until it is tested over a wide range of cases, it is possible that a particular case mix might be inappropriately disadvantaged by a low PRS. This tool is meant for a full range of referrals and is not disease specific. Other disease-specific tests (e.g., anti-CCP antibodies) or tools (e.g., for early inflammatory arthritis) have a differing purpose and may be used in addition or as an alternate to the PRS. Therefore, further tool testing continues and will be reported. Like any measurement instrument, it is anticipated that in the future new information and practice trends will emerge that may impact criteria or weights. None of these limitations can be dismissed. However, it remains important to compare this standardized method of determining relative urgency for referral to rheumatology across the spectrum of conditions to the other wide-ranging, nonstandardized varieties of referral processes that currently exist. Moreover, diagnosis-specific alternatives are too narrow for use in a full spectrum of rheumatology cases and will fail if the diagnosis is inaccurate.
This tool is now being pilot tested for use in clinical practice, with consideration of ease of use, acceptability, and comparisons to usual means of disposition following referral.
All authors were involved in drafting the article or revising it critically for important intellectual content, and all authors approved the final version to be published. Dr. Noseworthy had full access to all of the data in the study and takes responsibility for the integrity of the data and the accuracy of the data analysis.
Study conception and design. Fitzgerald, De Coster, McMillan, Naden, Armstrong, Barber, Conner-Spady, Noseworthy.
Acquisition of data. Fitzgerald, De Coster, Naden, Armstrong, Barber, Cunning, Hawker, Lacaille, Lane, Mosher, Rankin, Sholter, Noseworthy.
Analysis and interpretation of data. Fitzgerald, De Coster, McMillan, Naden, Armstrong, Barber, Conner-Spady, Mosher, Rankin, Noseworthy.