Measuring process of arthritis care: A proposed set of quality measures for the process of care in juvenile idiopathic arthritis




The ability to assess quality of care is a necessary component of continuous quality improvement. The assessment typically is accomplished by determination of compliance with a defined set of quality measures (QMs). The objective of this effort was to establish a set of QMs for the assessment of the process of care in juvenile idiopathic arthritis (JIA).


A 12-member working group composed of representatives from the American College of Rheumatology, American Academy of Pediatrics, American Board of Pediatrics, and Association of Rheumatology Health Professionals was assembled to guide the project. Delphi questionnaires were sent to 237 health professionals involved in the care of children with JIA. A total of 471 items in 23 domains were identified. The working group met via 4 live e-meetings during which results from the Delphi questionnaires were distilled to a reduced draft set. Each working group member selected a proposed QM to investigate and present evidence from the literature as to its attributes and appropriateness for inclusion into the set. Nominal group technique was used to come to consensus on a proposed set of QMs.


The proposed set contains 12 QMs within 4 health care domains. Each QM consists of a statement of 1) the assessment to be completed, 2) when the first assessment should be completed and a suggested frequency of assessment during followup, 3) recommendations of appropriate tools or methods of assessment, and 4) initial performance goals.


Implementation of the proposed QM set will improve the process of care, facilitate continuous quality improvement, and eventuate in improved health outcomes of children with JIA.


Pediatric rheumatology centers are engaged in local quality improvement efforts in their centers to varying degrees. Systematic efforts have been pursued at some centers for years, but for most the efforts are just beginning. Central to this process is the identification of “quality measures” (QMs; also commonly referred to as “quality indicators”). QMs must be developed in consideration of the evidence that 1) their performance results in improved health outcomes and evidence and 2) they are operationally feasible in a variety of clinic infrastructures. QMs focused on the process of care typically contain a statement of the assessment to be completed, a method for conducting the assessment, the frequency of reassessments during followup, and initial performance goals. Although some individual pediatric rheumatology centers have identified QMs and are using them to perform quality improvement, there is not a uniform QM set relevant to the process of care of children and adolescents with juvenile idiopathic arthritis (JIA) in widespread use. In this project, the focus was on process of care to facilitate development of a standardized method of assessment of quality improvement across centers. Once this method is established, then centers will be able to use these data in the future to develop validated quality improvement measures that focus on patient status and outcomes.

In 2008, the American College of Rheumatology (ACR) provided funding to develop a set of QMs for use by health care professionals involved in the care of patients with JIA. The chief objective of the project described herein was to convene representatives from major North American organizations involved in pediatric rheumatology to work together to develop a standard set of QMs, based on existing evidence and clinical experience focused on the process of care for patients with JIA. The development of this standard set of QMs was done to 1) facilitate the development of a network of pediatric rheumatology centers working in a coordinated fashion to improve the outcome of children with JIA, 2) serve as a basis for quality improvement projects for the ACR and other organizations, and 3) be used by the American Board of Pediatrics (ABP) as part 4 of the requirements for Maintenance of Certification in Pediatric Rheumatology (1).


We used a methodology similar to that used by MacLean et al to develop QMs for rheumatic diseases in adults (2). A QMs working group was assembled that was composed of 12 individuals, each selected by their respective organization from the pediatric rheumatology sections of the ACR and the American Academy of Pediatrics, the Association of Rheumatology Health Professionals, the ABP, the Childhood Arthritis and Rheumatology Research Alliance Consolidated Registry Committee, the Pediatric Rheumatology Improvement Network for Clinical Effectiveness, and 2 facilitators (JS, EHG) experienced in group process and consensus formation techniques. The working group's overall charge was to use consensus techniques to develop a QM set, based on available evidence and clinical expertise, for the process of care for JIA.

In order to obtain broad-based input regarding health care domains and assessments in those domains, the working group sent an online Delphi-type (3) electronic questionnaire (Delphi 1) to various stakeholders. The Delphi packets included an explanation of the rationale for need, the intended purpose of the survey, explanations as to what QMs represent, and their intended use. Recipients were instructed to state what they believed to be the top 5 measures of outcome or care in JIA. A total of 237 individuals received the survey, including 170 pediatric rheumatologists, 8 advanced practice nurses (APNs) involved in JIA care, and 59 patients and parents, representing 27 states and Prince Edward Island, Canada. Results of Delphi 1 were reviewed by the working group, which distilled the assessments offered by respondents, including grouping them into domains, during a series of teleconferences employing nominal group technique (NGT) consensus formation methodology (4). Replies that were redundant or represented the same concept, but stated in a different manner, were combined. Domains and assessments in the domains were then retained or eliminated based largely on feasibility (practicality, ease of use) and face validity (clinical sensibility) (5, 6).

A second questionnaire survey (Delphi 2) containing a reduced set of domains and assessments was developed and sent to the same 237 stakeholders, regardless of whether or not they had replied to Delphi 1. Participants were asked to rank order the reduced number of domains and assessments within a domain in terms of their perceived relative importance in assessing quality of care.

Working from these results, a series of 4 web-based interactive live meetings using iLinc (online at were held among working group members and facilitators during 2009. The purposes of these meetings were to come to consensus on the following questions: 1) should the assessment be retained in the proposed set; 2) do validated methods exist for conducting the assessment and, if so, should one method be recommended over the others; 3) when should the first assessment occur, and how frequently should reassessment be done during followup; and 4) what proportion of patients with JIA in a given clinical practice should meet the QM initially, referred to as the “initial performance goal.”

Prior to convening the series of live meetings, each working group member chose a specific QM to investigate by searching the literature and relying on his or her personal clinical expertise in order to answer the 4 questions shown above. The intent was to familiarize the working group with existing evidence of the utility and clinical relevance of each QM, thereby facilitating consensus formation and synthesis of a proposed set of recommendations. (The specific references used to obtain evidence on which to base recommendations made to the group by individual presenters are shown in Supplementary Appendix A, available in the online version of this article at

The format of the 4 live meetings was as follows. A facilitator introduced the topics for discussion and deliverables (e.g., questions to be addressed) for the call. This was followed by a presentation of the evidence for each QM by the member who had undertaken the review. The presenter stated his or her suggested answers to the questions above, and provided the evidence that formed the basis for their opinions. Following the presentation, NGT was used to come to a consensus about the answers to the questions stated above. In general, meetings lasted approximately 3 hours and 2 to 3 QMs were discussed during each meeting. Facilitators reviewed digital recordings of each meeting, developed a working document containing QM statements, and circulated it to the entire working group for revision. A finalized set of recommendations was then produced.


Response to Delphi 1.

The overall response rate to Delphi 1 was 40% (n = 95): 35% for pediatric rheumatologists, 50% for APNs, and 52% for parents and patients. Among the physician respondents, 85% were currently treating more than 50 patients with JIA per year.

Responses to Delphi 1 resulted in 471 proposed JIA QMs sorted into 23 domains (Table 1). Using conference calls, Delphi, and NGT, the working group reviewed the 471 QMs for feasibility, face validity, and redundancy. Item reduction resulted in 5 domains with a total of 19 quality measures: 1) disease control, 11 QMs; 2) safety monitoring, 4 QMs; 3) education, 2 QMs; 4) access/equity/timeliness, 1 QM; and 5) relationship/patient–family satisfaction with care, 1 QM.

Table 1. Summary of identified domains and assessments within domains resulting from the Delphi 1 survey on measures of process of care in juvenile idiopathic arthritis
 No. of assessments suggested for inclusion into the domain
Disease control69
Medication monitoring52
Physical function32
Patient centeredness13
Psychosocial functioning12
Physical growth8
Quality of life8
Transition to adult care2

These 5 domains and 19 assessments formed the basis for Delphi 2, in which participants were asked to rank order the domains and, for the 3 domains that contained more than one assessment (disease control, safety monitoring, and education), rank order the assessments within the respective domain.

Response to Delphi 2.

The overall response rate was 57% (n = 135): 57% for pediatric rheumatologists, 100% for APNs, and 50% for patients and parents. Of these respondents, 69% had participated in the Delphi 1.

Results from Delphi 2 that showed the highest-ranking assessments in domains with more than one assessment were 1) disease control: (tied) number of joints with active arthritis and physician global assessment of disease activity, 2) safety monitoring: patients undergo eye screening according to published guidelines, and 3) access/equity/timeliness: time to first available appointment with a pediatric rheumatologist.

The working group reviewed the Delphi 2 responses, and used NGT to come to consensus on a working set of QMs. The original 5 domains and 12 assessments within the domains were maintained in the working set (Table 2). This working set formed the basis for the 4 live web-based consensus conferences aimed at reviewing the evidence for maintaining the proposed domains and assessments, as well as tools for assessment, frequency of assessment, and initial and final quality performance goals.

Table 2. Working set of domains and assessments from the Delphi exercises and working group consensus on identifying measures for the process of care in JIA*
  • *

    Some items offered by Delphi participants represent outcomes rather than process of care assessments. Because the focus of this effort was process of care, such replies were not considered further for inclusion. JIA = juvenile idiopathic arthritis; TB = tuberculosis.

Disease control domain
 Pain management
 Achieve complete disease control within a period of x months
 Number of active joints
 Physician's global assessment of disease activity
 Physical functional ability
 Quality of life
Safety monitoring
 All patients undergo eye screening according to published guidelines
 Regular laboratory screening for toxicity to second-line therapeutic agents and biologics
 All patients are screened for TB prior to the start of biologic therapy
 Visual numerical scale to assess parent/patient confidence in ability to manage care of children with JIA
 Time to first-available appointment with pediatric rheumatologist
 Parent/patient satisfaction with care

The results of the 4 live web-based meetings were as follows. During the initial meeting, the working group decided that 5 assumptions were necessary in order to specify to what categories of JIA the QMs would be applicable, and to provide working definitions of the terminology used in QM statements. These assumptions are shown in Table 3. Consensus was reached during the live meetings on 8 of the resulting QM statements. The remaining 4 QMs were recirculated to working group members in a survey format in an attempt to arrive at consensus. When it became apparent that no consensus among the working group members was possible, outside opinion was sought from 5 external board-certified pediatric rheumatologists chosen by the working group, and with extensive experience in caring for children with JIA. The proposed set of QMs resulting from the working group meetings and followup surveys are shown in Table 3. A total of 12 QMs distributed among 4 health care domains are included in the proposed set.

Table 3. Assumptions and summary of the final proposed set of QMs for the process of care in JIA*
  1. * QMs = quality measures; JIA = juvenile idiopathic arthritis; VAS = visual analog scale; AAP = American Academy of Pediatrics; TB = tuberculosis; DMARDs = disease-modifying antirheumatic drugs; CBC = complete blood cell count; CrCl = creatinine clearance.

 QMs were developed under the assumption that they would be applicable to all categories of JIA.
 QMs, their frequency of assessment, and performance goals are minimum recommendations. In the spirit of continuous quality improvement, performance goals should be improved upon continually over time.
 QMs in which a timed frequency of assessment (e.g., every 6 months) is recommended imply that if the patient is seen less often than the recommended assessment schedule, then the assessment will be done at each visit.
 First visit implies first visit to the pediatric rheumatologist after a diagnosis of JIA has been made.
 If a physician practices within an institution or clinic that recommends a frequency and/or method of assessment of QMs that is comparable (and not less stringent), then the local guidelines should be followed in lieu of those below.
Disease control domain
 QM 1: assessment of arthritis-related pain
  Pain should be assessed in all patients at the first visit and at each subsequent visit that occurs at least 7 days apart. (The QM working group recognized 2 types of pain: acute pain intensity at a point in time and average pain over some period. Here, average pain over a period of 7 days is what is to be assessed.)
   Tools: in patients ages ≥7 years, pain should be assessed as an average over the last 7 days. In patients ages <7 years, the parent or guardian should be the proxy reporter of average pain. Any validated, reliable, age-appropriate tool to measure average pain may be used.
   Initial quality performance goal: ≥80% of patients receive an assessment of pain at the first visit and at each subsequent visit if at least 7 days apart.
 QM 2: rheumatologic joint count
  A full joint count of all 75 joints should be done on all patients at the first visit and at 6-month intervals. (Reduced or weighted joint counts may be acceptable in the future after these methods have been more fully validated.)
   Tools: included in the QM.
   Initial quality performance goal: ≥80% of initial visits and 6-month interval visits include a joint count.
 QM 3: physician's global assessment of disease activity (PGA)
  A PGA should be completed on all patients at the initial visit and at each subsequent visit.
   Tools: any validated, reliable method may be used to assess the PGA. An example is the 0–10 VAS, divided into 0.5 increments on a 10-cm scale (9).
   Initial quality performance goal: 80% of initial and subsequent visits include a PGA.
 QM 4: assessment of functional ability
  All patients should receive an assessment of functional ability at the initial visit and at a minimum of 6-month intervals thereafter.
   Tools: any age-appropriate, validated, and reliable tool for the assessment of functional ability may be used. An example is the Childhood Health Assessment Questionnaire (10).
   Initial quality performance goal: ≥70% of initial and 6-month interval visits will include an assessment of functional ability.
 QM 5: assessment of health-related quality of life (HRQOL)
  All patients should receive an assessment of HRQOL at the initial visit and at 6-month intervals.
   Tools: any age-appropriate, validated, and reliable tool that is capable of changing over a 6-month interval may be used to assess HRQOL. Examples include the Pediatric Quality of Life Inventory (11) and the Childhood Health Questionnaire (12).
   Initial quality performance goal: ≥70% of initial and 6-month interval visits will include an assessment of HRQOL.
 QM 6: eye examinations and documentation of compliance to guidelines
  Either the AAP (13) or modified Heiligenhaus (14) guidelines for eye examinations should be followed for patients with any category of JIA. Documentation of compliance to the guidelines should be performed at every visit at least 3 months apart.
   Tools: no single mechanism for obtaining written or verbal documentation of compliance is recommended, but should be one that is convenient for the practice, such as consultation with the ophthalmologist.
   Initial quality performance goal: ≥40% of patients provide documentation of compliance to the guidelines.
Safety monitoring domain
 QM 7: TB screening in JIA patients beginning biologic therapy
  All patients with JIA will undergo TB screening no longer than 3 months prior to the start of any biologic therapy and yearly thereafter as long as the patient remains on biologic therapy.
   Tools: TB screening may be done using the standard TB skin test or, if appropriate, a standard blood test. If, in the clinician's judgment, the patient is immunocompromised and the TB skin test is negative or the patient has received BCG vaccine so that skin test results are in question, a blood assay test is recommended prior to beginning any biologic therapy.
   Initial quality performance goal: 100% of patients receive TB screening prior to start of any biologic therapy and yearly thereafter.
 QM 8: laboratory monitoring
  All JIA patients receiving DMARD or biologic therapy will be monitored for toxicity by clinical laboratory methods. The minimal frequency of laboratory monitoring for DMARDs is every 2–4 weeks for the first 3 months of therapy, every 8–12 weeks after 3–6 months, and every 12 weeks after 6 months of therapy. Guidelines for laboratory monitoring of biologic therapy are evolving and recommendations in this quality measure will be added in the future.
   Tools: the minimal laboratory panel for monitoring laboratory toxicity for DMARDs includes CBC, platelets, and transaminases. CrCl should be assessed in new DMARD starts and at regular intervals in patients receiving cyclosporine or in patients with known renal disease. Pregnancy testing is at the discretion of the managing physician.
   Initial quality performance goal: an initial quality performance goal is 70% of patients receiving DMARDs undergo the recommended laboratory monitoring.
 QM 9: behavioral counseling about toxicity to DMARDS and biologic therapy
  Patients engaging in risk-taking behaviors that could have health consequences related to their medication or disease should receive counseling at each visit. Patients not engaging in risk-taking behaviors that may have health consequences related to their medication or disease should receive counseling at least annually.
   Tools: at the discretion of the health care professional.
   Initial quality performance goal: ≥70% of patients receiving DMARDs or biologics undergo counseling.
Assessment of self-efficacy and patient/parent satisfaction
 QM 10: assessment of self-efficacy (i.e., the belief that one is capable of performing in a certain manner to attain certain goals [15])
  Patients should be assessed for self-efficacy at periodic intervals. The first assessment of self-efficacy should be ≤6 months after the initial visit and every 6 months thereafter.
   Tools: acceptable tools for assessment of self-efficacy include 1) the General Self-Efficacy Scale (16), 2) the Children's Arthritis Self-Efficacy Scale (17), and 3) the Parent's Arthritis Self-Efficacy Scale (18). Other validated tools may be used.
   Initial quality performance goal: ≥50% of patients are assessed for self-efficacy ≤6 months after the initial visit, and every 6 months thereafter, if feasible.
 QM 11: assessment of patients'/parents' satisfaction with care
  Patients and/or parents should be assessed for their level of satisfaction with the quality of care provided to them and/or their child. The first assessment should be done no longer than 1 year after the initial visit, and annually thereafter. (Because the individual physician is often not able to influence the patient's satisfaction with the overall health care delivery system or insurance company, this QM focuses on satisfaction with the health care provided by the physician.)
   Tools: no one specific tool for use in JIA is recommended. Satisfaction may be assessed using any validated tool such as the Consumer Assessment of Healthcare Providers and System questionnaire (19) or other validated and reliable tool.
   Initial quality performance goal: satisfaction assessments are received from ≥30% of patients according to the recommended schedule. (This implies that the physician has made enough effort to receive satisfaction assessments from ≥30% of the overall JIA population in the practice.)
Access to care
 QM 12: time to third next-available visit
  As an initial goal, all patients with signs and/or symptoms of JIA should receive an appointment, based upon the third next-available (20), an average of ≤60 calendar days for a new patient visit from the time of referral. (Timing of followup visits is dependent on numerous factors, and no recommendation is made here.)


Health care providers strive to provide good care to their patients by virtue of their training and professional ethics; however, this does not necessarily translate into quality outcomes for the patients. In 1999, the Institute of Medicine (IOM) reported that an estimated 98,000 patients die from medical errors per year in the US, and mandated reformation in the health care system to reduce the error and improve safety for patients (7). The IOM further recommended that all organizations improve their performance by incorporating care process and outcome measures into their daily work for improvement and accountability. The quality improvement movement grew from this and subsequent efforts of the IOM and is now incorporated into a broad, complex, and incompletely coordinated landscape of agencies, institutions, health care insurers, health care facilities, etc. Furthermore, the importance of developing QMs derives from the contemporary era of mandated quality of care as set forth in virtually all arenas of the medical environment: patient care, education, and accreditation.

The JIA QM working group was organized with the purpose of developing a core set of QMs for JIA that would be adopted broadly. Moreover, it was thought to be critical that these QMs be developed by those most directly involved in the care of children with JIA, i.e., parents, patients, and pediatric rheumatology health care providers. The organizations that the working group members were drawn from represent the key professional, scientific, and accreditation organizations in pediatric rheumatology. The ABP, in accord with the recommendations of the Task Force on Competence of the American Board of Medical Specialties, has mandated that all pediatric rheumatologists seeking ABP recertification demonstrate competency through active participation in quality improvement. Representatives from the ABP were actively involved in the development of these JIA QMs and will incorporate them in Maintenance of Certification training and materials. The JIA QM working group chose to focus on measures of the system or process of care initially to establish regular assessments of the common important measures of care in a standardized fashion. This is a necessary and critical first step in quality improvement efforts. Once centers have adopted these process measures, we will be able to move forward as a group with addressing patient outcomes. This work will be informed and grounded in the results obtained from the current proposed core set of quality improvement measures for the process of care in JIA.

The domains developed in this project will assess dimensions of care that the IOM identified as needing improvement. The IOM-specified dimensions of quality care include safety, efficiency, efficacy, patient centered, equity, and timeliness (8). The domains in this JIA QM set include: 1) disease control, which reflects effectiveness; 2) safety monitoring, which addresses safety; 3) assessment of self-efficacy and patient/parent satisfaction, which are central to patient-centered care; and 4) access, which incorporates timeliness. All of the selected JIA QMs are measurable and were subject to evidence in the literature. Equity in closing racial and ethnic gaps in health status, recognized as a key area needing improvement, is an outcome- focused measure, and therefore not included in the current set of QMs. “Initial performance goals” given in the QM set are based largely on what the QM working group concluded were reasonable and feasible for most practices to achieve. Conversely, no “ideal performance goal” is given because continuous quality improvement implies that ever-higher performance goals are to be set, achieved, and then improved upon further.

The JIA QMs developed in this project hopefully will serve as a format for registry development and initiation of serial data acquisition. This will provide a baseline of data for the development of quality improvement projects, both for the individual practitioner or as a collaborative for a group of centers. Many pediatric subspecialty groups, including neonatology, pulmonology, and critical care medicine, have had strategies to conduct quality improvement through national and regional collaboratives. Benchmarking can be established because collaborative members are transparent in the data and share best practices. The optimal result of the quality improvement work done in either setting, individual or collaborative, will be the improvement of the processes of care that ultimately affects the outcome of the patient's disease status. The view of the working group members is that this improvement will be achieved in a much more efficient and timely manner for JIA if the effort is driven by the use of a standardized set of widely adopted QMs. QMs developed in this project provide parameters but leave latitude for choosing the specific tools or instruments to measure the assessment. The aim is to have relatively uniform data collection for optimizing the outcome of quality improvement projects. Broad-based efforts in pediatric rheumatology are ongoing at this time to develop a common JIA registry for longitudinal data collection in JIA patients in many pediatric rheumatology centers and to develop a sophisticated quality improvement network. The incorporation of these JIA QMs in both of these efforts will enhance the effectiveness of our shared goal of improving the systems of care for children with JIA so as to improve their disease, function, and quality of life outcomes.


All authors were involved in drafting the article or revising it critically for important intellectual content, and all authors approved the final version to be published. Dr. Lovell had full access to all of the data in the study and takes responsibility for the integrity of the data and the accuracy of the data analysis.

Study conception and design. Lovell, Passo, Beukelman, Bowyer, Gottlieb, Henrickson, Ilowite, Kimura, DeWitt, Segerman, Taylor, Vehe, Giannini.

Acquisition of data. Lovell, Passo, Beukelman, Bowyer, Gottlieb, Henrickson, Ilowite, Kimura, DeWitt, Segerman, Stein, Taylor, Vehe, Giannini.

Analysis and interpretation of data. Lovell, Passo, Beukelman, Bowyer, Gottlieb, Henrickson, Ilowite, Kimura, DeWitt, Segerman, Stein, Taylor, Vehe, Giannini.


The authors wish to thank all of the health care professionals, parents, and patients who participated in the Delphi surveys, and the 5 external experts who provided further consensus voting on the final set of QMs: Gloria Higgins, PhD, MD, Suzanne Li, MD, PhD, Paula Morris, MD, Karen Onel, MD, and Sarah Ringold, MD, MS.