Clinical performance among recent graduates in nine low‐ and middle‐income countries

Summary Objectives Recent studies have identified large and systematic deficits in clinical care in low‐income countries that are likely to limit health gains. This has focused attention on effectiveness of pre‐service education. One approach to assessing this is observation of clinical performance among recent graduates providing care. However, no studies have assessed performance in a standard manner across countries. We analysed clinical performance among recently graduated providers in nine low‐ or middle‐income countries. Methods Service Provision Assessments from Haiti, Kenya, Malawi, Namibia, Nepal, Rwanda, Senegal, Tanzania, and Uganda were used. We constructed a Good Medical Practice Index that assesses completion of essential clinical actions using direct observations of care (range 0–1), calculated index scores by country and clinical cadre, and assessed the role of facility and clinical characteristics using regression analysis. Results Our sample consisted of 2223 clinicians with at least one observation of care. The Good Medical Practice score for the sample was 0.50 (SD = 0.20). Nurses and midwives had the highest score at 0.57 (SD = 0.20), followed by associate clinicians at 0.43 (SD = 0.18), and physicians at 0.42 (SD = 0.16). The average national performance varied from 0.63 (SD = 0.18) in Uganda to 0.39 (SD = 0.17) in Nepal, persisting after adjustment for facility and clinician characteristics. Conclusions These results show substantial gaps in clinical performance among recently graduated clinicians, raising concerns about models of clinical education. Competency‐based education should be considered to improve quality of care in LMICs. Observations of care offer important insight into the quality of clinical education.


Introduction
In recent years, governments and global health actors have committed to achieving universal health coverage (UHC), aiming to improve health outcomes and increase financial risk protection for all people. While an essential step forward, the UHC movement has placed less emphasis on quality of care. However, poor quality may limit the beneficial impacts of UHC, particularly in lowand middle-income countries (LMICs), where baseline quality standards are often not met. Evidence indicates that overall quality of care is low and varies both across and within countries, even for basic maternal and child health services [1][2][3]. To reap the benefits of increased coverage, commensurate improvements in quality are required.
One important element of a high-quality health system is a competent health workforce, and many governments have dedicated resources to strengthening human resources for health. However, these efforts have largely focused on expanding health worker numbers and improving workforce distribution [4,5]. National health workforce efforts rarely emphasise quality of care [6,7], and little is known about the quality of services new clinicians in LMICs are providing. When governments do seek to improve health worker performance, efforts focus on in-service training for clinicians already in practice. Studies have found that such training has only a modest impact on quality and cannot by itself close the large quality deficits observed [8,9].
One contributing factor to poor observed quality of care may be weak health professions education, the 620 course of studies that prepares a health care worker for entry into practice [10]. Past studies have noted that while professional demands on health care workers continue to increase, educational systems are failing to keep pace, resulting in 'the mismatch of professional competencies to patient and population priorities . . . producing ill-equipped graduates from under financed institutions [11,12].' Health professions education is hampered by outdated curricula and pedagogy, poor adaption to local contexts, insufficient systems for ensuring educational quality, and a dearth of qualified tutors and clinical teachers [11,[13][14][15]. Schools note a lack of qualified students for training, congestion at clinical placement sites, limited mentorship, inadequate equipment and technology, and difficulties with faculty recruitment and retention. Students face issues of inadequate housing, transportation, and classroom space [15,16]. Furthermore, existing programmes often do not emphasise the importance of quality of care in their curricula [17,18].
This study assesses the performance of recent clinical graduates in completing fundamental clinical skills in practice in nine low-and middle-income countries. To this end, we constructed a Good Medical Practice Index, a set of essential clinical items required to make a correct diagnosis and provide appropriate treatment in three primary care service areas: (i) antenatal care, (ii) family planning, and (iii) care of sick children and compare performance across provider types and countries. Results can be used to inform efforts to improve clinical education and other strategies to improve quality of care.

Study sample
Data for each country were obtained from Service Provision Assessments (SPA), surveys of health facilities conducted by the Demographic and Health Surveys Program. age of the data, existence of a more recent survey from the same country, or data inaccessibility. The surveys use nationally representative samples, and censuses or near censuses (in Malawi, Namibia, and Rwanda) of a nation's health facilities. Within surveyed health facilities, up to five patients per provider per clinical area were selected for observation using systematic random sampling. Trained observers assessed first visits or follow-up visits in their entirety for antenatal care, family planning consultations, and sick child care consultations for children aged five years or younger. To assess the quality of pre-service education, we analysed the performance of providers in the first three years of practice post graduation. Providers were grouped into four categories: physicians, associate clinicians (e.g. clinical officers), nurses and midwives (e.g. registered nurses, nurse midwives), and other providers (e.g. counsellors, social workers) (Appendix Table A1). Analyses did not include those in the 'other providers' category, as the education these providers receive varies greatly both between and within countries.

Outcome definition and assessment
We developed the Good Medical Practice Index (GMPI) to assess the minimum clinical performance in assessing the patient that is essential for making a diagnosis and proposing correct management and that is expected of all clinicians providing clinical care across visit types (Figure 1 and Appendix Table A2). The GMPI was developed using previous quality indices and service-specific clinical guidelines [1,2]. It includes 28 basic clinical activities across antenatal care (ten items), family planning care (eight items) and sick child care (ten items) based on items asked in all Service Provision Assessment surveys matched with existing clinical guidelines. Six activities are repeated in two or more domains, resulting in 22 discrete clinical activities. Similar to an objective structured clinical examination (OSCE), the resulting index includes essential activities all providers should perform in every clinical visit across countries (and are therefore unweighted), and can serve as a flexible tool to objectively evaluate clinical competency among providers in low-and middle-income settings [19]. The index includes items covering history-taking, physical examination, and counselling actions that should be conducted for all patients regardless of the reason for presentation or the local epidemiology. As history-taking items included in the index may not apply to antenatal care follow-up visits, these items were excluded from GMPI calculations for relevant observations; all other index items apply to both first and follow-up visits across the three service areas. In instances where certain services were provided by a clinician other than the primary provider separate from the actual consultation (e.g. a nurse taking blood pressure measurements prior to the full examination), these activities were recorded as having been performed during the visit. The primary outcome is a Good Medical Practice Index score calculated as an average of the proportion of index items a clinician completed across patient encounters in antenatal care, family planning, or sick child care. The resulting score ranges from 0 to 1 with a higher score corresponding to greater performance of essential clinical actions.

Covariates
Several factors other than education may influence clinical performance among recent graduates. Using Rowe's framework for explaining health-worker practices, we identified facility and provider characteristics that corresponded to covariates in the data, and analysed the effects of these on the GMPI [20]. Facility characteristics were defined as infrastructure and environmental factors that might be associated with care quality, such as facility management (public or private) and structural quality. Structural quality was measured with an index of service readiness defined by the World Health Organization: general service readiness (50 indicators across five domains: basic amenities, basic equipment, infection prevention measures, diagnostic capacity, and essential medications).
Provider characteristics included provider sex, years of pre-service education, in-service training, and supportive supervision. In-service training was defined as receipt of any general in-service training or in-service training specific to one of the three service areas within the past six months. Supportive supervision was defined as a health care worker reporting supervision in the last six months that included discussion of problems encountered and receipt of supervisor feedback. We included a covariate for provider type in the model that included all clinical cadres. Finally, we used an indicator variable for each of the nine included countries as a proxy for national factors, including quality of a country's health professions education that may influence quality among providers in the first three years of practice.

Statistical analysis
To assess quality across items in the Good Medical Practice Index, we calculated the proportion of GMPI clinical items each clinician completed across his or her patient encounters in each service area. We estimated the mean and standard deviation GMPI score for each country and clinician type as an average of mean clinician scores across patient encounters. Clinicians were excluded in countries with fewer than ten providers per cadre. We also calculated 95% confidence intervals for the mean Good Medical Practice Index score of each provider type  in each country. The outcome was rescaled to have a mean of zero and a standard deviation of one. Multivariable models were constructed to test the association of each country with GMPI, controlling for facility and provider characteristics that may influence provider performance and confound the relationship of interest. The association was tested using ordinary least squares linear regression for all clinicians, and then separately among physicians, nurses and midwives, and associate clinicians; models were clustered by facility. Countries were excluded if they had fewer than ten providers in a given clinician cadre. Tanzania, which had a Good Medical Practice score near the median among each clinician type, was used as a reference group for all models. All statistical analyses were carried out using Stata version 14.2 (Stata-Corp, College Station, TX, USA).

Ethical approval
The original survey implementers obtained ethical approvals for data collection; the Harvard University Human Research Protection Program deemed this analysis exempt from human subjects review.

Results
The SPA assessed 6572 of 6755 health facilities across the nine countries of interest; the remainder were closed, empty, inaccessible, or refused assessment. Among *Supportive supervision is defined as supervision that included feedback and discussion of problems encountered in the past 6 months. †Service readiness index is a score from 0 to 1 assessing facility preparedness to deliver healthcare based on 50 items in 5 domains: amenities, basic equipment, infection prevention, diagnostic capacity, and essential medicine (WHO SARA report).
‡The Good Medical Practice Index is an index of fundamental clinical action items across history-taking, examination, and counselling that should be performed at every patient visit regardless of service type. See Figure 1 for components of the index. assessed facilities, 11 452 clinicians had at least one direct observation of care in one service area. Of these clinicians, 15% had one observation, 25% had two or three observations, 46% had four or five observations, and 14% had six or more observations. The analytic sample is composed of 2223 clinicians in the first three years of practice." Table 1 describes characteristics of the 2223 clinicians providing each type of care across the nine countries. Just over half (56%) of clinicians were female, and 51% were in the nurse/midwife category. Physicians and associate clinicians each composed nearly a quarter of the remaining clinicians. A slight majority (52%) of clinicians provided sick child care alone, while 20% provided antenatal care alone, and 10% provided family planning care alone. The vast majority of clinicians (87%) were in their second or third year of practice since completing their health professions education, and 77% were practicing in the surveyed facility by their second year from graduation. Only 17% of clinicians had received relevant in-service training in the past six months, and 51% received supportive supervision within the same time frame. The number of surveyed clinicians varied across countries, ranging from Uganda with 86 clinicians in the first three years of practice to Tanzania with 620 clinicians. Clinicians largely practiced in public, non-hospital facilities, such as clinics or health posts, in rural areas. These facilities had a relatively low average structural quality score of 0.67.
The overall GMPI score for the sample was 0.50 (SD = 0.20). Item performance varied substantially (Figure 2). Within antenatal care, six of the 10 items had average completions scores at or above 0.80. The provider 'asks about bleeding in current pregnancy' had the lowest score at 0.31. Sick child care items scored slightly lower and had a wider range, with only three items scoring over 0.75, and six items falling below 35% completion. Clinicians only counselled parents about one or more danger signs requiring return to the facility 15% of the time. Most family planning index items fell between 45% and 65% completion. However, 82% of women were counselled about one or more issues with one or more family planning methods, and two index itemsasking about STI symptoms and asking desired timing of a woman's next child-scored the lowest of all items at 0.14 and 0.13 respectively.
Based on available sample size, Good Medical Practice scores were calculated in all nine countries for nurses and midwives, in five countries for physicians, and five countries for associate clinicians (Table 2) Figure 5). Malawi's associate clinicians were the lowest scoring cadre in any country, completing only 36% of recommended clinical action items on average across patient visits. Figure 6 and Appendix Table A3 present the results of the fully adjusted multivariable regression models, focusing on the effect of country on GMPI. The analytical sample included 2150 clinicians composed of 497 physicians, 1132 nurses and midwives, and 521 associate clinicians with complete data on covariates. We found that clinicians from Uganda and Kenya have higher Good Medical Practice scores on average than Tanzania. The best performer, Uganda, was 0.38 standard deviations higher than Tanzania (95% CI 0.16, 0.59), which equates to the completion of approximately one additional clinical action item on average. Nepal, Malawi, and Senegal, the lowest performers, completed approximately one clinical action item less than Tanzanian clinicians on average. Among physicians, only Rwandan clinicians performed more clinical action items than Tanzania, but this was not a statistically significant difference; Haiti, Nepal, and Senegal performed significantly lower than Tanzania, with Senegal 0.63 standard deviations below the reference (95% CI À0.95, À0.31). Among nurses and midwives, all countries performed more poorly than Tanzania (though coefficients for Uganda and Kenya were not statistically significant); nurses and midwives in Nepal performed over one full standard deviation lower (b = À1.21, 95% CI À1.41, À1.01). Finally, associate clinicians in Uganda and Kenya had Good Medical Practice scores that were significantly higher than Tanzania, and Malawi significantly lower. Uganda, whose associate clinicians were top performers,

Discussion
We assessed clinical performance among recent clinical graduates providing primary care in nine low-and middle-income countries and found that on average clinicians are completing fewer than half of the clinical actions for antenatal care, family planning, and sick child care. This is concerning as the Good Medical Practice Index represents the most basic items required for assessment or differential diagnosis and assignment of appropriate treatment. For example, performance on these core items is only slightly better than on more demanding metrics of performance (i.e. adherence to guidelines) in antenatal care, and even lower in sick child care [1][2][3]. Overall, quality of care is worse for sick child care compared to antenatal care or family planning services [1,21]. This study benefits from the use of large, nationally representative samples or censuses that included direct observations of care for multiple provider and service types, allowing for comparison of quality performance both within and across countries.
Higher clinical qualifications did not guarantee superior quality performance. Nurses and midwives outperformed physicians and associate clinicians in most countries, completing approximately one to two additional clinical actions on average across patient visits than other clinician types [22,23]. Nursing and midwifery practice, though, remains suboptimal, with a Good Medical Practice score of only 0.57. Many countries are investing in the associate clinician cadre, such as clinical officers and assistant medical officers, to expand access. Previous evidence on quality of care provided by associate clinicians is mixed [24,25]. In this study, associate clinicians provided the poorest quality among the three clinical cadres in nearly every country. Differences in service quality between cadres may be due to differences in emphasis during training or infrequent provision of certain services, such as family planning care by physicians; however, the overall poor performance of providers in the study suggests that training deficiencies exist for all cadres.
There were statistically significant differences in performance across countries. These persisted after controlling for factors that influence health system quality and investment, such as facility structural quality, in-service trainings, and supportive supervision. While some countries with higher average quality, such as Kenya and Uganda, outperformed countries with lower average quality across all three clinical cadres, certain national cadres    diverged from a nation's overall performance. For example, Tanzania ranks highly in physician and nurse/midwife performance, but performs at the median for associate clinicians. While overall scores were low, we identified particular deficits in patient counselling, suggesting a need for increased patient-centred education. Differences between countries and providers may be due to several factors, such a selection of candidates, which may be influenced by quality of secondary education, quality of clinical education for each cadre, and length and quality of clinical exposure during training.
Studies of clinical training programmes have identified several challenges, including a lack of practical exposure, poor standardisation of curricula, insufficient quality assurance systems, pathology-based training models, and failure to emphasise acquisition of clinical skills [26][27][28]. Some nations lack standardised competency verification procedures beyond graduation to ensure fitness to practice, or procedures for ongoing competency assessment [13]. As governments expand health professions education to increase human resources for health, the quality of education offered by strained systems may weaken further [29]. In particular, lack of infrastructure and resources for teaching and learning limits institutional capacity to provide consistently high quality instruction to students and to innovate in clinical education [30]. Reforming health professions education to address these deficits is a challenge in many nations due to scarce Index is an index of fundamental clinical action items across history-taking, examination and counselling that should be performed at every patient visit regardless of service type. See Figure 1 for components of the index. Good Medical Practice score is rescaled to have a mean of zero and a standard deviation of one. Estimates were obtained using ordinary least squares regression clustered at the facility level. All models were adjusted for facility structural quality, management type, provider sex, years of education, training and supportive supervision. The all-clinician model was also adjusted for provider type. Twenty-two providers were excluded from the models due to missingness for at least one covariate. [Colour figure can be viewed at wileyonlinelibrary.com] resources, yet it must become a greater priority for improving and sustaining the health care workforce in low-and middle-income countries [10,31]. The Good Medical Practice Index, while a conservative measure of clinical performance, underscores the importance of evaluating fundamental clinical performance across various types of care and ensuring clinicians are well-prepared for practice. In 2010, The Lancet Commission on Health Professionals for a New Century called for increased use of competency-based education to ensure high quality medical practice for all providers [11]. Given our limited knowledge of clinical competence in LMICs, their recommendations have been largely unfulfilled; however, they remain highly relevant given the findings of this and other studies. One approach to improve clinical performance is to use standardised patients, actors trained to present with specific medical conditions, which can be used as part of graduating or licensing requirements. Many countries have adopted objective structured clinical examinations (OSCEs) using standardised patients to assess competence before graduation [19]. OSCEs test a broad range of clinical skills including problem-solving, communication, decision-making, and patient management abilities. By contrast, written examinations test only cognitive knowledge, which is only one aspect of competence. These simulations of clinical practice have been found to be reliable and valid, although costly [32][33][34][35]. Adaptation of OSCEs and other tools to evaluate clinical performance for low-income settings should be a global priority.
Data used in this study were based on large, nationally representative samples of facilities or facility censuses from each country; clinical actions were recorded by trained observers, a gold standard in quality measurement. However, this study is subject to some limitations. Sample size was small in certain clinical cadres, limiting precision of estimates. Data were collected by trained observers, which could lead clinicians to behave differently (Hawthorne effect), and is subject to observer error. Other studies that removed the first observation from analysis found similar performance [36]. Hawthorne effect would bias our results upward, suggesting that actual practice may potentially be worse than observed here, thus strengthening the concern about pre-service education. There may also be residual confounding in regression estimates from unobserved variables, especially at the facility and/or national levels, which may impact a clinician's quality performance such as leadership and governance. Furthermore, as many facilities in the sample have only one clinician in the first three years of practice, our ability to test the impact of facility characteristics on clinical performance across providers is limited. This analysis uses broad clinical cadres for analysis; more granular categories, such as separation of nurses and midwives, would aid analysis. Given differences in clinician categorisation across Service Provision Assessments and varying clinical responsibilities across countries, we were unable to further disaggregate these categories. We were also unable to provide a comparison with recently graduated providers in high-income countries which limits inference. Finally, the GMPI assesses clinical performance by processes of care, which is only one component of clinical competency; a full assessment of competency would also include how well an activity was executed to determine whether differential performance between clinical cadres impacts health outcomes. However, the objective of this study was to identify a readily measurable set of clinical items that could be used to assess clinical performance across a range of services and providers. Given our limited knowledge of clinical performance in low-and middle-income countries, the GMPI remains an important contribution to understanding global clinical quality.
This study demonstrates overall poor clinical performance in outpatient care for primary care conditions among clinicians in the first three years of practice. While many low-and middle-income countries focus on the number and distribution of available providers, our results highlight a significant opportunity to improve quality and health outcomes through a focus on health professions education. Renewed attention and innovative approaches, including the use of objective evaluative tools and increased competency-based education, may provide an opportunity to better prepare clinicians for practice and ensure a high standard of care from every clinician in every patient encounter.

Declaration
The sponsor of this study had no role in the study design, data collection, data analysis, data interpretation, writing of the report, or the decision to submit this report for publication. The corresponding author had full access to all the data used in the study and final responsibility for the decision to submit for publication.   95% CI in parentheses. *The Good Medical Practice Index is an index of fundamental clinical action items across history-taking, examination, and counselling that should be performed at every patient visit regardless of service type. See Figure 1 for components of the index. †Good Medical Practice score is rescaled to have a mean of zero and a standard deviation of one. ‡Estimates were obtained using ordinary least squares regression clustered at the facility level. All models were adjusted for facility structural quality, management type, provider sex, years of education, training, and supportive supervision. The all-clinician model was also adjusted for provider type. Twenty-two providers were excluded from the models due to missingness for at least one covariate. ***P < 0.01, **P < 0.05, *P < 0.1