Implementation research to assess a health workers performance‐based management system in Nepal

To test and refine a performance‐based management system to improve health worker performance in Nepal.


INTRODUCTION
Broad consensus and evidence show that the performance of healthcare workers directly impacts health service quality, efficiency and equity (1)(2)(3)(4). A well-performing workforce is seen as essential to the success of any health system (1). Many of the countries that most need to improve health care performance are also those that experience what is referred to as a 'health workforce crisis' (5). In Nepal, this crisis is characterised by shortages of staff; recruitment, deployment and retention difficulties; fragmented approaches to resource planning, management and development; and poor motivation and staff performance (6,7). While all of Nepal's human resource challenges affect the availability and quality of health services, the focus of this research is on improving health workers' performance. A well-performing workforce is described by the World Health Organization (WHO) as one in which staff are available, competent, responsive and productive (1). Strategies that empower district-level health managers in Nepal to improve the performance of health workers will ultimately lead to the provision of more effective health services.
In Nepal, health worker performance has drawn attention from both domestic policy-makers and international development partners as an area in which improvements should be a priority (8)(9)(10). National policy documents (11,12) highlight poor staff performance, including productivity, quality and availability, as one of the five key human resources for health priority areas. While the causes of under-performance are discussed, and activities for promoting a well-performing workforce are set out, the case for further research has also been made. Little is known about what approaches and interventions are successful in developing a well-performing workforce in lower-income countries (1,13). The existing evidence suggests that human resource issues vary greatly according to the local context (14)(15)(16), and so contextually tailored interventions are required.
In the context of needing to improve health workforce performance and a lack of evidence on 'what works', in 2009, Nepal's Ministry of Health (MoH) supported the piloting of a performance-based management system (PBMS) in Doti hill district. Learning from the pilot was used to inform the refinement of the PBMS, which was then implemented in three further districts. The objectives of our research were twofold: to assess the effectiveness of the PBMS on both health worker and service performance and to explore the processes and context that facilitate or hinder the successful implementation of the PBMS. The expected output was a tested and improved PMBS, including an implementation guide and training materials, which could be implemented more widely at district level. The development of a robust, context-informed PBMS was timely in supporting districts' responses to the Government's Human Resources for Health Strategic Plan (2011-15) (6).
The purpose of this study was to report whether, and if so, how and why, the PBMS was able to strengthen workforce performance and improve health indicators. The study contributes to much-needed knowledge on context embedded evidence-based interventions to address human resources for health.

METHODS
A mixed-methods implementation research approach (17) was used to explore whether the PBMS improved health worker and service performance, and the factors that facilitated or hindered the PBMS's effectiveness. The iterative approach to improvement that implementation research promotes (17) lent itself well to refining the PBMS. Concurrent mixed methods (18) were used at each stage of the research process to enable understanding of multiple perspectives (i.e. district officials, health workers and community representatives), processes and outcomes. Both impact and output indicators informed by the WHO Indicators for assessing health worker performance (1) were used to measure the impact of the PBMS. Outcomelevel indicators included service, staff and essential drug availability, and service utilisation in relation to specific maternal and child health services. Output indicators included health worker job satisfaction and staff presence. The implementation processes and technical and financial inputs were also monitored to determine whether and, if so how, they influenced the success of the PBMS.

Collaborative approach
Consistent with an implementation research approach, a collaborative relationship was fostered between the researchers and those overseeing and implementing the PBMS. The key objectives of collaboration were to strengthen implementers' engagement with the research, and ensure responsiveness to implementation issues and challenges. The research team consisted of individuals from the Health Research and Social Development Forum (HERD), the Liverpool School of Tropical Medicine (LSTM) and the University of Leeds (UoL). Implementation of the PBMS was overseen by central-and district-level government staff (see Fig. 1). Existing Health Facility Operations and Management Committees (HFOMC), comprising of a variety of community representatives and responsible for operation and management of local health facilities, managed local PBMS implementation issues and ensured community participation. In order to support collaboration between the researchers and implementers, members of the research team were represented on the Technical Working Group (central government level) and Performance Evaluation Committee (district level). Researchers and implementers made collaborative decisions, with implementation and research issues informing each other in an iterative and continuous manner. Collaborative activities included the joint preparation of the research protocol and decisions relating to the refinement of the PBMS.

Implementation strategy
The research took place in three distinct phases over a 28month period. These phases are outlined as part of the implementation research cycle in Figure 2. This study reports the methods used and findings from Phase 3 which consisted of four key stages and related activities, outlined in Figure 3.
Detailed guidelines and tools were prepared by the Technical Working Group to support districts and facilities to implement each stage of the PBMS and to ensure consistency in implementation across districts. The Performance Evaluation Committee supported the planning, implementation and monitoring of the PBMS across districts, while each facility's HFOMC met monthly to support facility managers and staff with any implementation issues.

Study areas and sites
The three purposively selected intervention districts included a hill, mountain and plain (Terai) district to reflect Nepal's geographical and cultural diversity.   Kathmandu. Rural and remote districts were included as it is particularly difficult to recruit and retain staff in these areas. Within each district, five health facilities were chosen to implement the PBMS. The facilities were selected on the basis of type (four health posts and one primary health care centre per district), performance (high-, average-and low-performing facilities represented) measured against the key indicators in the last year's district annual health review, staff vacancy rate (to ensure that recruitment and retention issues did not bias performance) and available services (a birthing centre). Facilities were selected across different electoral constituencies within each district.

Data collection and recording
Data collection was carried out over a 12-month period by HERD, with technical support provided by LSTM and UoL. One full-time HERD research officer was embedded in each district to ensure the robust and timely collection of data, and where possible were from or had a pre-existing relationship with that district. Table 1 describes the data sources and tools that were used to gather evidence to explore input, effect and output indicators.
Qualitative data: Interview participants were purposively sampled to ensure they were representative in relation to health worker role and level (i.e. health worker, facility supervisor and district manager), ethnicity, gender and type of health facility. Local leaders, women's groups and patients were selected as focus group discussion participants Topic guides were used to facilitate the interviews and focus group discussions. All sessions were recorded and then transcribed. Observation of the implementation process was carried out by the HERD research officers who were trained on using a structured observation guide during the study period and observations were recorded through regular visits, events and workshops, and reviewing health facility records, registers and the minutes of meetings. Research officers recorded observations by note-taking and use of a reflective diary. Observations records were verified by the HERD central research team during district monitoring visits which helped to address intra-and interobserve variation biases.

Quantitative data
Changes in health worker job satisfaction were measured at the start and end of the 12-month study period. For the purposes of planning (in Phase 1), the scale of Mobindyo and colleagues (18) was used to estimate the required sample size. Assuming a 10% change (i.e. 3 points) is of psychological significance, the tool estimated that a sample of 280 health workers was required to complete the questionnaire at baseline, and a different sample of 280 from the same facilities 12 months later. This sample would provide 86% power to detect a change in mean score, with an alpha level of 5%. Allowing for a 10% refusal rate, 250 analysable questionnaires would be available at both assessment times. However, on the exclusion of Rawusa (post-April 2015 earthquake), in order to gain a large enough sample at the end of the PBMS study period, all health facility staff in Baitadi and Dang were asked to complete the questionnaire.
Following the piloting of the tools of Mbindyo and colleagues (19) and Spector and colleagues (20), and further validation studies of similar tools in Nepal, the Spector questionnaire was selected to assess changes in health worker job satisfaction. The questionnaire uses 36 items to assess employee attitudes to their jobs under the areas of: pay, promotion, supervision, fringe benefits, performance-based rewards, operating procedures coworkers; nature of work and communication. There are six choices per item ranging from 'strongly disagree' to 'strongly agree'. Items are written in both positive and negative directions.
All health workers present on the day of data collection at the start and end of the 12-month study period were asked to self-complete the job satisfaction questionnaire. A HERD research officer provided clarification and support as required. Questionnaires were checked by a HERD supervisor (from the central research team) to ensure they were appropriately and fully completed. All questionnaire data were double-entered into the Census and Survey Processing System (CsPRO) and missing data and anomalies identified. Questionnaires were given a unique identifier to ensure anonymisation of data, enable robust data management, and the ability to go back to the original questionnaire if required.

Routine data collection
Routine district-level data on service utilisation were collected from Nepal's Health Management Information System (HMIS) and verified at district level as required. Drug registers were used to collect information on the availability of essential drugs. Attendance registers indicated whether staff were present, based on the day the field worker visited the facility.

Ethics
Ethical approval to conduct the research and use the findings to inform policy and practice was obtained from both the WHO Ethics Review Committee and the Nepal Health Research Council (NHRC). Informed verbal consent was obtained from each participant prior to participation in the study. Written informed consent was also obtained from those participating in interviews and discussions. The option to end participation at any point was explained, and the confidentiality of participants was respected at all times.

Data analysis Qualitative
Analysis of qualitative data obtained through field observations, interviews and focus group discussions was conducted in line with Ritchie and Spencer's (21) 'framework approach'. Early analysis informed the refinement of the topic guides. HERD, with support from LSTM and UoL, developed the coding frameworks, and HERD research officers analysed the transcripts.

Quantitative data
Checks were run on CsPRO (version 5 United States Census Bureau, Washington, DC 20230, USA) to identify missing data and any anomalies, and data were exported to SPSS (version 20, IBM Corporation, Armonk, New York 10504, USA) to carry out analysis. Levene's test for equality of variance and the independent t-test were conducted to identify any significant differences in health worker job satisfaction at the start and end of the study period.

Routine data collection
Frequency tables were used to compare service utilisation in the year prior to the implementation of PBMS, and after one year of implementation. A review of trend data from health sector annual reports over the previous five years (where available) avoided over-interpretation of data peaks.

Job satisfaction
A total of 264 health workers completed the job satisfaction questionnaire at the start of the study, and 325 health workers completed the questionnaire at the end. In the hill district of Baitadi, job satisfaction significantly improved in relation to at least one indicator within seven of the satisfaction domains (communications, contingent award, co-worker, fringe benefits, nature of work, operating procedure and promotion). Six of the eight indicators where improvement was demonstrated relate to positive statements, and two relate to negative statements (i.e. 'I have to work harder because of the incompetence of colleagues', and 'I have too much work to do'). In Baitadi, the only domains where no difference was seen across all indicators were 'pay' and 'supervision'. In Dang, job satisfaction increased in relation to at least one indicator in three domains (promotion, supervision and co-worker).
Three of the four indicators where improvement was demonstrated in Dang relate to positive statements and one relates to a negative statement (i.e. 'supervisors have little interest in the feelings of subordinates'). The permanent workforce reported more significant changes in job satisfaction than the temporary workforce (see Table 2). In relation to absenteeism, there were significantly fewer staff not in work in Dang after the implementation of the PBMS (odds ration [OR] 2.28, confidence interval [CI] 0.09-0.84). In Baitadi, there was also a reduction in staff not at work, although it was not significant (See Table 3).

Service utilisation
Across the intervention facilities in Dang and Baitadi service utilisation increased in relation to immunisation coverage (Dang: BCG increased by 8%, DPT-Heb B-3 [as a proportion of children who received DPT1 and also DPT3] by 12%, measles and rubella by 14%; Baitadi: BCG by 14%, DPT-Heb B-3 by 11%). The proportion of women who had a first antenatal check-up also having a fourth check-up also increased (Dang: 11%; Baitadi: 38%), as well as the proportion of women giving birth in a healthcare facility (Dang: 15%; Baitadi: 43%). In both districts, there were decreases in the proportion of pregnant women attending their first antenatal clinic check-up (Dang: 23%; Baitadi: 18%), and in the coverage of growth monitoring of children under 2 years of age (Dang: 15%; Baitadi: 15%) (see Table 4). Essential medicines were out of stock less frequently at the end of the study period in Baitadi, but more frequently in Dang.
Perceptions on the effectiveness of PBMS A total of 16 interviews took place in each district: 10 with health workers, four with members of the HFOMC and two with district supervisors. Research officers also undertook 15 observational visits in each district over the 12-month study period. During the visits, key documents, including minutes of both staff and HFOMC meetings and availability of treatment guidelines, were reviewed. In both Dang and Baitadi, staff indicated that PBMS contributed to improved health worker performance (i.e. through improved technical skills), and that this in turn led to improvements in service provision. Interviewees linked better performance with improvements in service quality, coordination of services, resumption of some services and the attitude of service providers to both service users and the local community.
'Good results have been achieved after the implementation of PBMS. For example, last year Salena Health Post was ranked in between 20th to 25th position in Baitadi. This year, it was ranked in the top five health facility. PBMS contributed to this achievement. Health facility staff set the benchmark and performance targets and allocate responsibility for all health workers with specified their activities. With successful completion of assigned activities at individual level, the institutional achievement has increased. . . the "total achievement" of the entire Baitadi District has increased by 3% this year as compared to last year. It is because of this [PBMS] that it has increased'. District Supervisor discussing the impact of PBMS during an in-depth interview.
In relation to performance planning and benchmarking, the majority of participants found the benchmarking process labour-intensive, but a worthwhile exercise which was   necessary to inform the health facility's Annual Plan. The increased focus on benchmarking and monitoring progress supported good information management and was helpful in identifying service delivery issues more quickly and ensuring a timely response. Assigning health workers to specific actions within the Annual Plan was viewed as helpful, as it encouraged health workers to take greater ownership over specific areas of work, increased health worker accountability and generated a positive team spirit. The involvement of the HFOMC (which consists of community representatives) in the development of the Annual Plan was also seen as crucial by health workers and district supervisors. They stated that HFOMC's involvement secured the engagement of the community in identifying areas for improvement and possible solutions, and their commitment to the successful implementation of the Plan. 'I personally feel and their [co-workers'] perception about benchmark setting is "positive". They think that it is necessary and that they have to do it anyhow. When the committee as well as all the staff sit together, discuss and set the "benchmarking", they feel that it won't be good if they can't fulfil the benchmark set by them. Hence, it stirs everyone at all the times to fulfil the target of the benchmarking'. Health worker in a discussion about the benchmarking aspect of PBMS during an in-depth interview.
The performance monitoring and evaluation aspects of the PBMS were well received by health workers and district supervisors. The monitoring of health facilities and workers by a team of district health supervisors and community members was seen as positive, helping to pro-actively identify service delivery issues and develop local solutions. The approach was also considered to be more objective, transparent, systematic and fair compared to previous practices and systems. Staff generally appreciated that they would be assessed against a transparent set of indicators and by multiple people (rather than just one). They said it increased their sense of responsibility, motivated them to work harder, and strengthened coordination among staff. The recognition of high-performing facilities and health workers was also seen as motivating. The involvement of service users as 'assessors' was, however, seen as problematic where service users did not have the information they required (i.e. feedback from other patients) to answer questions such as whether the healthcare professional was on time or maintained the privacy of patients.
'The PBMS is quite "systematic" if we look at it. If anyone asks on what basis he/she has been evaluated, we can answer that it is based on so and so indicators as well as based on their job description, responsibilities, performance and satisfaction of the service users with the services provided by them. . .. Instead of the system of marking by one individual, the marking is done by all and average is taken. So, if there is any gap, there is a collective ownership which will motivate us to perform well'. District Supervisor discussing the impact of PBMS during an in-depth interview.
A number of interventions and tools were utilised under the PBMS. They included outcome-based job descriptions, a 360-degree appraisal tool and supportive supervision.
While some voiced concern that health workers were using the 360-degree appraisal tool to evaluate their peers on the basis of their relationship rather than performance, outcome-focussed job descriptions were seen as positive. Staff stated that they improved health worker accountability, ensured greater clarity in relation to roles and responsibilities and contributed to improvements in service delivery performance.
'When the facilitator started listing all the work health facilities should do, I was surprised that the limited number of health workers are doing so much of work. The job description-making workshop helped us to understand clearly about roles and responsibilities of each of the health workers and HFOMC members'. HFOMC member in a discussion about the PBMS during an in-depth interview.
Supportive supervision was seen to improve communication between district managers and health workers, and lead to more constructive feedback, recognition and praise of good work and greater commitment from health workers to improve their performance. District supervisors also saw supportive supervision as an opportunity to share and communicate learning on how to best improve health worker performance. These findings support the results of the job satisfaction survey that demonstrated an increase in satisfaction in relation to 'organisational communication' and 'recognition of good work'. The checklist used to support supervision was found to be particularly helpful in improving reporting and tracking progress against agreed targets. A challenge raised in relation to supportive supervision was that there were a limited number of district supervisors, which meant that they did not always have time to supervise. Some supervisors need to be better trained and equipped to carry out their work.
'Before we used to be scared when there was supervision. Now, everyone gets excited when there is 'supervision'. Health workers think that the team coming for supervision will also praise their good works and they will also know about their weakness and get excited to correct them. The feedback that they [supervision team] provided has helped clarifying confusion in recording and reporting. This has also helped availability of staff'. Health facility manager discussing supportive supervision during an in-depth interview.
Full implementation of the PBMS could not be completed in the mountain district of Rasuwa, which was badly affected by the earthquake of April 2015, including damage to health facilities. As analysis of PBMS at the end of the study period could therefore not take place, the mountain district was excluded from the analysis.
The collaborative approach that was used to support implementation of the PBMS was seen as crucial. Interviewees said that the involvement of and leadership demonstrated, at all levels of the health system (central, district and local) meant that the implementation of the PBMS was well supported and coordinated. Committee meetings (from central to local level) served as a platform for joint decisionmaking in relation to implementation issues and the coordination of activities, and led to timely decisions being made during the study period, for example in relation to halting the intervention in Rawusa after the 2015 earthquake. The leadership provided was also seen as crucial in engaging and gaining the commitment of managers, supervisors, health workers and members of the community to successful implementation of the PBMS.

DISCUSSION
The PBMS has the potential to increase health worker job satisfaction, as demonstrated by the increase in satisfaction across six domains of the Spector tool. The relationship between the PBMS and job satisfaction is not linear, and a variety of determinants influence the satisfaction of different health workers at different times. Nevertheless, the nature of the intervention and the findings of the qualitative research make it likely that the PBMS contributed to improved satisfaction in some of the six domains, and specifically in relation to 'organisational communication', 'recognition for good work' and 'those who do well have a fair chance of promotion'. Where job satisfaction decreased over the study period, it is not possible to attribute these changes to the PBMS. However, because the intervention required health workers to become more accountable for achieving particular outcomes, it is likely that it contributed to health workers' perceptions that they have too much work to do. A limitation of the methodology is that two cross-sectional samples (at the start and end of the study period) were conducted rather than an individually linked survey. This was due to ethical concerns about the identification of health workers and the high turn-over of staff, although it does limit the ability of the study to demonstrate improvements in individual job satisfaction.
Dang experienced a significant reduction in the proportion of staff not in work at the end of the PBMS implementation period, which the PBMS may have contributed by enabling conducive work environment. However, as 'leave' was the main reason for staff not being at work, this could have been planned leave, which is not necessarily related to performance management.
The qualitative findings of this study suggest that PBMS was helpful in improving health worker performance, and led to observe improvements in service delivery. The service utilisation data to some extent support the perceived link between improvements in health worker performance and service improvement, with some child and maternal health services experiencing an increase in utilisation. However, there were also reductions in service utilisation for some indicators, and while it is unlikely that PBMS contributed to these reductions (as Annual Plans set targets for improvements across these areas), the reasons for this should be explored. There were also limitations with the service utilisation data; not all the health facilities properly managed all the recording and reporting systems, and some of the data were of poor quality, which led to its exclusion. This situation limited the assessment of service utilisation to maternal and child health indicators only. One of the lessons learned is that bringing about change in the performance of health workers, and subsequently service performance, requires considerable time and resources. Future research should allow for a longer evaluation period so that any changes may be better identified. This would also enable a wider range of performance indicators (and potentially confounding factors) to be measured and included in the analysis. The mixed picture in relation to essential medicines makes it difficult to identify whether or not the PBMS had an impact on drug availability.
A number of processes and inputs need to be in place for the PBMS to be implemented successfully. Job satisfaction and the performance of health workers are directly linked with the enabling environment where they work. The health facility environment should therefore be functional, with (as a minimum) key posts filled, basic physical facilities and a reliable and timely supply of essential medicines. A collaborative approach with central, district and local level leadership is also very important for the successful implementation and sustainability of the PBMS. Central government commitment and leadership through good governance arrangements (i.e. the Steering Committee) were found to be invaluable in ensuring that the PBMS was implemented across multiple health facilities and districts, and in a coordinated and sustainable way. At the same time, the involvement of the HFOMC at the health facility level (i.e. in setting targets, developing annual and improvement plans) helped to secure local ownership of the PBMS and the active participation of staff and community members. These findings support those of Dieleman and colleagues (13) who cite the involvement of local authorities, communities and management, and active involvement of local staff to identify and implement solutions to problems, as critical in the successful implementation of human resource interventions.
A number of contextual factors influenced the effectiveness of the PBMS, and help to explain why it may have been more effective in some health facilities than others. A nationwide health worker strike from December 2014 to January 2015 led to delays in training staff on the PBMS at the health facility level across all districts. Frequent transfers of staff to and from some health facilities also affected the delivery of planned activities, as new staff did not have adequate knowledge about the PBMS. Those health facilities with a workforce that was more informed about and committed to implementing the PBMS tended to be those that were higher-performing by the end of the study period. PBMS guideline clearly states six broad key areas and specific indicators for evaluation of each health facility performance. Targets were set for each indicator at the beginning of the fiscal year which were measured at the end of fiscal year to determine the performance level of health facilities. These were also where key elements of the PBMS were working well; particularly the annual performance management process, supportive supervision and group performance appraisal (at facility and individual level). Critical to these elements working well were the provision of technical support from central and district government, especially in benchmark setting and performance evaluation. The availability of well-trained supervisors to participate in supportive supervision was also key.
The greatest strengths of the PBMS were perceived to be its robust approach to performance planning and evaluation, supportive supervision and feedback, outcome-based job descriptions and a fair and transparent reward system. Staff felt that all of these components led to improvements in their performance. This study therefore supports the link between supportive supervision and reward systems and improved performance (2). As less is known about the links between performance planning and evaluation and health worker performance, this study provides a useful insight into how one can inform the other.
The implementation research approach was invaluable in enabling discussion and decision-making to take place, which informed the refinement of the PBMS model and the research process. The governance structure formed to oversee the planning and implementation of PBMS (Fig. 1) played an important role in ensuring that there was a space where such discussion and decision-making could take place. It also ensured the coming together of both policymakers and researchers periodically over the study period. The identification of a governance approach that supports policy formulation and implementation is of importance given that governance has been a neglected issue in the field of human resources for health (22,23). The active involvement of the MoH and WHO (i.e. on the Steering Committee and Technical Advisory Group) also meets Rowe and colleagues' (14) call for greater national government and international organisation involvement in helping to translate research findings into action to improve health worker performance. The focus on process, and how and why the PBMS was working well or not, also enabled identification of linkages between human resource management, performance and implementation, which little research has attempted or been able to address (24).

CONCLUSIONS
Improving health service performance demands a collective effort of individual health workers and facility management committee. In the light of the challenges relating to health worker performance across Nepal, a contextually tailored PBMS that has both demonstrated potential, and which can be implemented in a range of contexts, is helpful. Key advantages of the PBMS are that it offers a processes and tools to link facility-wide human resource management with improvement in health workers satisfaction thus leading to improved performance. Various actors of district health system and health facility and prevailing local context determine implementation of PBMS and its impact in improving health facility performance. Critical to its success are leadership and commitment to its implementation by those at all levels of the health system. This includes a commitment to make available the human resources and expertise that can ensure ongoing monitoring, supervision and performance improvement activities. Ideally, the PBMS should be integrated with other human resource programmes, as part of ongoing good practice.
This research also demonstrates the effectiveness of implementation research in testing and refining interventions in a way which maximises collaboration between the researchers and implementers at each stage of the research process.