A. M. Petrunkina1,2*
Communication to the Editor
Algorithm and metrics for a standardized evaluation of cell sorting service delivery
Article first published online: 2 AUG 2013
Copyright © 2013 International Society for Advancement of Cytometry
Cytometry Part A
Volume 83A, Issue 7, pages 602–607, July 2013
How to Cite
Petrunkina, A. M. (2013), Algorithm and metrics for a standardized evaluation of cell sorting service delivery. Cytometry, 83A: 602–607. doi: 10.1002/cyto.a.22297
- Issue published online: 2 AUG 2013
- Article first published online: 2 AUG 2013
- Manuscript Accepted: 20 MAR 2013
- Manuscript Revised: 17 MAR 2013
- Manuscript Received: 11 JAN 2013
To the Editor:
Specialist flow cytometry services in the biomedical environment are mostly organized, coordinated, and operated through the management of central multi-user core facilities or departments, both in academia and commercial organizations. The major specialist components of the service provided by most core facilities include cell sorting, training of internal and external customers, supervising and assisting in analytical experiments, and maintenance and development of equipment and assays.
Cell sorting is probably the most time-consuming component of the service provided by the technical specialists involved in the running of facilities. The importance of evaluating usage and efficiency of cell sorting services for research institutions has always been recognized, but the significance of implementing standardized procedures, algorithms, and key performance indicators that could be applied across resources and facilities has not yet received sufficient attention. Indeed, almost every flow cytometry facility providing sorting services will have their own metrics for evaluating output and capacity, and, given the complex contributions that core facilities provide for research institutions, it is not surprising how heterogeneous is the variety of performance metrics and approaches. Sorting hours, numbers of subscribers, publications output, revenue, cost recovery—the list of evaluation parameters can be endless. Although the Herculean task of bringing all of these approaches under one umbrella is probably unachievable, and may not even be ultimately desirable, nevertheless it would seem that at the very least a standardized comparison of sorting capacity, usage, and efficiency could be modeled in a way that would be applicable to any particular sorting resource. An adequate delivery measurement system, once developed and implemented, can be applied for monitoring and sustaining the services in order to achieve organizational goals.
In the vast majority of cases, the capacity of the sorting service is defined either as the equipment capacity for the existing sorting resource or as the maximal capacity of staff working full working hours. It is, however, important to distinguish between equipment and staffing capacities and also to evaluate resources in the context of the real working environment (by taking into account other duties demanded by job descriptions, infrastructure, and operational strategy). A recent report by Box et al.  does highlight the difference between theoretical and practical capacity of sorting resources, and gives indications why the practical capacity is lower than the theoretical one (non-productive hours, staff capacity, other duties); however, being primarily focused on cost accounting, it does not describe in detail how the theoretical and practical capacities can be calculated.
The major aim of this short report is to highlight the rationale behind the differences in practical and theoretical utilization and to provide a mathematically formalized way of calculating the capacity of the sorting resources and defining the metrics (or KPI, key performance indicators) for capacity utilization and efficiency.
Definition of Sorting Capacity
|Equation title and content||Mathematical expression||Equation No.|
|Summation of parameters from the Table 2 related to any given number of staff N (or working days M), e.g., A1 + A2 +…+AN is achieved by using the summation operator and the summation indices (i or j, running all consecutive numbers from 1 to N or M: = A1 + A2+…+AN)|
|Maximal theoretical sorter capacity (instrument time theoretically available for operation)|| ||(1)|
|Maximal theoretical staff capacity (staff time available for operation of instruments)|| ||(2)|
|Theoretical sorting capacity per staff unit per day (it shows how many hours on average one member of staff will deliver per day)|| ||(3)|
|Sorting usage (utilization of the resources)|| ||(4)|
|Cumulative number of operating days for all staff (real operating capacity for report period)|| ||(5)|
|Practical staff sorting capacity|| ||(6)|
|Capacity efficiency coefficient (ratio of practical capacity to theoretical capacity)|| ||(8)|
|N||Number of staff working in the facility and providing regular sorting service|
|Hc||Staff contractual hours per annum|
|P||Fraction of staff time dedicated specifically to sorting service. This would usually include set-up, configuration, operation and troubleshooting of the equipment cumulatively defined below as t, but would not include regular maintenance. In most cases, P is specifically defined in the Job Profile and reflects facility's service strategy. If not defined, an approximate estimation of P should be performed in advance.|
|A||Authorised absences (that will include sick leave, training days, emergencies, unpaid leave, annual leave etc)|
|t||Operating time other than actual sorting (i.e., set-up, configuration etc), or other essential duties which require time otherwise dedicated to operation of sorters but which cannot be classified as sorting service delivery: in service training, demonstrations, teaching|
|Sorting capacity based on the number of operating staff|
|Sorting capacity based on the available equipment|
|K||Number of sorters|
|M||Theoretical number of operating days per annum (in one-man facility, this would be the number of weekdays minus leave and bank/public holidays; in multiple-staff facility, this would be the number of weekdays minus public holidays). In UK, there are 8 public holidays, thus assuming 365 days p.a. and 52 weekends, there are 253 working days p.a.|
|C||Closures (efficiency closure, public holidays, work exchange schemes, refurbishment, upgrades etc)|
|Srecord||Actual sorting hours, recorded within a certain period (e.g., for monthly report, or retrospectively for the year)|
Then maximal theoretical sorter capacity will be defined by the number of sorters and time available for their operation, the latter limited by the staff working times according to their contracts and to labor legislation, less downtimes and times required to set up the equipment [Eq. (1)].
The theoretical sorting capacity (time in hours which can be dedicated specifically to sorting experiments) in terms of staff will be defined on the basis of staff contractual hours, percentage of time dedicated to the operation of sorters, and time required to set up, configure, clean, and shutdown the instrument [Eq. (2)]. Many contracts in UK specify time dedicated to the operation of instruments (P). However, in some situations such specification will not be available because of aiming to provide flexible service delivery. If sorting is a priority, staff capacity must expand or contract according to bookings. If P is not defined in the job description, or no specific job description does exist, a preliminary step must be taken to calculate this percentage, based on other duties, such as training, maintenance, organization, management, research, administration, defined by the size (both in terms of number of users and instruments) and strategy of the facility. An alternative procedure would be to proceed as described by Box et al.  and assign typical tasks performed by the facility staff along with their duration. In UK academia such an exercise will be associated with Higher Education Role Analysis (HERA) and potential grading of the post (e.g., http://www.admin.cam.ac.uk/offices/hr/grading/hera.html). In any case, it would be useful to define a reasonable range for the percentage of time that could be dedicated to the operation of sorters, in order to put evaluation mechanisms in place.
If the number of operators N is different from the number of sorters K, and/or staff members have other responsibilities beside operating sorters, and/or there have been downtime periods, staff sorting capacity will differ from equipment sorting capacity. In the case where the number of operating staff is smaller than the number of sorters, there will be some equipment under-utilization, whereas in the case where the number of staff is greater than the number of sorters, there may be a need to stagger shifts if the demand is too high. By defining standardized usage and efficiency parameters one will be able to use and as indicators for evaluating staff and equipment capacity and for monitoring possible need for upgrades.
Definition of Utilization
Ideally, sorting usage (utilization of the resources) is defined as the ratio of the number of actual delivered hours (sorting record) to the theoretical capacity [Eq. (4)].
However, such a representation does not show how the resources available within the report period were used, neither does it show how efficiently the resources were managed and/or organized. Here, a different approach is suggested.
The real operating capacity OD (cumulative number of practical operating days) can be defined as the number of working days corrected for any closures other than public holidays, authorized absences, and downtimes of the instruments, when the number of operators available on each particular day has been taken into account.
The practical staff sorting capacity can be described as a product of hours per staff unit per operating day and number of practical operating days [Eq. (6)].
Then for any period in question the utilization U related to the actual sorting hours Srecord is given by the ratio of sorting record and practical staff sorting capacity [Eq. (7)].
Definition of Efficiency
A second delivery metric for sorting capacity is the capacity efficiency coefficient which is defined [Eq. (8)] as the ratio of practical capacity to theoretical capacity (or, in other terms, the ratio of practical operating days [see Eq. (5)] to the product of the nominal number of working days M and the number of operating staff N).
From Eq. (5), substituting the average number of sick days for UK (5: see http://www.direct.gov.uk/en/Employment/Employees/Sicknessabsence/DG_185054), the average number of training and closure days not included within job descriptions , the average UK holidays entitlement in an academic institution (28), and downtimes under 3% (less than one day in month, 12 days p.a.), and the nominal number of 253 working days for M, we come to the following result:
As in all real physical processes (according to the laws of thermodynamics which state that the ratio of a useful effect to the cumulative energy expended cannot exceed one), the efficiency of the resource's capacity is always bound to be less than 1. According to the calculation above, it seems reasonable to define the range/threshold for an efficient service delivery as E = 0.8. The magnitude of the efficiency coefficient will be affected by the smoothness of operational processes (proportional staff attendance, management of downtimes, and breakdowns, etc.).
If the authorized absences other than annual leave and statutory closure/training days do add up to more than 10 days, and downtimes are higher than 3%, the efficiency coefficient will be under 0.8 and that will be an indicator to optimize the service.
Obviously, the product of utilization and the efficiency coefficient of the resource capacity will yield
which is identical to Eq. (4) in Table 1. The theoretical or maximal capacity, however, is a performance metric that represents an invariant, and therefore, although favored universally, does not reflect real conditions.
A Practical Example
The procedure described above can be applied to a comparison of monthly usage in two facilities (see also Supporting Information Excel data sheet):
One facility is staffed by one member of staff operating a BD FACSAria™ III cell sorter. The operator is employed at a scientist level, and their job description defines sorting duties at 50% frequency. In August, the operator was absent for three days attending a scientific conference, and for another two days attending a training course; the instrument was down for 5 days. The sorting output was recorded as 30 h.
The second facility is staffed by two technicians co-managing the facility, each with 65% of their duties allocated to the operation of two Beckman Coulter MoFlo™ cell sorters. In August, one of the operators was on sick leave for two days, and the second operator had five days of annual leave. They had no downtimes. The sorting output was recorded as 90 h.
At first sight it seems that the second facility had a higher usage. Indeed, two people produced 90 h between them (45 h per operator), while in the first facility only 30 h were delivered. Did the first facility perform significantly worse in terms of utilization of resources available to them?
Let us apply our model to both situations. Assuming hypothetical contract conditions, one would expect the operator in the first facility to work on average 40 h/week and deliver about 1,760 working hours per annum, with allocated 33 days holidays and 8 days public holidays. The technical staff would be expected to work on average 36.5 h a week (about 1,640 h per person per annum), with annual leave each of 36 days (including public holidays). Setting up times for MoFlo and AriaIII sorters would vary according to the complexity of the configuration and to the number of sorts run every day. For the purpose of this exercise it was assumed that 2.5 h a day required to set up, calibrate, and decontaminate two MoFlo sorters at the beginning and during the day and 1.5 h required to set up, calibrate, and decontaminate the Aria III, for 2 experiments each.
Thus, substituting into Eq. (2) for the first facility: M = 365 − 104 − 41 = 220, Hc = 1,760, P = 0.5, and t = 1.5 h, the theoretical capacity of the scientist providing the service is (1,760 · 0.5 − 220 · 1.5) h = 550 h and Coperator per day = 550÷220= 2.5 h
There were 10 operating days in August: 21 days – 1 public holiday – 5 days downtime – 5 days meeting and training. Thus, the total practical operating capacity was 25 h. By providing 30 h service, 120% of this capacity was delivered.
By substituting into Eq. (2) the figures for the second facility: M = 365 − 104 − 8 = 253, Hc = 2 · 1,640, P = 0.65, and t = 2.5 h, the theoretical capacity of the technicians is (0.65 · 2 · 1,640) − (2.5 · 253) = 1,500 h and Coperator per day = 1500 ÷ 506 = 3 h. There were 33 operator-days in August: OD = 2 · (21 − 1) – 2 – 5 = 33 and therefore practical operating capacity = 99 h; hence sorting record of 90 h represents about 91% of available practical capacity.
Thus, although it seems that the first facility delivered less, in fact its sorting service utilization was higher, at 120%; in the second facility only ∼91% of available resources were utilized.
Furthermore, in terms of capacity efficiency coefficient E, the first facility used 10 days out of 20 theoretically available (E = 50%), whereas the second one utilized 33 out of 40 days (E = 82%).
Clearly, using our model allows one to examine the impact of different factors involved in the evaluation of usage. The example indicates that, given the existing demand, the first facility is not staffed appropriately, since one scientist with 50% sorting duties, long holidays, and planned absences for conferences and career development cannot run the facility efficiently without technical support; either they will end up working unreasonably long hours, or the excessive demand will result in longer waiting times, customer dissatisfaction, and complaints. Although one could argue that this should be intuitively clear to the operating staff, still, the intuition would not satisfy the executive officers or administrators who may be far removed from the workfloor, unless they were provided with the quantitative metrics described here. Moreover, the low level of service output could have been mistakenly attributed to staff performance if no mechanisms were in place to judge practical capacity objectively.
Algorithm for Facilities Development
The flowchart in Figure 1 provides a means of analyzing sorting usage and assessing potential further actions. The practical approach is first to calculate the product of capacity per operator per day and the cumulative number of real operating days as the practical sorting capacity. Thereafter, the utilization of sorting resources (as ratio of sorting record to practical staff capacity) needs to be calculated and monitored on a regular basis. If the utilization of sorting resources is below 100%, it is likely to be caused not by inadequacy of human or instrumental resources but by low demand or lack of marketing. In the first instance, the core facility would need marketing, and, if possible, a broad development of the services it is offering. Such development could involve organizing seminars, open days, mini-symposia, user tours, and/or developing staff-employed in the facility in order to provide a better service. If the utilization is constantly at levels above 100%, the staff sorting capacity and the equipment sorting capacity would need to be compared. If the staff capacity exceeds the equipment capacity, there is a clear case for commissioning an additional piece of equipment, as justified by the metrics of usage. If the staff capacity is lower than equipment capacity, it could be a case of equipment under-utilization. Whether this under-utilization is due to inadequate staff levels or due to suboptimal efficiency of the service has to be investigated. In the first instance, a remedy could be provided by reviewing the operational strategy and job description/task allocations in order to provide maximum flexibility. This should be followed by the optimization of core processes to increase the efficiency coefficient. However, if the efficiency coefficient of capacity realization is at (or can be brought to) optimal levels, and the practical staff capacity still well below equipment capacity, the next logical step would be to hire an additional operator to keep up with the demands of the service.
Metrics of Service Delivery as Indicators for Performance Benchmarking
The discussion as to what levels of sorting usage are considered to be adequate is very popular. Cytometry forums often have interactive surveys trying to exchange experiences and define “reference” values. However, as noted by Box et al. , different facilities will have varying capacity limits for their resources depending on particular circumstances. As we have shown above, no reference values in absolute units (e.g. in hours) can be adapted from other facilities. A range of factors need to be considered—number of sorters, number of staff, their contractual hours, holiday entitlement, sickness and other absences, downtimes, operational strategy, etc. However, there are simple mathematical operations which could even be programmed into an Excel spreadsheet and used by any facility in their particular situation to evaluate their service delivery (see Supporting Information). The metrics of service delivery could be adapted as key performance indicators, and thereby become instrumental for optimizing organizational processes required to achieve the goals of the organization , or be used as measures for designing a performance management tool like a balanced scorecard . Moreover, standardized metrics could be useful for designing the process of performance benchmarking in flow cytometry core service [c.f. Ref.].
There are some other variables that may exercise an impact on the metrics derived. These would include scenarios where a new operator is appointed and some in-house training is required. Although the variable t is supposed to include all operating time which cannot be classified as direct provision of sorting service, this training will reduce total sorting hours. If the facility's operational policy does allow end users to operate sorters on their own, some operator time would need to be assigned for training users and troubleshooting their experiments.
The metrics described here would be helpful for cost-recovery purposes. Indeed, a calculation of adequate numbers of sorting hours is pivotal for calculating costs of one hour of operation, and allows one to incorporate the “hidden” costs of running a service. Moreover, if there is downward pressure to keep any fees as low as possible, these metrics can be helpful in calculating realistic amounts of revenue at current capacity and staff/running costs.
A helpful side-effect of such a standardization would be a unified approach with respect to demands imposed on scientific support staff. These days, many institutions simply compare the output of their facility by recording sorting output in hours and dividing it by the number of available staff. As it has been shown here, this approach is likely to be highly unjust and counter-productive, both with respect to researchers who would not be provided with an adequate service due to misconception of the research and infrastructure needs, and with respect to support staff whose performance would be judged on false assumptions. Although the measurement of service or facility management delivery and performance by looking at the correlation between performance and satisfaction by end-users is considered to be an essential step , the measurement of performance of the service suppliers (facility staff) must be based on just and achievable criteria.
By developing equations for the calculation of both equipment and staff sorting capacities, the major aim of this communication has been achieved: to define two metrics of cell sorting service in a standardized way and to establish a mathematical formalism that takes into account multivariate factors in order to monitor the sorting capacity of a resource. The general application of this formalism would provide a unified method for analysis and for potential comparison of sorting facilities in a just and fair way. Moreover, the calculations presented here could be used for inclusion in any cost accounting method (e.g., to justify the practical capacity for sorting or other service as indicated in the paper by Box et al. : his Table 2b, column “Practical capacity per day.”) In addition, a simple flow chart-based algorithm has been developed to help facility managers, technical staff, academic leads, and senior management of institutions to initiate a more complex analysis and to make clear and justifiable decisions with respect to applications for new capital equipment or for additions to or redeployment of facility staff. Obviously, the algorithm described here could be applied to evaluation of any technical resources providing experimental service that needs to be quantified (e.g., for proteomics and imaging facilities) and, potentially, for their performance and process benchmarking.
The author thanks Dr. Robin A.P. Harrison for English revision and proof reading of the manuscript.
1Department of Medicine, University of Cambridge School of Clinical Medicine, Addenbrooke's Hospital, Cambridge CB2 0QQ United Kingdom
2Unit for Reproductive Medicine of Clinics, Clinic for Horses, University of Veterinary Medicine Foundation, Hannover, Germany
- 2Key Performance Indicators. Developing, Implementing, and Using Winning KPIs.Hoboken, NJ: Wiley;2010..
- 4The Search for Industry Best Practices that Lead to Superior Performance.Milwaukee, WI USA: ASQC Quality Press;1989..
- 5The sphere of performance measurement in strategic facilities management. J Facilities Management 2012;10: Editorial..
Additional Supporting Information may be found in the online version of this article.
|cytoa22297-sup-0001-SuppFig1.tif||1536K||Supporting Information Figure 1.|
Please note: Wiley Blackwell is not responsible for the content or functionality of any supporting information supplied by the authors. Any queries (other than missing content) should be directed to the corresponding author for the article.