The effect of continuously updating control chart limits on control chart performance

An open topic within statistical process monitoring is the effect on control chart properties of updating the control chart limits during the monitoring period. The challenge is to use the correct data for updating the control limits as in‐control data could be incorrectly classified as out of control and therefore not used for re‐estimating the parameters, and out‐of‐control data could be classified as in control and therefore used for re‐estimating. In the present article, we study the effect of updating the Shewhart, cumulative sum, and exponentially weighted moving average control chart limits. We simulate different scenarios: the monitoring data could be in or out of control, and the practitioner may or may not be able to find out whether the process is indeed out of control when the control chart gives a signal. The results reveal that the variation in the performance of the conditional control charts decreases significantly as a result of updating the control chart limits when the updating data are in control and also when the updating data are out of control and the practitioner is able to classify correctly data samples that produce a signal. However, when a practitioner is not able to classify a signal correctly, the advisability of updating depends on the type of control chart and the level of data contamination.


INTRODUCTION
The control chart proposed by Shewhart, 1 the cumulative sum (CUSUM) control chart proposed by Page, 2 and the exponentially weighted moving average (EWMA) control chart proposed by Roberts 3 are tools that are applied for monitoring quality indicators. The advantage of the CUSUM and EWMA compared with the Shewhart control chart is that they can detect smaller sustained shifts in the process mean earlier. In order to construct the control chart limits, the distributional parameters of the process characteristic have to be estimated. In the literature, it is often assumed that the parameters are estimated only once during the so-called Phase I stage, and that the control chart remains fixed during the monitoring period, a stage which is usually referred to as Phase II (see, eg, Vining 4 ).
A control chart design is usually evaluated by determining the performance in expectation, ie, the unconditional control chart performance, and the distributional properties of performance metrics of the conditional control charts. Jones et al 5,6 and Saleh et al 7 studied the unconditional and conditional performance of the EWMA, CUSUM, and Shewhart control charts, respectively. All conclude that the variation within the performance of the conditional control charts is very large when the parameters are estimated.
Jensen et al 8 surveyed the literature on estimated control charts and identified 13 issues for future research. Issue 6 of Jensen et al 8 suggests reestimating the limits during the monitoring period: "Related to the previous research question is the effect on control chart properties when the control limits are updated in some future time that is not necessarily during a start-up period. If the process is in control, it would be reasonable to use the data to update control limits during Phase II and not continue to use the original limits indefinitely. It is not clear how control chart performance is impacted, but it seems that making use of earlier Phase II data would lead to better control charts." This issue is the subject under study in the present paper. In this context, we should also mention the self-starting CUSUM and Shewhart control chart designs proposed by Hawkins 9 and Quesenberry, 10 respectively. These designs can already be used when just a few samples are available. The performance of these charts was studied by Keefe et al 11 using a simulation procedure. However, they did not study the effect of out-of-control data and reset the parameters to the initial Phase I estimates when the conditional control chart gives a signal.
In the present article, we study the effect of updating for different scenarios: the updating data may be in or out of control, signals may or may not be correctly classified (depending on the scenario), and when the control chart signals the parameters are re-estimated, and the updating continues.
The article is structured as follows. Section 2 gives the relevant control chart designs. Section 3 describes the scenarios that will be included in the simulation procedure, which is described in Section 4. Then, the results during and after updating are given in Sections 5 and 6, respectively, and Section 7 offers conclusions and recommendations.

CONTROL CHART DESIGNS
This section gives the Shewhart, CUSUM, and EWMA control chart designs.
Let X ij denote the jth observation in sample i (i = 1, 2, .... and j = 1, 2, ...n), and let X i denote the vector containing the n observations of sample i. Further, let m I denote the number of Phase I samples for initial parameter estimation and m u the number of samples within the updating period. We assume that the m I samples in Phase I are in control (in this stage the practitioner determines the in-control state). The m u monitoring/updating samples may or may not be out of control. Further, we assume that the observations X ij in the first m I samples are independently and identically N( , 2 ) distributed and that the observations in the next m u samples are independently and identically N( + , 2 ) distributed with probability P, where the values of and P depend on the scenario. Finally, let t be the time stamp during monitoring (t = m I + 1, m I + 2, … , m I + m u ), and let X ic t−1 denote the samples that are classified as in control up to and including time t − 1. The number of samples within X ic t−1 is denoted by m t − 1 (so m I ⩽ m t − 1 ⩽ m I + m u ). For each of the three control charts, and have to be estimated, and we use the same estimators for each chart. The parameter is estimated by with as the indicator function. Further, is estimated bỹ where S i is the ith sample standard deviation defined by The following sections outline the Shewhart, CUSUM, and EWMA control chart designs.

Shewhart control chart
The estimated Shewhart control limits based on the samples in X ic t−1 , used for monitoring at time t, t = m I + 1, m I + 2, … , m I + m u , are given byÛ where X t−1 is given by (1) andS t−1 by (2), while 3 is the traditional constant used for known parameters to ensure that the false alarm probability (average run length) of the chart is equal to 0.0027 (370) * .

CUSUM control chart
As the Crosier CUSUM control chart outperforms the classical CUSUM chart (see Crosier 14 ), we will apply this design. Let C t , t = m I + 1, m I + 2, … , m I + m u , denote the standardized statistic defined as and V m I = 0. The monitoring statistic then becomes where k is the reference value that determines the point at which V t accumulates deviations from the target value X t−1 and is commonly set at k = 1 2 . The chart signals if |V t | > H, where H is the CUSUM control chart limit whose value depends on desired chart performance. To achieve an in-control average run length (ARL) of 370 for k = 1 2 and known parameters, H should be set at 4.3904 (see also Crosier 14 ).

EWMA control chart
The EWMA control chart is an extension of the CUSUM chart, adding weights to the cumulative sum of observations. The EWMA statistic is defined as where 0 < ⩽ 1 and Z m I equals the mean estimate X m I . For = 1, the EWMA control chart is equal to the Shewhart control chart. The EWMA control limits for monitoring the process at time t = m I + 1, m I + 2, … , m I + m u arê where and L determine the in-control performance. We set = 0.1 and L = 2.703 because for these settings, the in-control ARL is 370 when the distributional parameters are known and the chart is able to detect small mean shifts quickly (cf. Lucas and Saccucci 15 ). When Z t falls above (below)ÛCL t−1 (LCL t−1 ), the process is considered out of control.

SIMULATION SCENARIOS
In this section, we describe the various scenarios included in the simulation. For each scenario and control chart type, we simulate 100 000 conditional control charts. The scenarios that we consider are presented in Table 1. The case m u = 0 is used to investigate how control chart performance improves when updating (m u > 0) compared with not updating *The reason why we use the traditional constant is that we want to make a comparison with the CUSUM and EWMA control charts with traditional constants. We have, however, also studied the performance of the updated Shewhart chart with the constants for estimated parameters of Goedhart et al, 12,13 but the results are similar.  In scenarios 1 to 4, the monitoring/updating data are in control, and m I is 5 or 200, and m u is 1995 (for m I = 5) or 1800 (for m I = 200). In scenarios 5 to 8, 9 to 12, and 13 to 16, the monitoring/updating data may be out of control, with varying levels of contamination. The contamination is modeled as follows: each sample has a probability P that the observations are drawn from a N( + , 2 ) distribution. We assume that, when the process is out of control, then the process remains out of control until the control chart gives a signal.
Moreover, we investigate the effect of the incorrect classification of a signal. In scenarios 1, 2, 5, 6, 9, 10, 13, and 14, it is assumed that the operator cannot classify a signal correctly and therefore relies on the control chart. Thus, samples that give no signal are considered as in control and used for re-estimating the control limits, and data that give a signal are classified as out of control and not used for reestimation. In these scenarios, it is thus not possible to trace back the start of the out-of-control situation and filter out previous out-of-control samples. On the other hand, in scenarios 3, 4, 7, 8, 11, 12, 15, and 16, we assume that the operator is able to identify the cause of a signal; so data samples that give a false signal are included, and we assume that in this case, it is possible to retrace the start of the out-of-control situation and exclude previous out-of-control samples from estimation.

SIMULATION PROCEDURE
Below, we describe the simulation procedure.
Step 1: Generate conditional control chart In this step, we construct the initial conditional control chart based on the Phase I dataset of m I (5, 200) samples of size 5. We estimate and with the estimators X t−1 andS t−1 (t − 1 = m I ) given by (1) and (2), respectively, and determine the Shewhart or EWMA control limits using (3) or (7), respectively. The limit for the CUSUM control chart is 4.3904.
Step 2: Use conditional control chart for monitoring and update the chart For each t = m I + 1, m I + 2, … , m I + m u : for scenarios 1 to 4, draw a sample X t from N( , 2 ), and for scenarios 5 to 16, when the process is in control, draw a sample with probability 1 − P from N( , 2 ) and otherwise from N( + , 2 ), with the values of P and depending on the scenario (see Table 1). Calculate the test statistics (X t , V t , or Z t ).
When the test statistic falls betweenÛCL t−1 andLCL t−1 , the dataset of classified in-control samples, X ic t−1 , is augmented with X t and denoted by X ic t . The process parameters are recalculated using (1) and (2), and the control limitŝ UCL t andLCL t are determined with the new parameter estimates. When the test statistic falls outside the control limitsÛCL t−1 andLCL t−1 , the next step depends on the scenario. For scenarios 1, 2, 5, 6, 9, 10, 13, and 14 (the reason-unknown scenarios) the current data sample X t is considered out of control and therefore not added to the dataset of classified in-control samples. The limits are not recalculated. Thus, For the CUSUM and EWMA control charts, we set V t = 0 and Z t = X m I .
In contrast, when the test statistic falls outside the control limits, but the process is in control, and the operator is able to determine the cause of the signal-as is the case in scenarios 3,4,7,8,11,12,15, and 16-X t will be used for updating the control chart. The dataset of classified in-control samples, X ic t−1 , is augmented with X t and is denoted by X ic t . The process parameters are recalculated using (1) and (2), and the control limitsÛCL t andLCL t are determined with the new parameter estimates.
When the test statistic falls outside the control limits, the process is out of control, and the operator is able to determine the cause of the signal-as is the case in scenarios 3, 4, 7, 8, 11, 12, 15, and 16-X t will not be used for updating the control chart. Moreover, previous real out-of-control samples, the number denoted by N with N ⩽ m u , are excluded from For the CUSUM and EWMA control charts, we set V t = 0 and Z t = X t−N .
To assess the performance during updating, we determine for each conditional control chart the true alarm percentage (CTAP) and the false alarm percentage (CFAP) within each simulation run. Related measures were presented by Fraker et al, 16 Chakraborti et al, 17 and Frisén. 18 To this end, for 100 000 simulation runs, we count the number of correct signals, out-of-control samples, false alarms, and in-control samples.
Step 3: Assess the conditional control chart performance To assess the performance of a conditional control chart during updating, we determine the CTAP and CFAP as follows: #correct signals #out-of-control samples , and CFAP = #false signals #in-control samples .
To assess the conditional performance of a conditional control chart after updating, we determine the conditional false alarm probability (CFAR) and the conditional average run length (CARL). For the Shewhart chart, these values can be obtained by and For the CUSUM and EWMA control charts, CFAR is assessed by determining the number of false alarms on an interval of 100 000 samples, and CARL is determined by the Markov chain approaches given by Hany and Mahmoud 19 and Saleh et al. 20 Step 4: Assess the overall control chart performance To assess the unconditional control chart performance during updating, we determine the average true alarm percentage (ATAP) and the average false alarm percentage (AFAP) by averaging the CTAP and CFAP values for the R simulation runs.
The expected average run length after updating (AARL) and the expected false alarm rate after updating (AFAR) are determined by averaging the corresponding conditional values obtained in the R simulation runs. Moreover, we determine the 10th and 90th percentiles of the CARL and CFAR values, which are indicated by CARL 10 , CARL 90 , CFAR 10 , and CFAR 90 .
The next two sections present the performance results during and after updating.

PERFORMANCE DURING UPDATING
In this section, we consider the chart performance during updating for either m u = 1995 or m u = 1800 updates. The ATAP and AFAP values for the three charts are reported in Table 2.

Shewhart
The in-control behavior of scenarios 1 to 4 for the Shewhart chart is as expected. For the small mean deviations ( = 0.5) of scenarios 5 to 8, in the reason-unknown scenarios (5 to 6), the ATAP values are smaller than the known parameter detection probability (0.03), and the AFAP values are larger as well. In the reason-known scenarios (7 to 8), the ATAP and AFAP values are very close to the known detection probabilities for the Shewhart chart. For larger deviations = 2, 3 in scenarios 9 to 16, the detection percentages of Table 2 show almost perfect detection, as expected for the Shewhart chart when large mean shifts occur.

CUSUM and EWMA
As the results for the CUSUM and EWMA are comparable, we will consider them together in this section. The in-control AFAP performance is around 0.0027. As expected, the CUSUM and EWMA charts are more capable of detecting small mean shifts ( = 0.5) in scenarios 5 to 8 than the Shewhart chart. For = 2, the ATAP values for both charts are around   0.5 to 0.6, detecting more than half of the out-of-control observations. Furthermore, the AFAP values for both charts in Table 2 are acceptably close to 0.0027 in scenarios 9 to 12. However, in scenarios 13 to 16, for large and frequent deviations ( = 3, P = 0.1), the ATAP values indicate that the CUSUM and EWMA charts still miss a significant portion of the out-of-control observations. The CUSUM (EWMA) chart detects during updating only about 90% (70% to 82%) of the out-of-control observations.
The ATAP values are affected by the (contaminated) parameter estimates. Consider the mean estimates during updating for scenario 13 in Figure 1 for the three charts. For the first 100 updates, all three charts show very inaccurate estimation of the mean because of (mostly) the effects of an inaccurate Phase I mean estimate. Over time, the Shewhart chart improves to a very accurate mean estimate of around 0 as it perfectly detects all out-of-control observations. The CUSUM and EWMA chart converge to higher estimates of the mean, as because of their lower detection probability, they include contaminated samples in the mean estimate.

PERFORMANCE AFTER UPDATING
This section presents the performance results of the control charts after updating. The tables that correspond to this section are Tables 3 (Shewhart), 4 (CUSUM), and 5 (EWMA).

Shewhart
As we can see in Table 3, updating the chart after a limited initial Phase I dataset (scenarios 1 to 4 and 7 to 16) can result in a control chart performance that is similar to directly estimating limits from a very large Phase I dataset (reference scenario 3). More specifically, for these scenarios, a chart estimated on m I = 5 or m I = 200 samples, which is then used for monitoring and updating during m u = 1995 or m u = 1800 samples, respectively, results in a chart that has the same performance as charts estimated directly from a large Phase I dataset (m I = 2000 and m u = 0). This means that in these scenarios, monitoring can start quickly on a limited dataset. It should also be noted that a chart that is initially estimated on a small dataset (m I = 5), followed by monitoring and updating, leads to a similar performance as of a chart for which a larger Phase I dataset is used (m I = 200) before the monitoring and updating starts.
We can also conclude that, for a more serious level of contamination (scenarios 9 to 16), it does not matter whether the practitioner classifies signals correctly (scenarios 9, 10,13,14) or incorrectly (scenarios 11,12,15,16). The reason is that these out-of-control conditions are quickly detected so that there are fewer out-of-control samples in the estimation dataset. Moreover, excluding from the estimation dataset any samples that give a false alarm does not result in a significantly worse performance, as we have also seen in scenarios 1 to 4. When there is a little contamination (scenarios 5 to 8), performance depends on the operator's ability to classify out-of-control signals correctly. When it is possible to determine the cause of the signal and trace back the start of the out-of-control situation, the resulting control chart performs well. On the other hand, when it is not possible to identify the cause and retrace the start of the signal, then many out-of-control samples will be left in the estimation sample (because smaller out-of-control levels are not detected quickly by the Shewhart control chart), resulting in less representative final control limits and worse performance.

CUSUM and EWMA
As the conclusions for the CUSUM (Table 4) and EWMA (Table 5) control charts are very similar, we describe them together.
As with the Shewhart control chart, we can see that depending on the scenario, updating generates the same performance as direct estimation from a large Phase I dataset (reference scenario 3: m I = 2000, m u = 0). Mainly, in scenarios where the data are in control (scenarios 1 to 4) or where the operator is able to identify the cause of a signal and trace back the start (scenarios 7, 8, 11, 12, 15, and 16), control chart performance is similar to the performance of charts estimated on a large Phase I dataset.
When the operator is not able to identify the cause of a signal (and filter out previous out-of-control data samples), the performance of the updated control chart depends on the contamination level. Mainly, for major contamination (scenarios 13 and 14), out-of-control signals will not be detected quickly so that, until the chart gives a signal, many of the out-of-control samples are included in the estimation dataset, affecting the control chart limits.

CONCLUSION
In the present article, we have simulated 16 scenarios, differing in the size of Phase I datasets (m I = 5, 200), in the ability to determine the cause of a signal, and in the status (in or out of control) of the updating datasets. The charts' performances have been analyzed both during and after (at t = 2000) the updating period.
The results show improved chart behavior for updated limits when the updating dataset is in control, even when the size of the initial dataset is very small (m I = 5). This holds for all three charts and means that excluding samples that give a false alarm from the estimation dataset does not affect control chart performance for these values of m I and m u .
For a low level of contamination (eg, = 0.5), the limits can be updated safely for all three charts as long as the signal reason is known and the out-of-control data can be removed. When the signal reason is unknown, so that the origin of the out-of-control situation can not be retraced, the results show a decline in performance for all three charts. The Shewhart chart is especially vulnerable, as it has the least ability to quickly detect small shifts in the process mean.
For higher contamination levels (eg, = 2, 3), it is safe to update the Shewhart chart control limits even if the signal reason is unknown. This is due to the chart's ability to detect all out-of-control signals, preventing that contaminated samples are included in the data set. For the CUSUM and EWMA charts, updating is only safe if the signal reason is known and the origin can be retraced. If this is not the case, the data and evolving statistic will get contaminated, and the performance of the CUSUM and EWMA charts is quite poor.
In summary, we definitely recommend updating control limits for the Shewhart, EWMA, and CUSUM charts as long as the reason for out-of-control signals is known and the origin can be retraced. If this is not the case, the best strategy depends on the size of the expected mean deviation. For large deviations, the Shewhart chart is safe to use, but the EWMA and CUSUM charts are not. For smaller deviations, the Shewhart chart fails, and performance of the CUSUM and EWMA charts is better. We suggest further research on the behavior of updating the limits for various subgroup sample sizes, as well as on performance for varying distributional assumptions.