• Open Access

Multiresponse optimization with consideration of probabilistic covariates



In many complex experiments, nuisance factor may have large effects that must be accounted for. Covariates are one of the most important kinds of nuisance factors that can be measured but cannot be controlled within the experimental runs. In this paper a novel approach is proposed, based on goal programming, to find the best combination of factors so as to optimize multiresponse-multicovariate surfaces with consideration of location and dispersion effects. Furthermore, it is supposed that several covariates considered in the experiment have probability distributions of known form. One objective is to find the most probable values of each covariate. For this purpose, a multiobjective mathematical optimization model is proposed and its efficacy is demonstrated by two numerical examples. Copyright © 2010 John Wiley & Sons, Ltd.

1. Introduction

In the models used in the design and analysis of experiments, several input variables can affect the response, or output variables. If the input variables are known and are controllable factors, their levels can be set so as to optimize the functions of the observed responses. However, some of the input variables may be ‘nuisance’ variables which are not of interest and may not be controllable. Montgomery1 categorizes such variables into three classes:

  • Known and controllable variables whose effects are not of interest as a factor. The statistical technique of blocking can be used to eliminate the effects of these variables on the statistical analysis and thus on model building. Examples are systematic variations caused by shifts of workers or differences between manufacturing lines.

  • Known and uncontrollable variables are often called covariates. As the variables are suspected to be important, their values should be measured during the experiment. Inclusion of the individual effects of covariates and their interactions with other variables can help to improve the modeling of the outputs. The chemical properties of raw materials are an example.

  • Unknown and uncontrollable variables, where the relevance of the factor is unknown, and it may change levels during the experiment. Atmospheric humidity or age of feedstock are examples in this class. Randomization is the design technique used to avoid bias from such nuisance factors.

In the analysis of complex systems there are, almost invariably, several responses. Optimization of multiple quality characteristics (or response variables) is more complicated than optimization of a single one since we face different units, importance, and optimality directions. In this study a multicovariate-multiresponse model has been developed in which the covariates have stochastic patterns and the aim is to find the best arrangement of the input variables including covariates and factors so as to optimize the response variables. This paper is organized as follows. In Sections 2.1 and 2.2, a literature review is presented for both covariate-oriented studies and stochastic multiresponse optimization (MRO). In Section 3, a general model for multicovariate-MRO is proposed and in Section 4, computational descriptions and other properties of the model are illustrated by two numerical examples. Finally, conclusions and key remarks are presented in Section 5.

2. Literature review

In this section some previous work on covariate analysis and multiple responses optimization are reviewed separately.

2.1. Earlier studies on covariate analysis

Most of the research in the field of covariate analysis is directed to the elimination of covariate effects on response variability so as to provide more reliable and accurate estimates of factor effects. A few studies have focused on optimization of the experiment in which responses are affected by covariates. The Analysis of Covariance (ANCOVA) is the primary method for adjusting the observed response variable for the effect of the covariate or concomitant variable. If such an adjustment is not performed, the covariate will inflate the error mean square and make it harder to detect true differences in the response due to treatments. Thus, the ANCOVA is a method of adjusting for the effects of an uncontrollable nuisance variable1. Three major limitations of ANCOVA are:

  • Response is linearly related to covariate. Thus, the interaction between covariates and factors or probable curvature in covariate effect may be disregarded.

  • Covariate values are not different among the groups and covariates are fixed.

  • If more than one covariate is considered, there is no collinearity between covariates.

Rahbar and Gardiner2 proposed a non-iterative method of estimation when the response is linearly related to the covariate. Their method was nonparametric, with a single, integer-valued covariate. Gertsbakh3 considered life testing with factorial experiments and tried to find the best allocation of testing position and time. The response variable was assumed to have an exponential distribution and the parameters were estimated by maximum likelihood. This research includes multiple covariates and factors. Wu4 studied covariate effects in nonlinear mixed-effect models and used maximum likelihood to estimate the regression coefficients. In addition, a computationally efficient method for approximate likelihood inference was proposed to overcome computational difficulties such as the non-convergence caused by the nonlinear functions and high dimensional integrals of complete-data log-likelihood. Royston and Sauerbrei5 proposed a new approach for analyzing covariates with preliminary linear transformation of covariates producing less extreme leverage. Their method is related to fractional polynomial modeling and the proposed transformation incorporated a predefined shift of the origin away from zero. They confirm the reliability and stability of univariate or multivariable fractional polynomial models. Yeo6 proposed a new form of generalized additive model including the distribution function of the mean response and a weighted linear combination of distribution functions of covariates. This form avoids structural problems in linking the mean response and covariates. Markov-chain Monte-Carlo methods were used to estimate the parameters within a Bayesian framework. Das et al.7 considered RBD, BIBD, and CRD designs in which covariates were analyzed. They proposed are optimal structure for each of the aforesaid designs for better and more reliable analysis of covariates.

2.2. Multiresponse optimization

Different aspects of MRO problems have been studied in several areas. We can distinguish three general categories: (a) Desirability functions: in this category, researchers try to aggregate information from each response to obtain one response. Optimization is then performed on a single total desirability function. (b) Priority-based methods: If the responses have differing degrees of importance degrees, we first consider the most important response for optimization. If the solution is not unique, then find the best solution by comparing the status of the other responses for alternative solutions. These steps are repeated until all responses have been considered and a unique optimal solution found. (c) Loss function: In this category, based on the Taguchi loss function, all response values are aggregated and converted into one. Much research has been dedicated to developing and generalizing the Taguchi loss function for many specific cases.

The earlier work in MRO includes Layne8 who presented a procedure that simultaneously considers three functions—the weighted loss function, the desirability function, and a distance function—to determine the optimum parameter combination. Pignatiello9 utilized a variance component and the squared deviation-from-target to form an expected loss function to optimize a multiple response problem. However, this method is difficult to implement. The first reason is that a cost matrix must be initially obtained, and the second reason is that it needs much experimental data. Leon10 presented a method, which is based on the notion of a standardized loss function with specification limits. However, only the nominally-the best (NTB) characteristic is suitable for this approach, which may limit the capability of the method. Ames et al.11 presented a quality loss function approach in the response surface. The basic strategy is to describe the response surfaces with experimentally derived polynomials, which can be combined into a single loss function using known or desired targets; minimizing the loss function with respect to process inputs, locates the best operating conditions. Bashiri and Hejazi12 used Multiple Attribute Decision Making (MADM) methods, such as VIKOR, PROMETHEE II, ELECTRE III, and TOPSIS, to convert multiresponses to a single response in order to analyze robust experimental designs. The main advantage of their method was the consideration of standard deviation that was included in the robust experimental design. Also, because only one response regression function was fitted, the proposed method decreased statistical error. Tong et al.13 used the VIKOR method in converting Taguchi criteria to a single response and then found a regression model and the related optimal setting. Tong et al.14 also considered correlation of responses and used PCA and TOPSIS methods to find the best variable setting. Chiao and Hamada15 considered experiments with correlated multiple responses whose means, variances, and correlations depend on experimental factors. Analysis of these experiments consists of modeling distributional parameters in terms of the experimental factors and finding factor settings that maximize the probability of being in a specification region, i.e. all responses are simultaneously meeting their respective specifications. Kazemzadeh et al.16 proposed a general framework for MRO problems based on goal programming (GP), studied some existing works, and attempted to aggregate all the characteristics into one approach. Shah et al.17 illustrated the seemingly unrelated regression (SUR) method for estimating the regression parameters that may be useful when the response variables are correlated. Their method can lead to more precise estimation of the optimum variable setting. Bashiri et al.18, from the viewpoint of multiple objective decision making, proposed an approach in which Global Criteria (GC) have been applied to aggregate multiresponse surfaces in the simulation of a probabilistic inventory model. Kovach and Cho19 used a robust design and proposed MRO using nonlinear GP. They also used a combined array for incorporating the nuisance factors. Table I gives a summary of this literature together with comparisons of the existing methods and of the proposed procedure.

Table I. Some earlier works and their comparison
MethodsNo. of covariatesCorrelationInteraction effectsStochastic covariateResponse linearityFocus on statistical aspectsFocus on optimization aspectsRobustness
Rahbar andSingle     
Gertsbakh3Multiple Between factors   
Das et al.7Multiple     
Wu4MultipleBetween covariatesBetween factors   
Kazemzadeh Between responsesBetween factors   
et al.16        
Chiao and  Between factors  
Shah et al.17 Between responsesBetween factors   
ProposedMultipleBetweenFor all of the variables  
method covariates(factors and covariates)     

According to Table I, in the majority of publications on the effect of covariates, it is assumed that the relation between the response variables and covariates is linear and also that there is no interaction between factors and covariates. On the contrary, in this paper there is no limitation to the existence of interaction effects between any variables. Moreover, we try to maximize the occurrence probability of stochastic covariates.

3. Proposed approach

We develop an optimization model for multiresponse surfaces with probabilistic covariates in which response variation and covariate probability are considered as separate objective functions. In addition, the proposed approach makes it possible to solve the optimization with correlated or uncorrelated covariates. For this purpose, Principal Component Analysis (PCA) is used to get independent values from the original data. Figure 1 provides a general framework for the proposed approach.

Figure 1.

The proposed approach in schematic view

Some advantages of the proposed approach are listed below to compare with other studies mentioned in the literature review.

  • In this paper, a systematic approach is proposed to optimize multiresponse surfaces with respect to factors and covariates for complex processes.

  • This approach has some capabilities to analyze multiresponse problems with consideration of probabilistic covariates.

  • There is no limitation to the distribution type of covariate probability.

  • There is no limitation to the definition region of covariates (i.e. integer, binary, positive, etc.).

  • The proposed approach can consider correlation between the covariates.

3.1. Multiresponse-multicovariate model

In this section, a multiobjective mathematical model is developed based on the experimental design to optimize the following three sets of objective functions:

  • Location effects: The expected values of the response surfaces are optimized according to the type of response variables (NTB, STB, and LTB responses).

  • Dispersion effects: The estimated variances of the response variables are minimized to ensure robustness in the results.

  • Probability functions of the covariates: Maximization of this objective set increases the reliability of the results. As the covariates are stochastic and uncontrollable, this objective ensures that the covariates will occur with maximum probability.

The constraints of the proposed model are related to specification limits of factors and the definition region of stochastic covariates. According to the aforementioned objectives and constraints the proposed mathematical model (a second-order polynomial) and related parameters are defined below:

Parameters and variables:

  • R: Response variables.

  • β, α, γ, β', α', γ': Regression coefficients.

  • x: Controllable variables.

  • c: Covariates.

  • f(Ci): Probability distribution function of the ith covariate.

  • (L, U): Lower and upper bound of controllable variables.

  • Ω: Feasible region of covariates (according to their probabilistic pattern).

Mathematical model:

equation image(1)
equation image(2)
equation image(3)
equation image(4)
equation image(5)

The first objective set (1) shows the response surfaces to be optimized by settings of the factors and covariates. The second objective set (2) ensures that the dispersion effects are considered and minimized and objective set (3) maximizes the covariates at their modes. Constraint sets (4) and (5) include the limits of the factors and define the region of the covariates. Depending on the number of covariates, there are two sub-procedures for analyzing the covariate effect:

  • Single covariate approach: In this case, a proper function between the real values of input variables (factors and covariates) and output variables (response surfaces for location and dispersion effects), is developed.

  • Multiple covariate approach: In this case, depending on the following criteria, two solution methods are proposed as follows:

    • Case 1: If the correlation between covariates is not statistically meaningful, the proper regression functions between input and output variables can be fitted directly. In this case, the joint mode of the covariates could be used in place of the third objective function set.

    • Case 2: If the covariates are highly correlated, the regression procedure will face a collinearity problem that makes regression analysis difficult. In this paper we use PCA to provide proper information from correlated data. Figure 2 shows the PCA phase (a part of Figure 1) in flowchart view.

Figure 2.

The PCA phase in flowchart view

As shown in Figures 1 and 2, the proposed approach can be summarized in the following steps. The computational details are presented in the numerical examples.

  • Step 1: Identification of the factors, covariates, and responses. In addition, the continuous or discrete probability distributions for the covariates must be specified.

  • Step 2: Depending on the number of factors and the region determined by their limits, an appropriate design is selected for running the experiment.

  • Step 3: Depending on the number of covariates and their interdependencies, the proposed approach should be continued according to the following cases:

    • 3.1 If there is only one covariate, the general model can be applied and f(C) in (3) is defined as follows:

      equation image(6)

      Note that, a is an adjusting parameter that makes it possible to find an interval for calculating the probability value of continuous random variables.

    • 3.2 If there are two or more covariates in the experiments, the general model must be modified according to the following conditions:

      • (a) In this case, the covariates are independent and one modification should be made in the general model in which fT(C) can be defined by the following relations:

        equation image(7)


        equation image

        where f(C) is obtained from (6) for continuous or discrete distribution functions.

      • (b) In this case there is interdependence between the covariates. In such situations the regression analysis might be affected by collinearity20. In this paper PCA is applied to overcome this problem by using linear transformations to convert a vector of correlated variables into those that are uncorrelated. Hotelling21 initially developed PCA to explain the variance–covariance structure of a set of variables by linearly combining the original variables. The PCA technique can account for most of the variations of the original p variables via k uncorrelated principal components, where kp. More formally, if C = c1, c2, …, cp is a set of original variables with a variance–covariance matrix and the set of uncorrelated linear combinations can be obtained as the matrix

        equation image(8)

        where C′ = (c1, c2, …, cp)T, c1 is the first principal component, c2 is the second principal component and so on; A = (aij)P × P and A is an orthogonal matrix with ATA = I. Therefore, C can also be expressed as

        equation image(9)

        where Aj = [a1j, a2j, …, apj]T is the jth eigenvector of equation image. After performing PCA on correlated covariates, fT(C) will change to:

        equation image(10)


        equation image

        Note that C′ is an uncorrelated vector with a linear relationship to C. In some cases, finding the distribution function for C′ is complicated; thus the second formulation of fT(C) in (10) could be used. More details of this step are illustrated in the numerical example.

  • Step 4: In this step, proper regression functions (response surfaces) are fitted between response variables and inputs (factors and covariates).

  • Step 5: Finally, in this step, after constructing the mathematical model, the optimal values of the responses and the related values for factors and covariates are found using the GP approach.

3.2. GP model

As mentioned in Step 5, the aforesaid three objective sets have been aggregated by GP method. In this section the goal function and the related constraints are presented.

Goal function:Min Z = [fµk(dµk, dµk), fσk(dσk, dσk), fpl(dpl, dpl)]

Subject to

For location effects:

equation image

where d and d′ are negative and positive deviations from the target and should be minimized in the goal function.

For dispersion effects: equation image

As the variance cannot be negative, in this relation the target of response variance is set to zero and only positive deviation is allowed and should be minimized in the goal function.

For covariate probability: Probability function(cl) − dpl = Tll = 1, 2, …, m

Tl is a positive target for the occurrence probability of each covariate. Note that this target is a lower bound or minimum acceptable value for the probability, hence the positive deviation from the target should be maximized in the goal function.

For controllable factors: lixiuii = 1, 2, …, n

The upper and lower bounds for each factor is determined in the experimental design depending on the specification limits.

For covariates: cl∈Θ

where Θ is the definition region of the covariates according to the probabilistic distributions or stochastic patterns which are known.

For deviation variables: dk, dk, dl≥0∀k = 1, 2, …, p;∀l = 1, 2, …, m

In the proposed model, summation of the normalized weighted deviations is used to aggregate the objective function so that the final GP model can be expressed as:

equation image

Subject to

equation image

where W is a vector of importance degrees obtained from decision makers and equation image nd equation image are the standard deviation of the mean and variance of kth response variables, respectively, in experimental observations. Thus W/σ and W/T are the vectors of normalized weights used in the final additive objective so as to make it weighted and dimensionless.

In Section 3 we have described the proposed approach and the related mathematical model and given details of the steps. In the following section this approach will be illustrated through two numerical examples and the benefits and efficiencies assessed. Also some comparative results between the proposed method and some existing methods are presented.

4. Numerical examples

In this section, two cases are studied to illustrate the applications of the proposed approach. In the first experiment there are two factors and one normally distributed covariate that might affect two responses. The data are generated from three replicates of a Central Composite Design (CCD). Table II shows more information about these experiments.

Table II. Experimental data for case 1
   Response variables
FactorsCovariatesExpected valueEstimated variance
ABXN(15, 42)equation imageequation imageequation imageequation image
  Desired value366.2500
  Standard deviation5.222.6641.09170.477

As presented in Table II, the two responses (µ1, µ2) represent the location effects of the input variables, whereas the other responses equation image represent the dispersion effects used in dual response modeling. The fitted response surfaces for a simulated data set, obtained using the MINITAB software package, can be written as:

equation image

where X is single covariate and it is known that XN(15, 42). The response variables are of type NTB with desired values equation image. According to the response surfaces and desired values of each response, the mathematical model of the optimization problem can be formulated as

equation image

Subject to

equation image

In this example, the desired probability value of the covariate occurrence is initially set to 0.19 and, later, sensitivity analysis for this parameter and its effects on the results will be performed (this is an adjustable parameter that might be obtained by interaction between the analyst and decision maker). This parameter is a lower bound for the occurrence probability of covariates; greater values of this parameter make the feasible region tighter and the results more reliable. In this example all objective function sets have equal weights, that is wi = 1. Table III shows the optimum results for Case 1 obtained using the LINGO 8 optimization software.

Table III. Optimum solution for case 1 and comparison
Optimum valuesABXequation imageequation imageequation imageequation imagef(c ± α)
Proposed method−0.2414−0.71413.8812366.251.860.6890.19
Kim and Lin230.47660.2912.0533.365.7221.330.790.15
Kovach and Cho190.0153−0.079212.767366.251.50.9250.169

Considering the experimental data in Table II and the optimum values in Table III, it is clear that the proposed method has satisfied all response function constraints, including two location effects (µ1, µ2), two dispersion effects equation image, and one covariate expectation. Table IV shows the results of MANCOVA for the fitted response surfaces and also includes some measures for comparison. MANCOVA and its optimization phase have been performed in MINITAB 15 statistical software.

Table IV. Optimum solution for case 1 using MANCOVA in Minitab 15
Optimum valuesABXequation imageequation imageequation imageequation imagef(c ± α)
MANCOVA (Minitab)10.10622.156331.53616.85352.85571.41090.041
Proposed method-0.2414-0.71413.8812366.251.860.6890.19

For sensitivity analysis of the optimization model, contour plots of the response surfaces as functions of the input variable are shown in Figure 3.

Figure 3.

Contour plots of response surfaces for case 1

In Figure 3, we have set the third variable at its optimum value. In the right section of Figure 3, B is set to −0.714 and in the left section, A is set to 0.2414. It is clear in Figure 3, that if the covariate increases above its optimum value, the feasible region of the model becomes tighter. But small changes in the values of the controllable variables do not lead to unacceptable response values. As mentioned above the proposed method not only maximized the expected value of the covariate, but also satisfied all of the constraints on the response variables including location and dispersion effects.

Previously the target value of covariate probability was set to 0.19. Increasing this value makes the problem infeasible and in order to perform sensitivity analysis on this parameter it has been varied in the reasonable range. Figure 4 shows the effect of this parameter on location and dispersion effects and Figure 4 shows the sensitivity of goal function and covariate probability with respect to this parameter compared to each other.

Figure 4.

Sensitivity analysis of covariate probability and goal function

Figure 4 shows that in this example, increasing the desired value of covariate probability has an inverse relationship with goodness of optimal goal function which should be minimized. In addition, greater values of this parameter could lead to a more reliable result since the optimum covariate value may occur with higher probability. But in this example, if this parameter takes a value greater than 0.19, the model will be infeasible. The effect of changing this parameter on each response variable can be found in Figure 5. This figure shows that this parameter has different effects on equation image and equation image, while two location effects are not sensitive with respect to this parameter.

Figure 5.

Sensitivity analysis of location and dispersion effects

We now consider Case 2, a multicovariate model. As shown in Table V, the second case has two response variables (R1 and R2), three factors (A, B, and C) and two covariates (C1 and C2). The response variables are of type NTB and their desired values are equation image. In addition, the covariates have the probability distributions

equation image

In this case we ran a single replication of the design. As a result, dispersion effects cannot be considered.

Table V. Experimental data for case 2

The correlation between C1 and C2 is 0.865. Owing to the resulting collinearity between the predictor variables, we use PCA to provide a linear transformation to uncorrelated variables. Table VI shows the analysis using MINITAB 15 statistical software.

Table VI. Principal component analysis for two correlated covariates

According to Table VI, the required transformation of the two covariates to make them uncorrelated is:

equation image(11)
equation image(12)

By inverting (11) and (12) the original values of covariates can be obtained from (13) and (14):

equation image(13)
equation image(14)

Using PCA the regression function between inputs and outputs can be fitted more reliably. The two fitted response surfaces are

equation image

In this case all response sets have equal weights of importance (wi = 1). Now the final GP model can be written as follows.

equation image

Subject to

equation image(15)
equation image(16)
equation image(17)
equation image(18)
equation image(19)
equation image(20)

Similar to the first example, the objective of this model is to minimize the summation of standardized weighted deviations of targets. Constraints set (15) calculates the deviation from the target of NTB response variables. The next set is to find the optimum covariates that satisfy the minimum acceptable probability for occurrence which equals 0.05 for both exponential and Poisson covariates. The effect of changing this parameter on the results was elaborated previously in the first example. The linear relation between PCs and covariates are entered in the model by relations (17). The next three constraint sets (sets 20–23) show the limits of the decision variables depending on their definitions.

Table VII gives the optimal values of each variable resulting from optimization of the proposed model. This problem has also been solved using Minitab software24. The Minitab software provides MRO module in which the desirability function method is used to analyze such problems.

Table VII. Optimization results—case 2
Optimal valuesABCPC1PC2C1C2R1R2f(C1)f(C2 ± α)
Proposed methods-0.536-0.671.6555.4187.1990.2405.50.1250.077
Desirability function1.51-1.6818-0.181719.66405.50.0130.0015

As shown in Table VII, R1 and R2 reach their goals and the probability functions of the covariates have proper values. Although the desirability function method has also satisfied two response variables (R1, R2), it is obvious that the probability value of covariates occurrence resulted by the proposed method is considerably better than the desirability method. In other words, if we set factors as (A, B, C) = (1.51, −1.68, −0.18), we cannot expect that the responses will be satisfied unless the covariates are observed as (C1, C2) = (17, 19.66). Therefore, since these values of covariates occur with smaller probability, the factor setting will not be sufficiently reliable.

5. Conclusion

In this paper, a mathematical model has been developed to find the best settings of variables to optimize dual response models when single or multiple stochastic covariates are included in the experiments. In the case of collinearity among the covariates, the capability of the proposed approach has been improved by the use of PCA. As the optimum values of the covariates must be analyzed from a probabilistic viewpoint, optimizing the response variables by ignoring the covariates would be misleading. A second important point is that the proposed model tries to find the best factor levels considering the maximum probable values of each covariate according to their stochastic patterns, that is their probability distributions. Two numerical examples have been analyzed and the main conditions and calculation steps of the proposed approach are described in detail.

Biographical Information

Authors' biographies

Taha Hossein Hejazi received an MSc in Industrial Engineering from Shahed University, Tehran. His primary research interests include quality control and engineering, optimization and design of experiments, computer simulation, metaheuristics for optimization and Multiple Criteria Decision Making. Also he will start PhD programs at Amirkabir University of Technology (Tehran Polytechnic) in the next semester and he is a member of the International Association of Computer Science and Information Technology (IACSIT).

Mahdi Bashiri is an Assistant Professor at Shahed University. He received his BS in Industrial Engineering from Iran University of science and technology in 1999 and his MS and PhD in Industrial Engineering from the Tarbiat Modares University of Tehran in 2001 and 2005, respectively. He visited the statistics department at London School of Economics and political sciences at 2004 for six months. His research interests are Multiple Response Optimization, Design of Experiments and Facility Location and Layout.

Kazem Noghondarian is an Assistant Professor of Industrial Engineering at Iran University of Science and Technology. He received his BSc from University of Neveda and his MSc and PhD from Arizona State University and University of British Columbia (UBC), respectively. His primary research interests include statistical quality control, quality management, design of experiments, data mining and simulation.

Anthony C. Atkinson is Emeritus Professor of Statistics at the London School of Economics and Political Science. He received his MA in Chemical Engineering from Cambridge University in 1960 and his MSc and PhD in Statistics from Imperial College, London in 1965 and 1970. His primary research interests include: clinical trials; generalized linear models; simulation; robust statistical methods; regression diagnostics and optimum experimental design. He was editor of the Journal of the Royal Statistical Society, Series B and an associate editor of Biometrika and Technometrics. Professor Atkinson is a Fellow of the American Statistical Association.