Cooperative mobile edge computing-cloud computing in Internet of vehicle: Architecture and energy-efficient workload allocation

With the increasing number of vehicles, the generating vehicular data exceeds the capacity of mobile edge computing (MEC). Therefore, studying the interaction and collaboration of edge computing and cloud computing is of significance to provide vehicular users with low-latency high-rate services. This paper first proposes a MEC-cloud computing collaboration architecture for Internet of vehicles, then designs the interconnection/interaction framework between MEC and cloud computing. We consider reducing computation delay and power consumption, and formulate an energy-efficient workload allocation problem with load balancing and dynamic voltage frequency scaling technology, to obtain the optimal workload allocations of MEC and cloud computing. We then present the overall distribution optimization algorithm to solve this problem. The simulation and numerical results show that by saving communication bandwidth and reducing transmission delay, MEC significantly enhances the performance of cloud computing. Besides, the proposed workload balance scheme is better than the benchmark schemes in terms of power consumption and latency.


INTRODUCTION
The intelligent transportation system (ITS) is an important part of smart cities, and the Internet of vehicles (IoV) is able to reduce traffic accidents, alleviate traffic congestion and provide various real-time convenient information, such as navigation, multimedia entertainment, and remote fault diagnosis, 1 thus IoV plays an important role in ITS. However, with the rapid development of IoV applications, mobile devices, and the Internet of things (IoT), mobile users have become increasingly stringent in terms of quality of service (QoS). Providing users with low-latency and high-reliability vehicular services has become a research hotspot in the academia and industry.
To alleviate the computing burden of vehicles with limited resources, the road side unit (RSU) can collect and upload computing tasks to a central cloud server for cloud computing, and sends the computation results back to local users. For example, in some navigation applications, the cloud server grasps the congestion states of urban roads based on the GPS data dynamically uploaded by vehicles, then it plans the optimal driving route for vehicle users. However, IoV is a complex heterogeneous network consisting of vehicular ad hoc networks, IoT, roadside networks, and cellular mobile networks. Exploiting the cloud computing framework for big data management in IoV still exits some disadvantages like high latency, low efficiency and poor scalability. The root cause of these problems is the centralized computing and storage model of cloud services. Therefore, researchers have proposed some improvements (such as mobile cloud 2 and cloudlet 3 ) to promote the localization of cloud services while maintaining the dominant position of the cloud server. In addition, with the emergence of similar concepts such as mobile cloud computing, micro-cloud, and fog computing, mobile edge computing (MEC), as an extension of cloud computing in the edge network, is able to overcome the high latency obstacles when traditional cloud computing provides services.
By extending cloud computing to the edge network, MEC can provide big data computing services more economically and effectively, and it has become a hot topic today. 4,5 For example, Ahmed et al 6 emphasized that MEC provides high-rate and low-latency services, which is a beneficial computing platform for many IoT scenarios and big data services. Tran, 7 Taleb, 8 and Mao 9 reviewed literatures on MEC in recent years, they discussed a number of MEC application scenarios, such as vehicular networks, the smart grid, medical electronics, telecommunication cellular networks, etc. They also discussed the current security and privacy issues faced by MEC. Applying MEC to IoV to provide vehicular services with low-latency and highly-reliability has received widespread attention. For example, Elhajj et al 10 based on the concept of MEC designed a solution to reduce the user-perceived latency in IoT scenarios, and suggested to treat the computing resources of vehicles as virtualized miniature data centers for data sharing and local services providing. Yu et al 11 illustrated the concept of the roadside cloud, which is composed of local servers and RSUs, to process traffic data and provide edge computing services. In addition, MEC is also used in IoV in other aspects. Zhang et al 12 proposed the vehicular edge computing (VEC) architecture to simulate the computation offloading process of mobile vehicles. Deng et al 13 applied software defined network (SDN) to design routing protocols for vehicular networks, where the SDN controller issues routing control information based on the global network state information, and the MEC server controls the local network. Finally, Zhou et al 14 suggested that pushing the AI frontiers to MEC, is a promising solution to fully unleash the potential of the edge big data, and they reviewed the emerging efforts on AI model toward training/inference at the network edge.
On the other hand, technological advances in Internet data center (IDC) design have brought many new opportunities for power management of computing servers. At the server level, chip multiprocessing, dynamic voltage and frequency scaling (DVFS), sleep scheduling mechanisms, and virtual machine management have emerged and been gradually mature. At the data center level, efficient load balancing and task scheduling schemes can further improve the efficiency of power supplies. 15,16 These technologies provide technical support for complex network and physical interactions between the MEC layer and the cloud layer. Moreover, SDN as an emerging software-based network architecture and technology can integrate logical architecture and physical architecture by coding to maximize the resources utilization. For example, by aggregating similar services and sleep scheduling (operational/hibernating) computing servers while meeting latency constraints, the energy consumption can be saved by balancing the workload of computing servers and communication networks, when the system workload increases. This paper emphasizes the collaboration between cloud servers and MEC servers, and proposes a MEC-cloud computing collaboration architecture to support various IoV applications and users. Different from the central control of cloud computing, we apply a centralized SDN controller to monitor the entire network state, and support the interaction and collaboration between cloud servers and MEC servers. Then based on the proposed architecture, we design a hierarchical framework of interconnection/interaction between MEC and cloud computing to study the system power consumption and delay. The workload allocation problem with latency constraints is formulated and the objective function aims to minimize the system power consumption. We decompose the primary problem (PP) into subproblems (SP) corresponding to three subsystems (edge computing subsystem, cloud computing subsystem, WAN communication subsystem), use the existing optimization techniques to solve the SPs respectively, design an iterative algorithm to obtain the optimal solution of PP, and an energy-efficient workload allocation scheme is obtained. This scheme can achieve workload balancing between MEC and cloud servers, and dynamically control the power and working state (operational/hibernating) of the computing server based on DVFS technology, while meeting the certain latency requirements. Simulation and numerical results prove that in the MEC-cloud computing collaboration system, MEC is able to reduce latency, save communication bandwidth, and further enhance cloud computing performance. Moreover, the workload allocation scheme proposed in this paper can effectively reduce system power consumption and latency compared to other approaches.
The remainder of this paper is organized as follows. Section 2 presents the cooperative MEC-cloud computing architecture. In Section 3, we discuss the workload allocation model, describe the optimization problem and develop a distributed optimization algorithm. Section 4 provides simulation results to verify the advantages of the proposed method. Finally, conclusion is given in Section 5.

PROPOSED ARCHITECTURE
The development of IoTs, the increase in the users' demands and the advance of cloud computing technology have promoted the cooperation between MEC and cloud computing. Inspired by the hierarchical architecture proposed by some classic literatures, this paper proposes the MEC-cloud computing collaboration architecture as shown in Figure 1. For example, literature 17 introduced the concept of autonomous vehicle clouds as "vehicle clusters" consisting of highly autonomous vehicles form a vehicular cloud. Its computing, sensing, communication, and physical resources are coordinated and dynamically assigned to authorized users. Literature 17 defined the taxonomy for multiple clouds and came up with the concept of "grid" for vehicular services. For example, the hierarchical Vehi-Cloud architecture includes the IoV physical infrastructure layer and the V2V layer, in which each layer can access the cloud computing resources; the ITS-cloud is composed of nonscalable cloud layer, communication layer, and end-user layer, which aims to optimize the security and reliability of applications. Literature 18 extending the concept of layering to cloud computing, designed a layered and distributed framework of cloud computing to provide user-oriented services. Literature 11 proposed a novel layered vehicular cloud architecture, including central cloud layer, roadside cloud layer, and vehicular cloud layer. It further introduced the benefits of the layered cloud architecture in managing vehicle traffic and optimizing IoV resources. The MEC-cloud computing collaboration architecture as shown in Figure 1 is a dynamic and open-layered architecture, in which vehicular users can dynamically access cloud layer and/or edge layer for computing, storage and other services.
With the continuous development of cloud computing and edge computing, this architecture combining their advantages can provide high-quality services for the entire network system in a coordinated and effective manner. Cloud computing (at the top of Figure 1) equipped with powerful computing capacity, has developed rapidly in recent years and has been widely used in education, communication, medical and other fields. However, the inherent drawbacks of cloud computing, such as high latency and network overloads, etc., hinder the deployment of new IoT services and promote the emergence of cloud-based edge computing (at the middle layer of Figure 1). Edge computing is scalable and geographically closed to end users. It can be used to support real-time applications, save communication bandwidth, reduce the workload of cloud data center and improve resource utilization. In order to adapt to future application scenarios, it is necessary to combine cloud computing with MEC technology and upgrade traditional wireless base stations to intelligent base stations. Therefore, we propose the hierarchical MEC-cloud computing collaboration architecture including the cloud layer, MEC layer, and end-users layer. Applications running on the end users can dynamically select cloud layer or MEC layer or both to provide services according to QoS requirements.
In particular, the MEC-cloud computing collaboration architecture can efficiently handle IoV big data and provide services such as mobility support, service customization, cloud resource selection, distributed computing/storage, etc. Finally, the layered architecture utilizes the cloud and MEC resources in a coordinated manner assisted by SDN technology. This architecture not only can create new services, but also improve system performance, such as execution time, parallel execution, edge processing, MEC security, resource utilization, and energy efficiency.

System framework
The system framework of the interaction between MEC and cloud computing is shown in the left part of Figure 2. The RSU acts as a user interaction interface for receiving vehicular requests. RSUs then input these requests to the connected ). There is a group of cloud servers in the cloud computing subsystem. Each cloud server is equipped with several computers of the same type, and the vehicular requests are partially offloaded from each MEC server to the cloud server through the WAN. Due to the large coverage of the WAN and the huge amount of traffic data communicated through the WAN, the communication delay and bandwidth constraints in the WAN should not be ignored. In the rest of the paper, the power consumption and computing/communication delay are mainly considered in the three subsystems, namely the edge computing subsystem, the WAN communication subsystem and the cloud computing subsystem. We list some significant symbols and their meanings in Table 1. In addition, mathematical notations like the all-ones vector 1 and the all-zeros vector 0 are also used in the following.

MEC subsystem
Power consumption: For simplicity, we can model the power consumption of MEC server i as a function of the computation workload x i , such as quadratic functions and piecewise linear functions. 19 Without loss of generality, we use the following monotonically increasing and convex function to denote the power consumption P MEC where a i > 0 and b i ,c i ≥ 0 are predetermined parameters.
Computation delay: With request arrival rate x i and service rate v i , the MEC server i is considered as a queuing system and we define its computation delay D MEC

Cloud computing subsystem
Power consumption: We assume that every cloud server is equipped with a set of homogeneous computers and each computer at the same cloud server has the same configurations (eg, CPU frequency) and power consumption profile. The power consumption of each computer at cloud server j is approximated as a function of the computer CPU frequency f j ∶ A j f e j + B j , where A j and B j are positive constants, and e varies from 2.5 to 3. 20 In the cloud computing subsystem, the working state of the cloud server changes with the workload. 21 The operational/hibernating state of cloud server j is denoted by a binary variable O j (ie, 1 denotes the server is operational and 0 is hibernating). In addition, the quantity of operational computers at the cloud server j is denoted by an integer variable q j . Therefore, the power consumption P cloud j of cloud server j is given as 22 Computation delay: Characterized by the Erlang-C (or M/M/n queuing) model, each cloud server's computation delay is [(C(n, ∕ ))∕(n − ) + (1∕ )], where n denotes the quantity of operational computers, and , respectively, denote the request arrival rate and service rate, and C(n, ∕ ) denotes the Erlang's C formula [ 23,Ch. 2]. Assume that j is the service rate of each computer at cloud server j. We can usually do the conversion: With the operational/hibernating state O j , the quantity of operational computers q j and the traffic arrival rate y j , the computation delay D cloud j of cloud server j is given as

WAN subsystem
Communication delay: The transmission delay of the WAN path from MEC server i to cloud server j is denoted by d ij . 24 Therefore, when the MEC server i dispatches request rate ij to the cloud server j, the communication delay in WAN D comm ij is defined as

Power consumption minimized problem formulation
The power consumption of the MEC-cloud computing system is mainly generated by all MEC servers and computers equipped at cloud servers. The system power consumption function as the objective function is defined as Constraints include the workload balance constraints, MEC server constraints, cloud server constraints and WAN communication bandwidth constraint that are discussed in the following. The decision variables include the workload x i of MEC server i, the workload y j of cloud server j, MEC server i dispatching the request rate ij to cloud server j, the computer CPU frequency f j and the operational/hibernating state of the cloud server j. While minimizing the system power consumption, it is essential to guarantee the delay requirements, including computation delay and communication delay. The system delay function is then defined as The PP of this paper is to minimize the power consumption of the system while ensuring the latency requirements: Workload blance constraints MEC server constraints Cloud server constraints

WAN communication bandwidth constraint
To design an energy-efficient workload allocation scheme in the MEC-cloud computing system, we make the tradeoff between the power consumption and the vehicular users experienced delay, which will be carefully discussed in the subsystems.

Constraints
Constraints of workload balance: L is denoted as the sum of the request arrival rate at LAN, l i is denoted as the request rate distributed by RSUs to MEC server i. The relationship between L and l i satisfies L ≜ ∑ N i=1 l i . Vehicular requests are processed by the MEC server or the cloud server. X and Y are, respectively, denoted as the request rate assigned to the MEC subsystem and the cloud computing subsystem, x i and y j are, respectively, denoted as the workload of MEC server i and the workload of cloud server j. Their relationship satisfy X ≜ Moreover, the workload balance constraints of servers are a. workload balance constraint of MEC servers b. workload balance constraint of cloud servers Constraints of MEC servers: The computing capacity of MEC server i is limited, thus the workload x i of MEC server i should be less than its maximum computing capacity x max i , while not exceed the request arrival ratel i : Constraints of cloud servers: Cloud server j constraints include a. the basic constraint where f min j and f max j are the minimum and maximum setting of the computer CPU frequency for cloud server j, respectively. c. the quantity of computers where q max j denotes the upper bound of the quantity of computers at cloud server j. d. working state which indicates the operational/hibernating working state of cloud server j. O j = 1 indicates that the cloud server j is operational; while O j = 0 indicates that the cloud server j is hibernating and the quantity of operational computers equals 0.

Constraint of WAN communication bandwidth:
The request rate dispatched from MEC server i to cloud server j is ij . Assume that these transmission paths do not interfere with each other, and the bandwidth used by each transmission path is limited. Therefore, the constraint of WAN communication bandwidth is

Decomposition and solution
Note that in PP, the decision variables come from different subsystems and are tightly coupled with each other, which makes the relationship between the workload allocation and the power consumption-delay tradeoff not clear. Moreover, the decision variables contain integer variables and linear variables, which makes PP general nonconvex and NP-hard.
To address this issue effectively, we develop an approximate approach to decompose PP into three SPs corresponding to the edge computing subsystem, cloud computing subsystem, and WAN subsystem, which can be, respectively, solved via existing optimization techniques. The decomposition and the interaction of each SP/subsystem are illustrated in the right part of Figure 2.

Energy-efficient workload allocation in edge computing subsystem
SP1 is obtained by minimizing the weighted sum of power consumption and computation completion time in the edge computing subsystem: where the adjustable parameters e i , t i ∈ [0, 1] denote the weights of power consumption and computation completion time for the MEC server i, respectively.
Given the workload X assigned to the edge computing subsystem, SP1 is a convex optimization problem with linear constraints, and the interior point method (References 25-27, chapter 11) is used to solve this problem in Appendix A1.
After obtaining the optimal workload allocation x * i for MEC server i, the power consumption and computation delay of the edge computing subsystem can be calculated, respectively:

Energy-efficient workload allocation in cloud computing subsystem
The service response delay of cloud server j should be less than a certain threshold according to the service level agreement, when it processes some delay-sensitive requests, that is D cloud j ≤ D cloud,th j . SP2 is obtained by minimizing the power consumption and ensuring the delay constraint of the cloud computing subsystem: SP2 is a mixed-integer nonlinear programming (MINLP) problem, which is generally difficult to tackle. Since generalized Benders' decomposition is an effective method to solve this problem with guaranteed optimality, given the workload Y of the cloud computing subsystem, we use generalized Benders' decomposition (GBD) (References 28-30, chapter 13) method to solve SP2 in Appendix B1. After obtaining the optimal workload y * j of cloud server j and its optimal CPU frequency f * j , the quantity of operational computers q * j and the working sate O * j , the power consumption and computation delay of the cloud computing subsystem can be calculated separately:

Delay minimization for dispatching in WAN subsystem
SP3 is obtained by minimizing the communication delay in the WAN subsystem to determine the optimal request rate ij dispatched by MEC server i to cloud server j: Given the workload assigned to edge computing subsystem X and cloud computing subsystem Y , the optimal workload x * i for the MEC server i and y * j for cloud server j are determined from (17) and (19), respectively. Therefore, SP3 is regarded as a task assignment problem. To get the optimal combination of SP3 in polynomial time using Hungarian method, 31 the detailed Hungarian algorithm steps are illustrated in Appendix C1. After obtaining the optimal request rate * ij dispatched by MEC server i to cloud server j, the communication delay in the WAN subsystem can be calculated:

Energy-efficient workload allocation in MEC-cloud computing system
The system power consumption comes from MEC servers and cloud servers. Therefore, the function of the system power consumption is defined as The latency of the system includes the MEC server computation delay, the cloud server computation delay and the WAN communication delay. Therefore, the function of the system latency is defined as After solving the above three SPs, we can approximately solve PP by considering the following approximate problem named the PP-approx: Combining the algorithms of the above three SPs, we design an iterative algorithm to solve this problem in Appendix D1. The approximation ratio is dependent on the choice of the adjustable parameters e i , t i and D cloud,th j . If these parameters could be chosen appropriately, then the solution to PP-approx would be the optimal solution to PP. How to evaluate the approximation ratio of the proposed decomposition is left as our future work.

NUMERICAL RESULTS
For simplicity but without loss of generality, in the simulation, we simulate a MEC-cloud computing system with five MEC servers and three cloud servers (regarded as an IDC). The similar results can be obtained if the system includes more MEC servers and more cloud servers. Referring to literatures, 20,23 let e = 3, K = 1, and some important parameter values are shown in Tables 2 and 3. The MATLAB simulation platform is used to obtain the following results. For comparison, we take the following two most related schemes as benchmarks.
• Optimal allocation without DFS: This scheme corresponds to solving the SPs under fixed f j . The resulting optimization problem can be solved by using similar method in this paper. Here, we set f j = 0.6f max j .
• Average allocation: In this scheme, the total request arrival rate at LAN is averagely allocated to the edge computing subsystem and the cloud system, that is, X = Y = L 2 .
We first evaluate the convergence of the proposed iterative algorithm and the impact of the workload of LAN on the system power consumption. We plot the convergence of workload L = 6 × 10 4 ,8 × 10 4 ,10 × 10 4 by the proposed iterative algorithm for solving PP and the dynamics of their power consumption in Figure 3. We can observe that (i) the proposed iterative algorithm can achieve converge about 100 to 120 iterations for all workloads. An iteration takes 0.3 to 0.5ms, therefore, our iterative algorithm takes at most 60 ms to finish the heaviest workload; (ii) the heavier workload has the lower speed of convergence, that is, the workload of LAN can increase the number of iterations.
To evaluate the power consumption P MEC (X) and computation delay D MEC (X) of the edge computing subsystem, we change the workload X of MEC subsystem within 0to10 4 . The algorithm in Appendix A is used to solve SP1 and determine the optimal workload allocated for the MEC server i, then we calculate P MEC (X) and D MEC (X). The results are shown in Figure 4A. It can be seen from Figure 4A that both the power consumption and computation delay of the edge computing subsystem grow with the workload assigned for edge computing subsystem.
We then change the workload Y of cloud computing subsystem within 10 4 to10 5 , and solve SP2 using the algorithm in Appendix B to determine the optimal workload y * j allocated for the cloud server j and working state f * j , m * j , O * j . Then, we calculate the power consumption of cloud computing subsystem P cloud (Y ) and the computation delay D MEC (Y ) respectively, the results are shown in Figure 4B. It can be seen from Figure 4B that the power consumption of the cloud computing subsystem increases with the workload assigned for cloud computing subsystem, and the computation delay keeps equal to the delay constraint of cloud computing subsystem.
With those determined variables, we solve SP3 using the algorithm in Appendix C to obtain the workload dispatching delay in the WAN subsystem D comm (X,Y ) and calculate the power consumption P sys (X,Y ) and the latency D sys (X,Y ) of the system, respectively. The results are shown in Figure 4C. By comparing the curves in Figure 4, we note that when the workload of edge computing subsystem becomes heavy, the power consumption of MEC servers increases and the power consumption of the system also increase, while the workload of cloud computing subsystem, the communication delay of WAN subsystem and the system latency also decrease, and vice versa.
Therefore, we can make the conclusion that the system power consumption is mainly generated by the edge computing subsystem, while the system latency is mainly generated by the WAN subsystem. The reason is that, in the MEC-cloud computing system, the computing capacity of cloud servers is more abundant and more energy-efficient compared with MEC. However, MEC has the advantage of being geographically closed to end users that enables MEC to

F I G U R E 4
Energy-efficient workload allocation in mobile edge computing-cloud computing system. A, Edge computing subsystem; B, cloud computing subsystem; C, MEC-cloud computing subsystem

F I G U R E 5 Comparison of
the workload allocation scheme in the mobile edge computing-cloud computing system reduce communication delay and process some computing requests without consuming WAN bandwidth. The conclusions of the simulations in this section are of great significance for designing efficient computing offloading and resource (such as communication and computing resources) management schemes in the MEC-cloud computing system. Based on the above conclusions, we design a bipartite iterative algorithm in Appendix D1to obtain the optimal workload allocation scheme of the system, then, calculate the power consumption and the latency of the system. As shown in Figure 5, the workload allocation scheme proposed in this paper is significantly better than the other two schemes. First, our scheme can dynamically adjust the workload of edge computing subsystem and cloud computing subsystem based on their computing capacity. Then, our scheme applies DVFS technology, each cloud server can adjust its CPU frequency according to its workload, which effectively saves power consumption and reduces system latency. Finally, our scheme can control the operational/hibernation working state of the cloud server and the quantity of operational computers based on the workload of cloud computing subsystem, that further reduces system power consumption.

CONCLUSION
This paper has introduced the architecture and vision of collaborative MEC-cloud computing in IoV, and designed a layered framework for interactions of MEC and cloud computing, to study the power consumption and latency of the system. Based on workload balancing and dynamic power control, we propose an energy-efficiency optimized workload allocation scheme. The PP is decomposed into three SPs corresponding to three subsystems, which can be solved within, respectively. Simulation results show that the cooperation of MEC and cloud computing can improve the performance of the system, and the workload allocation scheme proposed in this paper effectively reduces system power consumption and latency. In the future work, we will consider the computing capacity of vehicles and study the collaboration among local computing, edge computing, and cloud computing to minimize the computing latency in IoV.

CONFLICT OF INTEREST
No conflict of interest exits in the submission of this manuscript, and manuscript is approved by all authors for publication. We would like to declare that the work described was original research that has not been published previously, and not under consideration for publication elsewhere, in whole or in part. All the authors listed have approved the manuscript that is enclosed.

ORCID
Thus, SP1 can be denoted as where To make (A2) with inequality constraints be approximate to the one with equality constraints, (A2) can be rewritten as where u > 0 , and its approximate function isÎ (u) = − log(−u), where t > 0 is the parameter that decides the approximate accuracy. By replacing I_ withÎ_ in (A3), the approximate form of (A2) is given as Then the logarithmic barrier function of SP1 is defined as To simplify the symbols, we multiply the objective function of (A4) by t, and get its equality The optimal solutions for (A4) and (A5) are the same. For t > 0, the optimization problem (A6) with an equality constraint can be solved using the Newton method and obtain its unique solution x * . We can iteratively calculate the corresponding x * for a series of increasing t values until t ≥ 1∕ , where is the predetermined accuracy parameter. The iterative steps are summarized in Algorithm 1. while end while Solve (A6) using Newton Method.
Thus, (A6) can be denoted as At the feasible point x, to derive the Newton direction Δx ng of problem (A8) with an equality constraint, we replace the objective function of (A8) with its second-derivative Taylor approximation around x where v is the variable. And (A9) is a convex quadratic minimization problem with an equality constraint. The Newton direction Δx ng at the feasible point x is determined by where w is the optimal dual variable of the quadratic problem (A9). Problem (A10) can be solved by The Newton decrement for problem (A8) is defined as to (7).
We use the GBD method to solve SP2, referring to chapter 13 of Reference 32. For MINLP SP2, continuous variables are y and f , integer variables are q and O; the optimal solutions are denoted as y * , f * , q * , and O * . Obviously, the key to solve MINLP SP2 is to find the optimal solution q * and O * . When we obtain the integer solution, MINLP is simplified to a linear programming (LP) problem, that is, after q * and O * are determined, it is not difficult to obtain y * and f * .
The GBD algorithm is used to iteratively solve MINLP SP2. The underlying steps are described in the following. Based on the GBD method, MINLP can be decomposed into a master problem (MP) (B3), which is an integer optimization problem, and a SP (B4), which is a linear optimization problem. The integer constraints are considered in the MP to optimize the integer variables and obtain the lower bound (LB) of the original problem. After obtaining the integer solution, MINLP is reduced to SP to optimize the continuous variables and obtain the upper bound (UB) of the original problem. When the predicted LB is bigger than or equal to the UB, the algorithm terminates. Generally, the initially obtained integer variables are not the optimal solution, but we can narrow the search/feasible space by adding new integer constraints in the MP to approximate the optimum. In other words, SP is able to find the feasible solution but UB is larger than LB (ie, the determined q and O are not the optimal solution). In order to improve the optimal solution of q and O, the feasible constraint (40a) is added to MP such that the new LB is larger than the previous LB. When the SP is infeasible to find a feasible solution, to prevent obtaining the infeasible solution again, an infeasible constraint (40b) is added to the MP. When it meets the stopping criterion |UB − LB| ≤ , the optimal solution is converged, where is the rate of fault tolerance. The iterative process is summarized in Algorithm 2, and the involved definitions are listed below. ifsolution is feasible then Solution (y k ,f k ) and Lagrangian multipliers ( k , k ) are obtained;

else if solution is infeasible then
Solve (q k , O k ) based on the dual decomposition method; Solution (y k ,f k ) and Lagrangian multipliers ( k , k ) are obtained; (B5)

APPENDIX C. SOLVE SP3 USING HUNGARIAN ALGORITHM
The Hungarian algorithm is a combinatorial optimization method which solves the task assignment problem in polynomial time. To transform SP3 into the standard form of the assignment problem equivalently, we first make the definition Thus, the standard assignment problem is given by where ij denotes that the assignment of MEC server i to cloud server j, and the completed assignment is denoted by ij = 1, otherwise ij = 0. Since the constraint matrix is completely unimodular, we can always find an optimal integer solution with relaxation of ij . In order to illustrate the application of the Hungarian algorithm to (C2) specifically, without loss of generality, a simple case is considered in this paper with | | = 5, || = 3. Since the rows of the cost matrix  should be equal to its columns , we add two additional dummy cloud servers CS 4 and CS 5 . The above problem can be illustrated graphically: five MEC servers MECS 1 , MECS 2 , MECS 3 , MECS 4 , and MECS 5 , three cloud servers, CS 1 , CS 2 and CS 3 , and two dummy cloud servers CS 4 and CS 5 . The lines from MECS i to MECS j represent the values of cost d ij C ij , with all d i4 C i4 and d i5 C i5 setting to 0. Without loss of generality, the cost matrix is defined as the n × n matrix The basic idea of the Hungarian algorithm: Subtract a constant from the rows and columns of the cost matrix C, reduce C to contain n zero elements located in different rows and different columns. Then, we get the optimal assignment of zeros which corresponds to the positions of the n entries in the original cost matrix. Finally, the sum of the n entries is the smallest possible cost. For the assignment problem, after adding (or subtracting) an identical number to any row (or column) of the cost matrix C, the optimal solution for the new cost matrix is also optimal for the original. 32 Given a n × n cost matrix, Theorem 1 is applied in Algorithm 3 to find an optimal assignment. In order to clearly explain the Algorithm 3 used to solve (C2), without loss of generality, we demonstrate the matrix transformation process in the following. The detailed steps are as follows.
Step 1 : subtract 0 from each row and find the smallest element of each column.
Step 3 : cover all zeros in the rows and columns with the least horizontal or vertical lines. Since nis bigger than the quantity of covering lines, we find the smallest entry not covered by any line is 10.
Step 4 : subtract 10 from all rows without covering lines, and add 10 to all columns with covering lines.
Step 5 : cover all zeros in the rows and columns with the least horizontal and vertical lines. Since the quantity of covering lines is 5, an optimal assignment of zeros is obtained. Start with a row or column with a minimum of 0 elements, circle all zeros with ○, and then cross out the remaining zeros in the same rows and columns.
After obtaining an optimal assignment of (C2), Algorithm 4 is used to update the corresponding parameters in SP3, then a new task assignment problem (C2) is formulated. Similarly, to get two sets of nodes with equal size and the corresponding cost, we add two additional dummy cloud servers. Again, Hungarian Algorithm 3 is applied to solve (C2), an optimal assignment is obtained, and Algorithm 4 is applied to update parameters in SP3. Repeat this process until all the unprocessed requests have been dispatched from MEC servers to cloud servers.

Algorithm 4. Update parameters in SP3
else if C ij = =y j then /* Remove j */ * ij ← y j ;