Review of distributed control and optimization in energy internet: From traditional methods to artificial intelligence ‐ based methods

Energy internet (EI) can alleviate the arduous challenges brought about by the energy crisis and global warming and has aroused the concern of many scholars. In the research of EI control systems, the access of distributed energy causes the power system to exhibit complex nonlinearity, high uncertainty and strong coupling. Traditional control and optimization methods often have limited effectiveness in solving these problems. With the widespread application of distributed control technology and the maturity of artificial intelligence (AI) technology, the combination of distributed control and AI has become an effective method to break through current research bottlenecks. This study reviews the research progress of EI distributed control technologies based on AI in recent years. It can be found that AI ‐ based distributed control methods have many advantages in maintaining EI stability and achieving optimal energy management. This combination of AI and distributed control makes EI control systems more intelligent, safe and efficient, which will be an important direction for future research.

� It replaces nonrenewable energy such as fossil energy with renewable energy such as solar and wind energy; � Energy is generated, stored and consumed at the same time to achieve efficient system operation; � Energy flows in both directions, and users can be both energy producers and energy consumers; � Cold, heat, electricity and gas networks interconnect with each other to accomplish the conversion between different forms of energy; � A large-capacity energy storage system is required to ensure a stable output of energy; � The energy flow and the information flow circulate in both directions.
The implementation of EI requires a large number of advanced supportive technologies, such as control technology [14][15][16][17], energy storage technology [18,19], power grid security technology [20], renewable energy-based power generation technology [21] and big data technology [22]. It is worth mentioning that EI is currently at the level of theoretical research. Many demonstrative projects have also implemented EI's theoretical techniques and achieved relatively satisfactory results, providing valuable experience for EI's research and development. For example, Beijing Yanqing Energy Internet Comprehensive Demonstration Zone in China, Shanghai Chongming Energy Internet Comprehensive Demonstration Project, France's Greenlys Project, Sweden's Stockholm Royal Sea Port Project and the US Irvine SG Demonstration Project [23].

| Distributed control technology in energy internet
This study mainly focuses on the control technology of EI. In recent years, the research on EI control and energy management strategies has yielded many results. For example, Hua et al. [24] describe the energy management problem as a stochastic optimization problem, which can be solved using dynamic programming. The modelling process combines the recurrent neural network with Ornstein-Uhlenbeck process to obtain the accurate power model of the photovoltaic (PV) panel and load. The proposed control method can not only effectively prolong the service life of energy storage equipment, but also realize the reasonable use of microturbines and avoid the situation of overcontrol. In [25], a short-term wind power forecasting algorithm based on noncooperative game theory and deep learning is proposed in microgrid energy management. The algorithm uses a stacked automatic encoder to extract features from the training data, back-propagation algorithm to calculate the weight of the overall neural network and genetic algorithm (GA) to optimize the learning speed of the entire process. Their experiments show that accurate wind power prediction results are helpful for the design of management schemes. Kumrai et al. [26] propose a fitnessbased modified game particle swarm optimization algorithm to minimize the operating costs of microgrid and multimicrogrid systems while minimizing pollutant emissions.
Energy management and control problems are usually solved as optimization problems. Most existing solutions can be divided into two categories, namely centralized methods and distributed methods [27,28]. A large number of existing projects use centralized methods to solve management and control problems. However, as more and more power devices are deployed in the distribution network, the centralized approach faces many challenges.
Centralized methods usually require the establishment of a central controller to monitor the system and make decisions [29,30]. As the number of distributed generation devices increases, centralized methods may become increasingly difficult to operate. The main reasons are listed as follows [27,31,32]: � Lack of specialized management units; � Large amount of computing; � Difficulty in timely communication due to large geographical span; � Complicated to redesign, even replacing only one unit will affect the central controller; � Lack of data sharing; � Reliability and security issues of the central controller.
In contrast, distributed control technology fully considers the interaction between units, and assigns control tasks to different units according to the control objectives of different periods [32]. All smart devices work together to reach a collective decision based on the set goals. Each controller only needs to communicate with neighbouring nodes. Global information about the network (i.e. the status of all nodes) is not required to make control decisions [33].
Compared with centralized control, distributed control has many advantages. First, distributed control algorithms are robust to the failure of a single controller node [34]. Moreover, only limited information is shared between each pair of nodes, which not only improves the security of the network, but also reduces the construction cost of the basic communication 64 facilities [35]. Since parallel computing is enabled, the computing speed and the scalability of the system can be superior to the centralized algorithm [36]. In addition, distributed algorithms can protect privacy well, which is of great significance in future practical applications. Therefore, more and more studies focus on distributed control in recent years.

| Artificial intelligence-based control
With the development of artificial intelligence (AI) technology and computer hardware, using AI to solve complicated problems has become a research hotspot in recent years. In the studies of renewable energy power generation and load forecasting, traditional physical methods usually require a large amount of computing and are extremely sensitive to initial conditions. It is difficult to improve their prediction accuracy at the current level. Deep neural network can fuse massive data information through the association of feature variables to improve the accuracy of prediction and early warning [37,38].
In the field of control, taking stability research as an example, the randomness of renewable energy power generation makes it difficult to determine the operation mode of the power grid, and the complexity of control continues to increase. Outdated offline control strategies may not match actual working conditions. The comprehensive guarantee technology based on AI, such as machine learning, fuzzy set theory or multiagent, can effectively improve the stability of EI. For example, machine learning can continuously monitor the operating status of the system [39], automatically determine abnormal conditions, early warning of possible risks [40], and reduce the risk of misoperation and refusal of relay protection. The use of reinforcement learning (RL) can improve the degree of matching of emergency control strategies with real working conditions [41]. Data-driven response technology can cope with small probability accidents and prevent system crashes.
AI is an effective tool for solving complicated situations such as nonlinear problems [42]. Modelling errors in traditional methods can also be reduced by AI-based methods. Besides, traditional control methods may sometimes be difficult to achieve the desired control effect when the practical system's operating state deviates unexpectedly from the theoretical assumptions or models. In contrast, AI-based control methods can be more proficient in continuously tracking the changes of the system, adjusting control strategies and improving the ability to deal with uncertainty [43]. AI-based methods also have unique advantages in terms of computing speed, modelling of complicated problems and system automation degree.

| The difference between microgrid, smart grid and energy internet
In particular, EI mentioned is a broad concept, including smart grid and microgrid. A microgrid is a small energy system composed of distributed power generation devices, energy storage devices, energy conversion devices, loads and related control and protection devices [44]. It can accomplish self-control and self-management due to its capability of operating either in parallel with the external grid or in isolation [45]. Compared with microgrid, smart grid considers various problems in the energy system based on the overall situation of regional power grid. It uses sensors to monitor critical devices for power generation, transmission and power supply in the energy network in real-time and further integrates and analyses the acquired data [46]. Smart grid can achieve optimal management in a more extensive energy network according to the analysis results. EI, in contrast, further expands and deepens the concept of smart grid. It differs with smart grid in the following ways [47][48][49]: � The physical entity of an EI is composed of electricity, natural gas and transportation systems, while the physical entity of smart grid is mainly the power system; � The energy in an EI can be transformed into various forms, such as electric energy and thermal energy. In contrast, the energy in smart grid is only transmitted and used in the form of electric energy; � There are more participants in an EI. In addition, its energy consumption forms include both local consumption and wide-area coordination. In smart grid, energy consumption is mainly local.
Due to the lack of existing research work on the combination of AI and distributed control in high-voltage energy systems, the technologies discussed mainly focus on the lowvoltage part. Therefore, the term 'EI' refers to the lowvoltage type, a local energy system composed of microgrids or smart grids.

| Contributions of this study
Today, more and more projects choose AI-based methods to solve specific problems in energy management and control [50][51][52][53]. This study mainly focuses on the distributed control based on AI technologies rather than the application of AI technology in distributed systems. This study does not attempt to list traditional distributed control methods and AI-based distributed control methods. Instead, this study first describes the development trend from centralized control to distributed control and then to distributed control based on AI, and then analyses the contingency and inevitability of this trend combined with the development direction of energy system. So far, although there is not much research work on the combination of distributed control and AI, this is a meaningful research direction. This study summarizes and analyses some existing work to show readers a variety of research methods and ideas in related fields and provides specific reference value for scholars engaged in EI control research. HUA ET AL.

-65
The rest of this study is organized as follows: Section 2 briefly describes traditional distributed control methods in EI; Section 3 briefly introduces the AI technology; Section 4 summarizes the research work of the combination of AI and distributed control in the study of EI; Section 5 concludes this study and provides outlook for future work. The architecture of this study is shown in Figure 1.

| TRADITIONAL DISTRIBUTED CONTROL METHODS IN ENERGY INTERNET
This section briefly introduces traditional distributed control methods in EI and outlines some existing research works. These works do not use AI methods in modelling or problem solving.

| Distributed control technology in energy internet
The distributed control technology in EI needs to fully consider the interaction between units. According to different control objectives with respect to different time periods, tasks are assigned to different units. Each unit retains sufficient autonomy. When there are enough units, one way to perform coordination strategies is to establish a control hierarchy. Depending on the required time frame, the control hierarchy can be divided into primary control, secondary control (also called energy management system) and tertiary control [32]. Primary control is the fastest and responds to system dynamics in real-time. It is often used to ensure that the voltage and frequency are within controllable ranges. The use of secondary control can alleviate long-term voltage and frequency deviations while coordinating units to achieve other goals, such as power quality optimization or loss reduction. Tertiary control is the most advanced control and is responsible for managing multiple microgrids.
As summarized in [32], distributed control technologies commonly used in EI include distributed model predictive control-based techniques [54], consensus-based techniques [55], agent-based techniques [56] and decomposition-based techniques [57]. In the current research of EI control systems, distributed control has been widely used in economic dispatch, frequency conversion speed regulation, voltage control and many other fields.

| Applications of traditional distributed control in energy internet
The studies reviewed in this section use traditional distributed control without involving AI methods in modelling and solving.

| Traditional distributed control for system stability
The stability of EI refers to the ability of the EI system to resist disturbances. The massive access to renewable energy increases the disturbances that the entire power system may face. In the absence of an effective control scheme, voltage and frequency fluctuations in a wide range caused by disturbances may interrupt the entire power system and cause significant losses. How to ensure the long-term stable operation of the system is a problem that scholars are more concerned about.

Energy storage
Energy storage system is an important device in EI, which can be used to maintain the stability of the system. Bahramipanah et al. [58] use a decentralized adaptive model with battery energy storage systems for real-time power grid control. Its control objectives include voltage control and congestion management. The author in [58] partitions the entire energy network into areas and conducts control to multiple areas.
Voltage regulation in real-time network control is achieved by considering the accurate dynamic model of battery energy storage systems. Compared with the previous work in [59], the distributed design in [58] effectively reduces the communication cost and computation workload. In order to coordinate energy storage units, PV panels and controllable load units in single-phase low-voltage microgrids, Golsorkhi et al. [60] propose a novel distributed cooperative control framework to regulate the voltage, and coordinate the charge and power state between each energy storage unit. In addition, measures to limit PV power are also configured in the system to avoid overcharging or overdischarging of energy storage units. The distributed method proposed can avoid the disadvantages of single point failure or high communication cost that may occur in the centralized method. It achieves better performance by avoiding power quality degradation due to frequency and voltage deviations.

Reactive power optimization
Reactive power optimization is a measure of reactive power adjustment that optimizes one or more performance indicators of the system under given structural parameters and load conditions. It aims to maintain the voltage level by reasonably allocating reactive power flow. Therefore, reactive power optimization is an important means to maintain grid stability.
In [61], an optimal reactive control scheme based on a fully distributed multiagent system is established. Compared with previous centralized and semidistributed control methods, this fully distributed control scheme can effectively reduce the probability of single point of failure. In addition, it can not only respond to environmental changes in a timely manner to ensure the stability of the system, but also has scalability for systems of different sizes and topologies. Similarly, Shafiee et al. [62] propose a fully distributed control methodology for secondary control of AC microgrids. This method guarantees global voltage and frequency adjustment as well as accurate active/reactive power sharing in droop-based microgrids. Each power supply participates in reactive power support according to its predetermined rated power. The method also uses active power measurements to successfully synchronize the frequencies among multiple microgrids, so the controller no longer requires additional measurement equipment, thereby reducing costs.

Active power sharing
There are also some other works that study active power sharing between microgrids. Considering the more practical situation where multiple microgrids are interconnected, an event-based distributed consensus-based control approach is designed in [63]. The advantage of adopting the event-based method is that the communication between agents is greatly reduced, and the flexibility and stability of the entire system are improved. The use of distributed methods also enables the plug-and-play function of the system, which can still maintain the effectiveness in the case of islands and communication link loss.

| Traditional distributed control for optimal energy management
Optimal energy management is also an important research direction of EI. The measures proposed in the studies of this area can minimize the cost of power generation [64], maximize social welfare [65,66] and achieve economic dispatch, thereby making EI operate more rationally and efficiently.

Social welfare maximization
Social welfare maximization is a goal to reduce the total production cost of all power generators as much as possible, while maximizing the total utility of all users [67]. On this issue, Xu et al. [68] propose a distributed optimal control algorithm. The construction of the objective function takes into account both the generator and the load user. Each unit uses a consensus algorithm to find the common incremental cost by minimizing the incremental difference between adjacent units. The adjustment rate is then controlled to optimize the power generation or load change process. Therefore, the proposed control method can achieve the dynamic minimization of adjustment costs while ensuring the balance of smart grid power generation demand. The algorithm is robust to communication failures due to the distributed control method. Moreover, it is adaptive to communication topology changes. Future research on this issue should focus more on improving distributed solutions, such as introducing energy storage system constraints.
The Social Welfare Maximization energy management problem in smart grid is also studied in [69]. The study aims to maximize the overall social welfare that balances power generation costs, user-side payments and transmission costs. Through continuous information exchange, the distributed projected control algorithm can obtain the global optimal solution asymptotically. In order to save communication resources, the event-triggered condition of each generator and each load is used to determine when its related states should be sampled and transmitted to adjacent loads.

Demand response
Demand response research can promote the development of the power industry towards higher efficiency. To achieve optimal energy management scheduling between users and utility companies, a distributed real-time scheduling algorithm is designed in [70]. The algorithm uses dual decomposition technology to decompose the original problem into several independent subproblems, which overcome the obstacles caused by spatial coupling constraints.
Since a noncoordinated response of customers may lead to severe peak rebounds at periods with lower prices, it is sometimes necessary to coordinate demand to avoid peak rebounds. Safdarian et al. [71] propose a system-wide demand response management model to coordinate the demand response of residential customers. The model is first described as a bilevel optimization problem. Then the problem is converted into an equivalent single-level problem, which is finally solved by an iterative distributed algorithm so that the impact to total load curve by user demand is minimized. Nevertheless, the method in [71] fails to consider network constraints, which can be a future research direction.
On the other hand, Diekerhof et al. [72] propose a hierarchical robust distributed optimization method suitable for day-ahead and intraday scheduling of flexible devices (electric thermal units) in urban areas. The optimization is based on direction alternating of multipliers, which can prioritize each individual customer and its own private objective, and fully consider the needs of customers in the scheduling process.
There are some other works focusing on minimizing the total power generation cost while satisfying the total demand and the power generation limit of a single generator. The distributed algorithm provided in [64] is based on the results of [73,74] and incorporates the robust control methods in [75,76]. This algorithm can be used to solve optimal coordination of distributed energy resources in communication networks with packet loss. Compared with some previous research works, the method in [64] is more robust and has a smaller computational load. Further studies could extend the method under more constraints, for example, transmission line loss, power flow and transmission line flow constraints.
For the problem of inaccurate prediction, Nguyen et al. [77] develop a distributed controller based on the work of [78]. A distributed model predictive controller is embedded in the universal smart energy framework. There is also an aggregator layer above the prosumer layer. These two layers are coupled by an objective function to form a three-tier structure, which balances the responsible party, aggregators and prosumers. The flexibility of the system is quantified in order to distribute the day-ahead planning to various integrators, and then a model predictive controller is developed to minimize the imbalance between grid forecast and actual supply and demand. The improvement of [77] lies in integrating multiple tiers, such as flexible consumption and congestion management, into one model, which is more in line with practical application requirements.
In [79], the real-time scheduling problem of energy hub under dynamic pricing market is studied. The interaction between energy hubs is modelled as a potential game, given the accurate potential function of the energy centre game. The authors prove that only the Nash equilibrium corresponds to the global maximum of the potential function. The Nash equilibrium is then determined by a distributed energy scheduling algorithm. This scheduling algorithm can be executed by the energy management system of each energy hub in real-time to determine the profit maximization strategy of the user's electrical and thermal devices.

| From traditional methods to artificial intelligence
Nowadays, advanced AI algorithms are becoming more and more consummate, and the functions of computer hardware are constantly improving. Although the massive data generated by EI devices increase the complexity of system control, they provide possibility and feasibility for the practical application of AI technologies at the same time.
In controlling and retrieving massive data streams, traditional methods usually require a local infrastructure to access each device. This not only leads to increased costs, but also limits the size of the data being processed. Therefore, adaptive algorithms and AI-based coordination mechanisms are needed to achieve flexibility and distributed data management [80][81][82].
In addition, big data in the power grid conceal a lot of valuable information. Through the analysis and utilization of these data, AI technology can realize the automation and intelligence of the EI control system, thereby completing more precise and intelligent control and scheduling. Traditional methods may have overlooked the value behind these data. A detailed comparison between traditional methods and AI methods is shown in Figure 2.
On the prediction problem, AI methods can effectively improve prediction accuracy and break through the bottleneck of traditional methods. For example, in electricity price prediction, existing technologies include statistical models, time series methods and AI-based methods. Compared with the high volatility of independent and dependent variables in statistical models, AI-based methods have significant advantages in terms of estimation accuracy [83]. In addition, AI technology can also deal well with nonlinear problems related to short-term electricity price forecasting [84].
AI technology is also widely used in the prediction of renewable energy power generation [24]. Wind speed, light intensity and other factors that may affect the power generation of renewable energy could bring strong nonlinearity and great uncertainty to the control problem, which makes it rather difficult to solve the problem by traditional power generation forecasting methods. AI methods such as neural networks and GAs, however, are important means to solve nonlinear problems [85]. These methods can discover pattern from a large amount of historical data and improve prediction accuracy. For example, extreme learning machines and direct quantile regression can be combined to achieve nonparametric probability prediction of wind power generation [86]. In addition, the hybrid of integrated deep learning framework and an attention mechanism can be implemented to predict PV power output. This high-precision prediction of the power generation equipment is indispensable in future EI systems.
In terms of EI system control, traditional modelling methods inevitably have errors, and sometimes they have difficulties achieving the ideal control effect. For some complicated problems, traditional physical modelling is even infeasible. In contrast, AI-based modelling methods can not only improve the accuracy of the model, but also reduce the difficulty of modelling complicated problems. For instance, RL methods have the unique features of 'no model' and 'no prior information required'. In addition, the input and output data dimensions of EI are very high at present, and the data are usually interrelated. Traditional control methods are difficult to obtain comprehensive control and optimization strategies, while AI methods such as deep learning, RL and transfer learning are effective ways to solve these problems [87,88].
In other aspects of EI, AI is also a good auxiliary tool. In power grid stability analysis, an AI-based method can significantly improve the efficiency of analysing large-scale power grid data [89]. Moreover, the online load modelling method based on big data not only improves the accuracy of the model, but also increases the processing speed while reducing the repetitive workload of the staff. In the grid dispatching problem, the method based on AI can transform dispatching method from empirical to intelligent, and complete better adjustment under more constraints [90]. In the field of power grid protection and control, emergency control schemes based on RL can quickly provide real-time control schemes based on the operating state of the power grid, helping the grid resume normal operation faster [91].
The operation and management of future EI will develop in the direction of becoming more and more intelligent. Massive data have been difficult to process with the experience of the staff, and human participation in the operation of the power grid needs to be reduced. AI is an effective way to realize this concept. Similarly, as one of the important control methods in recent years, the combination of distributed control technology and AI will be a promising direction. The combined use of these two technologies can not only solve some nonconvex, nonlinear and other complicated problems, but also has the advantages of fast calculation speed, low calculation cost and good privacy.

| OVERVIEW OF ARTIFICIAL INTELLIGENCE TECHNOLOGY IN ENERGY INTERNET
AI technology can be generally divided into four areas, namely expert system (ESs), fuzzy logic, artificial neural networks (ANNs) and GAs or generalized evolutionary computation [92]. This section will briefly introduce some commonly used AI methods in EI, including ANNs, RL, GAs and ESs and the application of these methods in EI.

| Artificial neural networks
An ANN is an operation model composed of a large number of interconnected nodes, also called neurons. Neurons can handle the complicated behaviour of the system by the connections between neurons and weight parameters [93]. Perceptron is a commonly used model of neuron. It accepts multiple different inputs, sums them with specified weights and then gets the output through the activation function [94]. In general, multiple parallel perceptrons form a layer, and the layers are serially connected. The output of the previous layer is used as the input of the next layer, forming a multilayer network architecture as a whole. ANNs can solve problems through massive data training. The main training modes are supervised mode and unsupervised mode. The advantages of ANN include adaptive learning, self-organization, fault tolerance and easy integration with existing technologies [83].
In the EI system, ANN can discover the nonlinear relationship between variables in complex environments through good learning ability [83]. As a consequence, ANN has a F I G U R E 2 From traditional methods to artificial intelligence HUA ET AL.
-69 significant effect in solving prediction problems such as the output power prediction of PV systems [95,96], household energy consumption forecasting [97,98] and power system state prediction [99]. In the research of demand response, the use of ANN can complete the modelling of controllable loads under complicated constraints. This modelling method is simpler than traditional modelling methods, and the resulting model is more accurate. For example, Mosaddegh et al. [100] establish Bayesian regularization back-propagation algorithm to obtain a neural network model of controllable loads based on the history of load data and achieve optimal energy management. In addition, ANNs have also been widely applied to EI energy management [101,102], fault detection [103], network security [104] and many other research topics.

| Reinforcement learning and deep reinforcement learning
RL is an important group of machine learning algorithms. In RL, the agent learns in a 'trial-and-error' manner, and the action taken for each 'trial-and-error' is random. Agents guide their subsequent actions based on the reward and punishment obtained from actions taken in the current environment. The ultimate goal of the training is to enable agents to obtain the maximum reward, so that the external environment can best evaluate the learning system in a certain sense.
RL has a wide range of applications in EI since it is proficient in solving decision problems under uncertain conditions. At the cybersecurity level of the grid, the online anomaly detection can be described as a partially observable Markov decision process problem, and the model-free RL framework of partially observable Markov decision process problems can be utilized in establishing a general robust online detection algorithm [105]. The algorithm can detect network attacks against the power grid in time, which is convenient for the system to take reasonable countermeasures before any damage is caused by the attacks, ensuring the network security of the system. In addition, RL is a common solution in the fields of energy trading [106], dynamic pricing and energy consumption scheduling [107] and demand response [108].
At present, the combination of RL and deep learning has also brought a new field, deep reinforcement learning (DRL). Some works have begun to use DRL in solving many complicated problems. For example, Wan et al. [109] describe the real-time charging scheduling of electric vehicles as a Markov decision process with unknown transition probabilities, and propose a model-free optimal scheduling method using DRL to obtain charge and discharge scheduling. Mocanu et al. [88] use the deep policy gradient method as part of the DRL method to perform online optimization of energy management system scheduling. An et al. [110] propose a DRLbased scheme to detect integrity attacks in AC power grid. In [111], the DRL method is used to obtain an optimal energy management strategy, such that the operation cost of the considered EI scenario can be minimized.

| Metaheuristic algorithms
Metaheuristic algorithms mainly refer to a general type of heuristic algorithm. They are the product of the combination of randomized algorithms and local search algorithms, such as GA, simulated annealing algorithm and ant colony optimization. These algorithms have great similarity in the optimization process, and they all have 'neighbourhood search' structure. A typical metaheuristic algorithm starts with a set of initial solutions. Under the control of the key parameters of the algorithm, the neighbourhood function generates multiple neighbourhood solutions, and continuously updates the key parameters and states until the convergence criteria are satisfied. The optimization mechanism does not depend too much on the organizational structure information of the algorithm, and can well solve combinatorial optimization and function calculation. This study mainly introduces GAs commonly used in EI.
GA is a randomized search method that borrows from the evolutionary laws of the biological world (such as survival of the fittest). Through the genetic operations of replication, crossover and mutation, the group of 'chromosomes' represented by the problem code can 'evolve' from generation to generation. When the result eventually converges to the most suitable group, it can be considered that the optimal or satisfactory solution to the problem is found. GA has the advantages of simple principle and operation, strong versatility, unlimited constraints and parallelism and global searching capabilities. At the same time, as a stochastic optimization method, GA considers probabilistic factors in the algorithm, which helps it escape from the local optimum and find the global optimal solution [112].
There have been some studies using GA to solve problems in EI. The method proposed in [113] accomplishes a two-step forecasting of electricity prices: in the first step, a set of relevance vector machines (RVM) is adopted, and each RVM is used to make individual advance price predictions; the second step is to integrate RVM prediction into multiple linear regression ensemble, and use GA to get regression coefficients. In order to achieve route optimization of electric vehicles, a learnable partheno-GA combining GA with a knowledge model can be utilized to solve the optimal path model [114]. Acquiring useful expert knowledge from these dynamically updated solutions helps guide the subsequent searching process to quickly discover a more accurate electric vehicles route.
In EI, the application of GA can handle some optimization problems pretty well. However, GA also has the problem of premature convergence, especially when the problem is nonlinear and there are multiple local minima. This defect can be solved by making appropriate improvements to GA. In [115], a memory-based GA can automatically and optimally fairly share power generation tasks among the distributed energy resources in microgrid. It is further pointed out that it is beneficial to improve the performance of GA by using memory schemes to reuse the stored useful information.

| Expert system
ES is an intelligent computer program based on Boolean logic, which covers massive knowledge or experience in a specific field and can be utilized to solve problems in this field. The core components of ES mainly include the knowledge base and the inference engine. The knowledge base is composed of knowledge, data, facts and sentences that support this knowledge, which are the basis of reasoning. The inference engine is used to control and coordinate the entire system. It relies on the knowledge in the database to obtain the results of the problem through algorithms.
In EI systems, ES has a wide range of applications. In terms of improving power quality, Moreira et al. [116] propose an ES to select the most suitable compensator through k-nearest neighbour pattern recognition algorithm and the knowledge base, thereby reducing losses and increasing power quality. Compared with the technology based on decision trees or neural networks, the classification system based on ES has higher classification accuracy in such problems. On the issue of energy management, ES can be combined with a variety of learning algorithms to enhance the classification function to achieve energy saving and management of smart homes [117]. In addition, ES can also be applied to problems like power grid fault recovery [118].

| DISTRIBUTED CONTROL BASED ON ARTIFICIAL INTELLIGENCE IN ENERGY INTERNET
By deeply integrating energy systems and the internet, EI emphasizes the characteristics of energy equivalence, openness, intelligence and timely response. Traditional technologies generally have difficulties in establishing accurate models, obtaining results in a short time and meeting the requirements of high intelligence [119]. On the other hand, intelligenceenabled modelling, control and optimization methods can quickly adapt to the environment and have dynamic predictability, strong fault tolerance and robustness to disturbances [120]. That is why AI-based distributed control and management methods are more and more favourable in solving complicated problems.

| Distributed control based on artificial intelligence for system stability
The stability of the power system has always been regarded as an important guarantee for the safe and efficient operation of EI. Stability refers to the ability of the power grid to withstand disturbances [121,122]. With the large-scale access to distributed energy and the integration of information technology, EI faces disturbances from both the physical layer and the network layer. Therefore, maintaining system stability becomes more challenging [123]. Optimization studies with system stability as the research goal, such as transient voltage stability, are of great significance for maintaining the effective and safe operation of power systems [124]. Among many system control methods, distributed control can give consideration to remote data and minimize the requirements of communication. At the same time, distributed controllers are more reliable in terms of network security [123]. In view of the advantages of AI technology, research on AI-based distributed control methods with respect to system stability has achieved some results, which are summarized in Table 1.

| Voltage control
Voltage instability is one of the most common causes of power quality degradation of the system. In extreme cases, a voltage collapse will cause the entire system to power off [131]. It is a basic idea to keep the voltage stable within a controllable range and avoid large fluctuations during operation. AI approaches such as neural networks and machine learning can be well combined with distributed methods to provide effective solutions for voltage control.
For example, Karim et al. [125] bring up a distributed secondary control method for maintaining rated voltage in an independent microgrid. This method trains a distributed machine learning algorithm based on different voltage stability conditions. The algorithm first takes available wind energy, available solar energy, controllable load and load mutation as input attributes, and takes a binary class representing system stability or instability as the target attribute. It then uses a set of bagged decision trees to prepare for the classification process. If the classifier predicts possible instability, an appropriate neural network will be selected based on cluster values corresponding to the specific events prepared in advance. The selected neural network will then make necessary modifications to the main controller in a single cluster. Elmitwally et al. also propose a control scheme without energy storage that uses pulse width modulation to track the maximum power of the PV array [132]. In addition, a fuzzy logic-based diesel generator speed control scheme is designed for the same research problem. This method is sufficiently effective for diesel PV power generation systems, but it fails to suit microgrids based on wind PV, which indicates the meaningfulness of [125].
There are other research ideas about distributed secondary voltage control methods, like the distributed collaborative control strategy adopted in [126]. In more detail, it combines radial basis function neural network with sliding mode control to stabilize the system in a short time. The radial basis function neural network is used to adjust the switching gain of the sliding mode control in real-time to reduce chattering, where the sliding mode control is used to restore the microgrid voltage. However, the microgrid model in [126] does not conform to the real situation because the authors fail to address the delay and interference of communication links. It would be more sensible if future research can be conducted in a more realistic microgrid model.
On the other hand, Amoateng et al. [127] design an inverter-based distributed voltage controller based on ANN and collaborative control theory under the multimicrogrid structure. In their study, the model-based controllers are first designed using Lyapunov theory and the dynamics of the distributed generation system. Then ANN is used to approximate these dynamics and minimize the cooperative tracking error function, thus obtaining a smart controller that does not require much prior information. The proposed controller achieves good active and reactive power sharing in distributed multiple microgrids, and it has strong robustness to power system disturbances. Compared with the previous work in [133], the controller proposed in [127] is simpler and requires less information. Future studies can explore how the controller of [127] can keep running in the presence of system failures.

| Frequency control
In the EI system, distributed power generation has great uncertainty. Some power system components also have nonlinear characteristics, so they are prone to frequency fluctuation issue, which affects the stability of the power grid [134]. Frequency control is also an important means to ensure the stable operation of the power grid.
Regarding the imbalance between power generation and load, the traditional centralized load frequency control structure is not convenient for exchanging information in large scale. In addition, the increasing calculation and storage costs make this structure more and more difficult in practical implementation. To solve this problem, Singh et al. [128] propose a distributed controller that combines RL and multiagent systems. The frequency controller in [128] has lower communication costs, higher flexibility and better effectiveness. It is used to implement load frequency control in a smart grid environment where the communication topology can change dynamically. Using the event-triggered control method, the proposed solution improves the dynamic system performance and reduces the burden of network communication.
Similarly, Sun et al. propose an actor-critic neural network that integrates a distributed RL control scheme to compensate for the frequency regulation of the power grid [129]. The online learning algorithm of this neural network is derived from the constructed error function. The purpose of the learning process is to reduce the error between the actual value and the estimated value of the radial basis, so as to approximate the strategic utility function and optimize the control output. The network structure also establishes the relationship between control output and performance estimation, which further improves the efficiency of energy utilization. Compared with previous methods of separating actors and critics in [135,136], this combination of actor and critic neural network yields two advantages. First, the relationship between the strategic utility function estimation and the expected control output estimation is established to improve the long-term performance. Second, the stability and the bound of performance can be obtained through theoretical analysis.

| Power grid monitoring and fault recovery
Real-time monitoring during power grid operation and timely recovery when a fault occurs can further improve the stability of grid operation and the resilience to small faults. In order to achieve the above purpose, Karim et al. [130] integrate the concepts in [137,138] and propose a novel algorithm that detects dynamic events from distributed generator data in a sectionalized way. Its purpose is to facilitate the decisionmaking process after a fault occurs, so that the independent microgrid can resume normal operation without intervention from the central station. As for data preparation that requires a lot of time and resources, the algorithm considers an alternative method to avoid real-time feature selection by implementing a set of preprocessed input features. In dynamic event detection algorithms and fault recovery mechanisms, machine learning methods are used to improve their performance. Compared with traditional methods, this method reduces the calculation cost and is suitable for practical applications. T A B L E 1 Distributed control methods for system stability

Control problem Scenario Methodology Reference
Voltage control Wind-PV-based isolated microgrid Distributed secondary control based on machine learning. [125] Inverter-based islanded microgrid Secondary controller using radial basis function neural network sliding mode control algorithm. [126] Multimicrogrid structure Adaptive voltage control using distributed cooperative control and adaptive neural networks. [127] Frequency control Smart grid An intelligent controller with communication topology changes using multiagent RL. [128] An actor-critic neural network that integrates a distributed RL control scheme. [129] Power grid monitoring and fault recovery Stand-alone microgrid Feature selection-based distributed machine learning approach. [130] Abbreviation: PV, photovoltaic; RL, reinforcement learning.

| Distributed control based on artificial intelligence for optimal energy management
In addition to stability-oriented control and management strategies, there are also many works that aim to optimize energy use by minimizing costs or maximizing benefits, extending the life of energy storage systems, or minimizing the energy utilization cost. This study summarizes the two main research directions of optimal energy management, namely the demand response problem and the economic dispatch problem that do not consider demand response. Table 2 provides a brief comparison of existing works on these two problems. Note that some existing studies adopt distributed approaches, others use centralized approaches. This study mainly focuses on distributed control methods based on AI since distributed methods have many advantages in the optimal operation problem over centralized methods [27].

| Economic dispatch
The goal of the economic dispatch problem is to establish a reasonable dispatch plan based on predicted energy production and consumption conditions, in order to minimize the total operating cost and achieve the economic operation of EI. In [139], a fully distributed algorithm based on neural network is designed to reduce the total cost. The essential feature of the proposed neurodynamic optimization method is its inherent parallel computation and theoretically guaranteed optimality that can be obtained in real-time without specific initialization. This algorithm can solve the problem when the objective function is not necessarily strictly convex and smooth, with the existence of multiple coupling constraints. Compared with previous methods in [149,150] that only consider local constraints, their results have a wider range of applications.
From the perspective of the operator, the authors in [140] design the energy management algorithm for networked microgrids using the registration minimization and online alternating direction method of multiplier (ADMM) in machine learning. Standard ADMM requires forecast data, and inaccurate forecast results may increase the cost of power generation. What is more, when the standard ADMM uses robust optimization formulation, it may lead to conservative results. Combining ADMM with machine learning and registry minimization can make up for these defects. Furthermore, the algorithm proposed in [140] is implemented in a distributed manner, which significantly reduces the workload in computing and communication. Although [151] also proposes an online optimization algorithm for single microgrid based on regret minimization, the underlying physical power network is ignored in the algorithm design. When designing online energy management, [140] considers both the underlying grid and the networked microgrid, so the method in [140] is more complicated.
In economic dispatch, existing control methods not only consider the operating cost, but also consider other constraints such as the combination of cost-effectiveness and system stability, so that the proposed control method can simultaneously optimize multiple problems.
Kohn et al. [141] propose a new distributed intelligent control and management architecture based on hybrid systems. The uniqueness of this architecture is that it includes a distributed inductive engine in learning local dynamics of generators and loads in the microgrid. Aiming at solving the problem of insufficient accuracy of the load model in traditional methods, an optimization method based on machine learning is adopted, and the load prediction can reflect the T A B L E 2 Distributed control methods for optimal energy management

Economic dispatch
Energy internet A fully distributed algorithm based on neural networks, applicable for nonsmooth and general convex objective functions [139] Networked microgrids A distributed algorithm for energy management based on online alternating direction method of multipliers and machine learning [140] Microgrid A fully distributed algorithm based on neural networks, capable of solving convex optimization where objective function is not necessarily strict convex or smooth [141] A cooperative RL algorithm [142] Smart grid PI frequency controller and neural network-based frequency controllers are used to implement distributed economic dispatch control [143] Multiple energy carrier systems A novel multiagent bargaining learning algorithm [144] Demand response Smart grid A GA-based solution [145] A novel deep transfer Q-learning method associated with a virtual leader-follower pattern [146] Stand-alone microgrid Multiagent cooperation system based on fuzzy Q-learning [147] Microgrid Distributed energy and load management approach based on RL [148] Abbreviations: GA, genetic algorithm; RL, reinforcement learning.
HUA ET AL.
dynamic change of the load in real-time. In addition, the control method has good scalability, meaning that the calculation amount of each node remains unchanged as the number of nodes increases. Although some existing research works can achieve optimal economic dispatch [152][153][154], the acquisition of accurate a priori statistical information of all distributed generator sets and loads in the microgrid is not simple, which limits the practical application of these methods. In order to avoid establishing a random model in advance, when trying to use RL-based methods, the studies in other aspects, such as household energy management [155] and power generation control [156], have achieved good results. However, in the distributed economic dispatch of microgrids, the state space and decision variables are continuous. Classical RL faces the problem of 'curse of dimensionality', and the fuzzy Q-learning algorithm that solves this problem has a slow convergence rate.
Based on this, a collaborative RL algorithm is designed in [142] for microgrid economic dispatch. This algorithm not only minimizes the operating cost of the microgrid, but also keeps the voltage stability of the entire system. A coordination mechanism is introduced in the RL algorithm with function approximation to make up for the deficiencies mentioned above. In this distributed collaboration mechanism, each controller makes action decisions based not only on its own state, but also on the state of neighbouring controllers. The algorithm uses 'trial-and-error' interaction with the dynamic environment to find the optimal decision sequence to minimize operating costs. Future work may as well consider designing a hierarchical RL structure to achieve coordination between multiple microgrids, or adding more constraints.
When studying multiobjective optimization problems, some research works consider reducing the energy loss of the system in the process of economic dispatch. By embedding frequency control into a distributed economic dispatch method based on consensus, the scheme developed by Li et al. [143] can overcome the shortcomings of previous works, such as relying on a centralized information centre to calculate the initial value of mismatch and strong assumptions about the availability of power mismatch [65,157,158]. In addition, Li et al. also show an idea of combining a consensus protocol with a control algorithm, which can be generalized in the future.
For the distributed energy hub economic dispatch of the multiple energy carrier systems, the use of the multiagent bargaining learning method can significantly reduce energy loss while ensuring the minimum total cost [144]. In order to avoid the shortcomings of slow convergence, curse of dimensionality and weak disposal ability to deal with continuously controllable variables in previous research [159][160][161], Q-learning with associative memory is adopted for the learning process of each agent. In addition, nonuniform mutation operators are used to process continuous control variables. This combination has the advantages of fast convergence speed and strong global search ability. It has strong competitiveness compared with other distributed heuristic optimization algorithms.

| Demand response
Solving the demand response problem needs to consider the supply and demand relationship between customers and suppliers. In order to reduce or shift the power load within a certain period of time and respond to the power supply, a reasonable energy management plan can be formulated by combining the energy consumption and load distribution of EI.
In [145], Mosaddegh et al. propose a distributed computing architecture based on smart grid communication middleware system. This architecture is used to solve the distribution optimal power flow model of the distribution network. To achieve voltage and reactive power control of large-scale systems based on the network model and reduce the computation cost, previous works have proposed neural networks and heuristic algorithms that decompose the problem into subproblems. Although the methods introduced by [162,163] reduce the complexity of the distribution optimal power flow model and the amount of calculation, the solution obtained might be suboptimal. Accordingly, Mosaddegh et al. [145] adopt a GA-based method to solve the distribution optimal power flow model. The distributed computing method is applied to the smart grid communication middleware system, which reduces the calculation time and obtains the optimal solution of controllable distributed feeder devices.
For the supply-demand Stackelberg game in the smart grid, a novel deep transfer Q-learning algorithm based on a virtual leader-follower model is proposed in [146]. Its goal is to maximize the total revenue of all agents on the premise of satisfying the power balance between supply and demand. Compared with traditional gradient-based optimization methods, such as Newton's method, quadratic programming method and interior point method, deep transfer Q-learning can better achieve global search and avoid falling into local optima. In addition, compared with centralized metaheuristic optimization algorithms, deep transfer Q-learning has a faster convergence speed, stronger online learning capabilities and can effectively protect users' private information.
To conduct energy management for stand-alone microgrid, Kofinas et al. [147] also propose a cooperative multiagent system. This method takes into account the uncertainty of user demands, and can ensure the power supply of the independent microgrid while maintaining the stability of the entire system. The learning method utilizes local rewards and state information related to each agent. As a result, the state space is reduced and the learning mechanism is enhanced. In addition, fuzzy Q-learning is introduced in each agent to deal with the continuous state space and action space. Compared with previous works, the algorithm in [147] can obtain the management strategy faster. Therefore, this technology can be applied to more complex EI scenarios in the future, for example, the EI systems with wind turbines or hybrid electric vehicles.
There are also some research results on the issue of electricity market transactions. Foruzan et al. [148] design a 74distributed energy and load management method based on multiagent strategies. Through RL, agents can adapt to competitive and random markets, and optimize the utility of both supply and demand in the hourly market based on microgrid auctions. The model-free Q-learning algorithm ensures that each agent can find the optimal strategy, thereby maximizing its own profit. Different from most research work based on multiagent systems, [148] models the energy supply and demand sides of the microgrid as a single unified agent to further study the interaction and demand of both sides. In addition, the distributed design in [148] can effectively reduce the volume of information exchange and improve the response speed.

| CONCLUSION
This study reviews the distributed EI control methods based on AI in recent years. Compared with centralized control methods, the traditional distributed control method has made great progress, with fast calculation speed, low communication cost and high security. However, there are still some limitations in solving nonconvex and nonlinear problems. The rapid development of computer hardware makes AI technology widely used in electronic information systems, and provides effective solutions to problems that traditional methods are difficult to solve. AI-based distributed control methods not only maintain the advantages of distributed control itself, but also have good adaptability to the characteristics of nonlinearity, strong uncertainty, strong coupling, and multivariables of EI system. In addition, flexibility has a positive effect on improving the stability, operating efficiency and intelligence of electronic information systems.
There are not many research results in this area currently, but it will become a research direction with great potential. Future work can try this combination more, or try to add more constraints in previous studies. At present, some existing projects have been completed under ideal conditions. Although they provide good ideas, they are still far away from practical applications, so it is recommended that future works consider situations that are more in line with actual conditions. The concept of EI covers low-voltage, medium-voltage, and high-voltage energy systems. Most existing researches on the combination of AI technology and distributed control focus on the low-voltage side, while the research on the highvoltage side is rare. In the future, the combination of AI technology and distributed control on the high-voltage side will also become a direction of great potential. In addition, some existing projects are too slow to achieve real-time control. Therefore, optimizing the time cost of solving control problems is another important goal for future research.