Joint energy and flexiramp market equilibrium model for regional‐integrated electricity‐heat systems considering flexibility exploitation of district heating networks

The increasing penetration of renewable energy resources into regional‐integrated energy systems (RIES) holds higher standards on operational flexibility. To alleviate the negative effects of insufficient operational flexibility, this paper proposes a joint market equilibrium model to obtain flexible resources in microgrids (MGs) and district heating networks (DHNs) through economic incentives. Based on the incorporation of the flexiramp (FRP) into the energy markets, a joint market mechanism is first established. Through multi‐energy synergy, the FRP provisions on the supply and demand sides in MGs are then introduced. Considering the flexibility exploitation of DHNs, a collaborative FRP supply method is developed to formulate the optimal bidding strategy for MGs. Moreover, a hierarchical model is designed to achieve joint market equilibrium, where MGs strategically bid in the external layer and the RIES operator optimally clears the market in the internal layer. The combined utilization of the Q‐learning algorithm and path‐tracking interior point method is applied to solve the hierarchical model efficiently. The performance of the proposed approach is verified by case studies. The results demonstrate the efficacy of the proposed approach for operational flexibility improvement through minimizing renewable energy curtailment, the overall cost, and the average energy prices.

for operational flexibility. In the context of sustainable energy supply, the regional-integrated energy system (RIES) emerges and rapidly develops to break down the barrier between different forms of energy. The importance of RIES is due to the fact that different energy systems are tightly coupled topologically, and the expanded operating boundary of the system offers a tremendous opportunity to harness the synergistic effect of multiple energy resources.
However, the increased use of energy coupling equipment, such as combined heat and power (CHP) units, has dual effects on the operational flexibility of RIES. On the one hand, more constraints are imposed on the operation, control, and market in RIES. On the other hand, the operational boundaries of RIES can be further expanded through the multi-energy interconversion.

| Literature review and research gaps
In terms of the exploration of the potential operational flexibility of RIES, the market-based method is a promising way to reserve adequate capacity to cope with multiple fluctuations and uncertainty. Additionally, the booming integrated energy market lays the foundation for it. Various researches have been conducted to optimize system operation by applying the market-based method. In, 4 an optimal energy flow method is presented to verify the effectiveness of strategic biddings on optimizing clearing results, but PDNs and district heating networks (DHNs) operate independently from each other. A cooperate energy market mechanism is designed in 5 by applying a generalized locational marginal pricing method, and the price linkage between the electricity and heat markets is analyzed. However, few works on the integrated energy market are carried out from the perspective of improving operational flexibility.
Considering traditional ancillary services fail to deploy flexible resources efficiently, flexiramp (FRP) in the electricity market provides a new train of thought. It is capable to reserve capacity for fluctuation and uncertainty of net load in the following time interval by dispatching flexible resources. 6,7 An optimal day-ahead electricity-FRP market framework is proposed in, 8 where the influence of FRP on market equilibrium is analyzed by establishing a bi-level optimization model. To determine the capacity of FRP more precisely, a Gumbel-copula based multi-interval method is modified in. 9 The above studies mostly focus on FRP provision on the supply side. In, 10 electric vehicles are qualified to participate in the auxiliary service market. Furthermore, a real-time dispatching model considering the electric vehicles providing FRP is established in, 11 laying the basis for the demand side to serve as the FRP provider.
There is a growing interest in exploiting RIES' flexibility in the supply, network, and demand sides. Authors in 12 explore the insufficient flexibility for a small-scale integrated energy system, and establish a thermal inertia model of the buildings and various auxiliary equipment to fully utilize the flexibility on the demand side of the system, thus promoting renewable energy consumption. An operation optimization model of an integrated electricheating system is established in, 13 which takes the virtual energy storage characteristics of heating network on the network side into account, so as to effectively improve the energy efficiency of the system. The flexibility analysis of both energy storage and network are conducted in, 14 and in light of the flexibility quantification framework, effective countermeasures are provided for system flexibility enhancement by utilizing various flexible resources. Authors in 15 analyze the operational flexibility shortage of RIES and put forward the application of multi-energy conversion equipment to improve flexibility. A flexibility evaluation method based on the generalized heat storage model is designed in 16 to quantify the flexibility of DHNs in RIES, and the effectiveness verification of flexibility exploitation of DHNs is carried out. Nevertheless, these studies ignore the mechanism establishment from the perspective of the market. As a result, the application of FRP in the integrated energy market is rather blank, and the effect of FRP on improving the operational flexibility of RIES is not explored thoroughly.
Clearly, the market participants in RIES are microgrids (MGs) that integrate multi-energy coupling facilities. [17][18][19][20] In the context of the research tendency, an integrated energy system with multi-energy microgrids is studied as complex multi-agent optimization projects in. 17 A hybrid sharing framework of energy transactions between multiple microgrids is proposed to improve the flexibility of the thermoelectric integrated energy system. 18 Based on demand response, a distributed multi-agent optimization model is constructed in, 19 which effectively reduces the operating cost of the system. In addition to microgrids, the integrated demand response (IDR) of the users is a significant way to enhance the flexibility of RIES, so based on IDR, a Stackelberg game model for RIES optimal scheduling is proposed in 20 to balance the interests between integrated energy system operators and users. The optimal bidding model of MGs in the energy and FRP market is proposed in, 21 but the prices are modeled using stochastic programming rather than being cleared by the biddings. In, 22 a decision-making method is introduced, which reinforces the bidder to learn the feedback on the effect of their biddings on the clearing results, but the proposed approach is carried out for single bidder. Few studies in this field concentrate on the equilibrium problem when multiple MGs are bidding simultaneously. In, 23 the unreasonable effect that MGs act as the price taker is elaborated. Hence, it is crucial to study the day-ahead market equilibrium with multiple bidders under appropriate pricing mechanism for the establishment of well-functioning and coherent markets. 8,24 To the authors' best knowledge, there are the following research gaps in terms of the market-based method for operational flexibility enhancement of RIES: (i) Few studies focus on the joint mechanism of the energy and ancillary service markets for operational flexibility promotion in RIES. Moreover, the pricing interaction between the energy and the ancillary service markets lacks thorough analysis. (ii) Comprehensive leverage based on a market-based framework is absent for the flexible resources on the supply, network, and demand sides in RIES. In particular, the flexibility of DHNs on the network side is not fully exploited. (iii) Existing researches mainly regard the MGs as price takers instead of strategic bidders in the clearing mechanism of the regional-integrated energy market. However, this treatment fails to trade off the autonomy of MGs and the overall optimization of RIES.

| Contributions and paper organization
Inspired by the related work, this paper proposes a novel market-based approach to improve the operational flexibility of RIES. The main contributions of this study are threefold: (i) To enhance the operational flexibility of RIES, a joint market mechanism incorporating the ancillary FRP market to the energy markets is designed. The proposed mechanism can properly illustrate the bidding and clearing for electricity, heat, and FRP between multiple MGs and the RIES operator (RIESO). In this regard, the market equilibrium and pricing interaction are thoroughly analyzed. (ii) To form comprehensive leverage for the various flexible resources of MGs, the FRP delivery by the supply and demand sides is established, respectively. Additionally, a collaborative method for the joint FRP supply by CHP and DHNs is developed. This method equips the MGs with higher flexibility in the designed market and is capable to exploit the quantified and marketized flexibility of DHNs. (iii) To trade off the autonomy of MGs and the overall optimization of RIES, a hierarchical model is proposed to achieve the joint market equilibrium. The proposed model is conducive to optimizing the bidding strategies of MGs and the market clearing of RIESO. Furthermore, the realization of market equilibrium broadens the scope of the operation mode for the localized joint market.
The remainder of this paper is organized as follows: Section 2 introduces the problem of operational inflexibility in RIES. The mechanism of the joint energy and FRP market is designed in Section 3. Section 4 establishes the optimal bidding strategy for MGs with the flexibility exploitation of DHNs. The equilibrium model of the joint market is detailed in Section 5. The performance of the proposed approach is verified by case studies in Section 6. Finally, the main conclusions of this study are drawn in Section 7.

| PROBLEM DESCRIPTION
As the integration of MGs with renewable energy sources increases in RIES, the uncertainty in electrical power will confront the system with a greater challenge in operational flexibility. For PDNs and DHNs that are tightly coupled in RIES, the uncertainty influences both electrical and heating networks through the energy coupling nodes. Referring to the conception in, 25 the operational flexibility of RIES can be defined as the ability to utilize flexible resources to cope with the fluctuant and uncertain changes in load caused by the integration of MGs.
To maintain the balance between multi-energy supply and demand, RIES should realize the real-time net load following. For the electrical network in RIES, the net load represents the difference between load and renewable energy generation. The fluctuation and uncertainty of the net load curve are exaggerated as the increasing integration of MGs in RIES. As shown in Figure 1, the flexibility requirement of RIES consists of the requirement caused by fluctuation and uncertainty of the forecasted net load. The fluctuation of the forecasted net load means its variation between the adjacent time intervals and the uncertainty of the forecasted net load is the forecasted error under a certain confidence level.
When the aforementioned flexibility requirement fails to be satisfied, the operational inflexibility of RIES will emerge in different aspects. From the market perspective, the inflexibility performance includes the following aspects 25 : (i) limited access of renewable energy generation, thus resulting in the waste of market resources; (ii) balance violation between the supply and demand, thus resulting in an extremely high penalty and the increase in operational cost; (iii) rapid recovery from the balance violation, thus resulting in the temporal extreme fluctuation in market prices.
To exploit the flexible resources and mitigate the above inflexibility performance, it is necessary to conduct a localized market practice, where the market price signals are transparent and the flexible resources are properly rewarded, for operational flexibility enhancement in RIES.

| FRP market for flexibility enhancement
To deal with the fluctuation and uncertainty of the forecasted net load, FRP is introduced to the localized market, despite the fact that some of the existing ancillary services can balance the real-time change of the net load to a certain extent. There are three major distinctions between FRP and other ancillary products, which can be summarized as follows: (i) FRP is designed to meet the flexible ramping requirements in the operation of RIES, thus improving the operational flexibility, while the regulation service aims to keep the frequency stability and the reserve service is intended to deal with sudden blackout for operational safety. (ii) FRP is implemented in different time scales from other services and reserved for the following time interval, which lasts from 5 to 30 min, while the regulation service balances the generation and load simultaneously. (iii) FRP procures sufficient flexible ramp in the cooptimized market-clearing process with the most utilization of all the schedulable resources, while the regulation service pursues higher requirements for the providers at a higher price.
Compared with distribution power systems, the integration of MGs enables RIES to have more flexibility in resources on the supply and demand sides, which can be marketized to provide FRP. In MGs, some fast start-up units, such as controllable CHP units, batteries (BT), can provides FRP on the supply side. The IDR that employs the interruptible load (IL) and the alternative electric-heat load (AL) is also competent to provide FRP on the demand side. A marketized FRP provision on the supply and demand sides is described in detail as follows.
On the supply side, the ability of a unit delivering FRP is determined by its ramping and generating capacities, as well as its current operational state. The ramping and generating capacities are inherent physical characteristics of the unit, while the current operating state determines its actual FRP capacity in multi-interval market biddings. The FRP capacity provided by a unit in different operating states is illustrated in Figure 2. The unit is in the shutoff state at t 0 , which leads to the sole provision of upward FRP because its FRP deliverability is determined by the operating state at t 1 and its upward ramping capacity. As for the FRP deliverability at t 1 , t 2 , and t 3 of the unit, it is adequate to provide both upward and downward FRP for its appropriate operating state. The unit can only supply downward FRP at t 4 for the capacity constraint binds when it operates at its maximum output.
Due to the natural fluctuation and uncertainty, it is generally deemed that the multi-energy loads on the demand side, especially the electric load, are incompetent to provide FRP. The IDR technology, on the other hand, allows the load to be controllable enough to supply FRP. To be specific, IL is able to provide upward FRP by reserving the capacity, and AL is able to provide upward and downward FRP by switching to the heat supply mode. The concrete FRP provision by the IDR is shown in Figure 3.
The forecasted electric load in Figure 3 consists of the actual value and the confidence interval under a specific confidence level. The available FRP capacity on the demand side is dependent on the IDR's equivalent ramping capacity and actual load level, and a similar relationship is also suitable for the supply side. Considering that the uncertainty might lead to a failure of providing the affirmative FRP capacity, the ramping capacity could be conservatively defined. The available FRP capacity at t 1 is determined by the lower bound of the confidence interval at t 1 , the actual IDR load at t 1 , and the lower bound of IDR load profile at t 2 .

| Joint energy and FRP market mechanism
The established joint operation market is depicted in Figure 4. There are three sub-markets in the joint market, namely the distribution electricity market, the district heating market, and the FRP market, respectively. The joint market participants are MGs and RIESO. Besides that, the natural gas market and the electricity wholesale market supply RIES with energies at prices independent of the joint market.
In terms of the clearing mechanism, the ways that RIESO clears the energy and FRP markets are different. The supply and demand functions are established in the energy market to maintain the balance. The electricity and heat balance constraints in each interval are bounded as The marginal clearing prices are calculated based on the Lagrange multiplier of (1) and (2). By analogy with the definition of nodal electric price, the nodal energy price is defined as the system marginal cost for meeting the increasing unit demand. 5 Thus, the implementation of the nodal electric price theory can obtain the energy price, which is a market signal that reflects the marginal cost accurately and stimulate MGs to optimize the bidding strategies.
In the FRP market, the upward and downward FRP supplied are under the constraint that the sum of the availability and surplus should be no less than the requirement, which is expressed as It is worth noting that the reward for FRP is the compensation of the capacity reserved for the flexible ramping that cannot be deployed to provide energies, so MGs do not bid for the prices of FRP. RIESO sets the payment for FRP based on the opportunity cost of MGs. When the resources fail to meet the flexibility requirement, the price will be cleared by utilizing the demand curve.
According to the clearing mechanism of three submarkets, the operation mode of the joint market is described as follows: MGs, as bidders, optimize their bidding variables in the corresponding market they choose to participate in based on their qualifications. RIESO optimizes the clearing results based on the market mechanism and all the received bids.
Therefore, the following two sections are arranged to present the optimal bidding strategy of MGs, the optimal clearing scheme of RIESO, and the holistic equilibrium model under the joint market mechanism, respectively.

| Flexibility quantification of DHNs
By changing the supply temperatures of heat resources and mass flows of working fluid in pipelines, DHNs can realize their heat storage capability, which is competent to enhance operational flexibility. In this section, the collaborative operation of CHP and DHNs is utilized to quantify and marketize the improved flexibility that DHNs contribute.
To quantify the flexibility of DHNs, specific indices are selected according to the FRP provision method in Section 3.1, namely the equivalent input heat capacity and ramping capacity.
The operating state that both the temperature and mass flows are fixed is set as benchmark state (BS) mode. In this mode, the supply temperature and supply mass flow rate of heat source nodes i s are fixed. Therefore, the nominal input heat power P h,nom i s (t) at i s can be determined by the heat power injected at t. In compliance with the safe operation constraints of RIES, the limits of the input heat power P h i s (t) at i s under active control mode are represented as As passive heat storage, the ramping capacity of DHNs is restricted by the maximum/minimum input heat power and P h,nom i s (t), which is expressed as In a typical market framework in RIES, RIESO is both the market clearer and the physical network operator. However, the market mechanism places a clear stipulation that the clearer cannot conduct biddings. The market value of the quantified flexibility cannot be realized. Hence, a collaborative method is conducted to marketize the flexibility of DHNs. By appending the constraints of DHNs' flexibility indices in (4) and (5), the operating domain of the CHP units is expanded. The outputs of the CHP units are P ′e,m CHP (t) and P ′h,m CHP (t), as depicted in Figure 5. The expanded operating domain enables MGs to be more competitive in market bidding with higher operational flexibility, which leads to a change in the clearing result for each MG dispatched by RIESO.
It is obvious that DHNs should be rewarded for contributing flexibility. The Shapley value method 26,27 in the cooperative game is adopted to allot the increased revenue portion. Integrating the characteristics of individual, alliance, and global rationality, the Shapley value is capable of reasonably quantifying the contribution that MGs and DHNs make to the increased revenue. The reallocated revenue ensures the operational flexibility of DHNs well marketized.
The allocations are formulated based on revenues that MGs and DHNs gain through the individual and collaborative supply. The flowchart of the collaborative FRP supply method and revenue reallocations is shown in Figure 6. This flexibility exploitation not only diversifies the flexible resources that provide FRP in RIES but also extends the range of operational flexibility that MGs can provide.

| Operating principle of MGs
MGs maximize their revenue according to the established operating principles, which should be formulated first in light of the joint market mechanism and MGs' structure. The structure of MGs is composed of the production, demand, and storage of multiple energy. The multi-energy production can be realized by CHP units, wind turbines (WT), photovoltaic (PV), gas boiler (GB), and so on. The produced energy can meet the electric and heat load formed by the rigid load, IL, and AL. The electric energy is also stored by BT. In addition, to analyze the heat storage characteristics of DHNs accurately, other types of heat storage are not considered.
To realize the optimal bidding in the joint market, MGs can formulate the following operation strategy according to their specific structure. The electric and heat load in MGs should be preferentially supplied. If the produced electricity exceeds the consumed energy, the electricity produced by WT and PV is first to meet the load demand. The electricity surplus of the renewable energy sources and the electricity generated by the CHP unit are available for BT charging or bidding in the joint market. On the contrary, if the produced electricity is insufficient, MGs should manage to discharge BT or purchase electricity from the joint market to meet the demand. Analogously, CHP units are employed to meet the heat load, and GBs serve as supplementary equipment. The surplus energy is available for trading in the joint market. MGs make use of the heat stored in DHNs or purchase heat from the joint market when the produced heat is insufficient.
The capacities of the traded electricity, heat, and FRP in the m th MG are determined as The first and third constraints in (6) are the aforementioned operation strategy of MGs. The second, fourth, and fifth constraints in (6) are IDR constraints. Moreover, the last two constraints are the FRP provision constraints of the m th MG utilizing the production, demand, and storage equipment.
On the basis of the original electric output limits of the CHP unit, the reserved upward/downward FRPs are added to reform the according constraints in (7), respectively. The capacity constraints of CHP units considering FRP are represented as The same changes are made to capacity constraints for BT and IDR in (8) and (9). The capacity constraints of BT considering FRP are described as The reserved upward/downward FRPs also make changes to the ramping constraints, which are expressed as The sum of ramping rate and reserved FRP capacity of the equipment between two adjacent time intervals is restricted by its maximum upward/downward ramping rate. Normally, P RU ,m CHPmax and P RD,m CHPmax are 1% of CHP's maximum capacity per minute because it can rapidly change the output. For BT's high flexibility, its ramping rate can be taken as its maximum capacity per time interval. Additionally, for the demand side, the ramping rate of AL is determined by the installed equipment. The ramping rate of IL is acquired by the consumption satisfaction coefficient s IL of the electric consumer, which is represented as

| Optimal bidding strategy formulation
From the perspective of energy trading in the joint market, MGs can conduct a bidirectional energy exchange with RIES. Corresponding to the behavior of selling and purchasing, the provisioning function and the inverse demand function of MGs can be distinguished. Based on the two functions, the optimal bidding strategy of MGs is established.

| Bidding strategy for selling electricity and heat
According to the provisioning function of MGs, the bidding strategy for selling energy can be expressed as where k e,m (t) and k h,m (t) are the bidding coefficients of electricity and heat to indicate MGs' strategic behavior, which will be optimized through the bidding decision-making. The variables should not exceed the following limits: When MGs supply energy to the market, the bidding strategy is the affine function of the marginal production cost function of the equipment. The marginal production cost function varies with different equipment. For CHP units, the marginal production costs of electricity and heat are calculated as The marginal production cost function of other equipment in MGs can be derived according to the cost functions in. 5

| Bidding strategy for purchasing electricity and heat
Based on the inverse demand function of MGs, the bidding strategy for purchasing energy is described as The bidding strategy is the integral of the inverse demand function, which reflects the relationship between the acceptable energy price of MGs and the consumed energy.

| Bidding strategy model
The m th MG strategically bids in the joint market to maximize the revenue in the market. The optimal bidding strategy model of MGs can thus be summarized as where u e.m S (t) and u e.m B (t) indicate the selling and purchasing status of electric energy trading, which are determined by the operating principles in Section 4.2 and constraints in (6). The revenue/cost that MGs gain from heat trading can be confirmed in the same manner. MGs also make a profit by providing the upward/downward FRP. The objective function in (20) is the implicit function of k e, m (t) and k h, m (t), for the optimal strategy problem is constrained by the internal clearing problem of which the clearing result differs as the strategic behavior changes. The technological constraints (7)-(15) are the generation constraints and the ramping capacity constraints of various resources in the MGs considering the FRP provision. The constraints ensure that the bidding capacities offered to RIESO are within reasonable limits. To maintain market order, bidding coefficients offered by the MGs should be restricted under the range stipulated in (17).

| EQUILIBRIUM MODEL OF THE JOINT MARKET
In the context of the proposed mechanism of the joint energy and FRP market, multiple MGs participate in the market to bid according to (19) in Section 4.3, and the market clearing is conducted by RIESO in accordance with the strategy set of all MGs. In this section, the optimal clearing scheme of RIESO is first introduced. Then, based on the optimal bidding strategy and the optimal clearing scheme, a hierarchical model is established to achieve the joint market equilibrium.

| Objective function
The objective of the RIESO's clearing follows the rule to minimize the overall cost of RIES, which is represented as As shown in (21), the first line is the cost of purchasing electricity and heat. The second line is the cost of FRP procurement from MGs. The third line is the cost of purchasing electricity from the wholesale market and the penalty cost of renewable energy curtailment.

| Constraints
The optimal clearing scheme is performed subject to the following constraints. The clearing scheme result includes the prices and the corresponding allocated capacities of electricity, heat, and FRP. The allocated capacities are restricted within the available bidding range offered by the MGs. The price of electricity, heat, and FRP is calculated in the optimal clearing scheme model by adopting the path-tracking interior point method. By introducing slack variables, the original clearing problem can be transferred to a Lagrange function, of which the Lagrange multipliers corresponding to the demand-supply balance constraints are the prices of electricity, heat, and FRP. The thermal dynamics model of the heat load considering thermal inertia can be described as The nodal flow balance in DHNs is given as The various temperature and mass flow limits constraints during the DHNs operation are expressed as in (28) and (29), respectively.
Based on the first law of thermodynamics, the temperature mixing in the nodes of supply and return networks in DHNs can be calculated as

| Hierarchical structure of the equilibrium model
In light of the optimal bidding strategy of MGs and the optimal clearing scheme of RIESO, an equilibrium model for the joint market is established, which has a hierarchical structure, as presented in Figure 7.
There are two layers in the equilibrium model. After receiving information about energy and FRP requests from the internal layer, all MGs in the external layer conduct decision-making processes in which they perform strategic biddings. The optimal bidding strategies are made by weighing the capacity allocated to energy and FRP to maximize the revenue according to Figure 6. In the internal layer, RIESO conducts a market-clearing process, from which the optimal clearing result is sent to the external layer. The objection of the clearing process is to minimize the overall energy cost of RIES according to the market mechanism. It is evident that the model is , i, j ∈ I h characterized by an interaction between the external and internal layers. Each MG tends to change the bidding strategy to attain a higher revenue, and the clearing result varies accordingly. When all bidding strategies no longer change and the optimal clearing result is obtained, the market equilibrium is deemed to be achieved. The proposed model reflects the behaviors of entire market participants including MGs and RIESO. Once the market equilibrium is achieved, all the participants can reach their optimal revenue in this recursive process.

| Equilibrium model solution
For solving such a hierarchical model, one of the existing methods is to convert the bi-layer model into a singlelayer model by employing the Karush-Kuhn-Tucker (KKT) condition equivalent to the optimal clearing problem. 22 However, in the practical market, the bidders' decision-making processes are rather complex, and the mentioned single-layer reformation requires the optimal clearing function to be convex. Therefore, in accordance with the characteristics of this bi-layer model, it should still be solved hierarchically, and equilibrium is acquired recursively.
The solution process of the equilibrium model is illustrated in Figure 8. For the optimal bidding in the external layer, the Q-learning algorithm 28 is adopted to make MGs accumulate experience in multiple interactive decisions and gradually converge to the optimal decision for profit maximization. For the optimal clearing in the internal layer, the path-tracking interior point method is adopted to minimize the overall cost of RIES.

| System configuration
In order to verify the effectiveness of the proposed approach, a RIES composed of the IEEE 33-bus PDN and the 32-bus DHN of Barry Island is adopted, as depicted in Figure 9. The detailed network parameters can be obtained in. 5 The proposed joint market is built based on this system, and five MGs are taken into account. The energy production, consumption, and storage equipment in each MG are displayed in Table 1

Biddings
Schedules Prices electric, and heat load profiles are collected from, 29,30 which are modified based on pragmatic considerations. The net load uncertainty is estimated according to the forecasted probability distribution in. 10 The FRP requirement is obtained through estimated net load fluctuation and uncertainty. There are 96 bidding intervals in the joint market within a day. That is, each bidding interval is 15 min. The parameters of the Q-learning algorithm for the external layer are obtained in. 28 The state and action spaces of each MG are discrete so that the algorithm can be applied. For the bidding variables, the price and the bidding coefficient are discretized in 100 integer values. Hence, each MG employs 96 look-up tables, the size of which is determined by the discrete state and action spaces, with the Q-values of state-action pairs being stored and updated every 15 min. The simulation study is performed on MATLAB R2016a in a laptop with Intel (R) Core (TM) I5-6500H CPU and 16G RAM.

| Analysis of joint market equilibrium
The algorithms shown in Figure 8 are applied to achieve the joint market equilibrium. When the Q-values of each MG's look-up tables converge, equilibrium is reached because no market participant's bidding strategy changes. Consequently, the clearing results of RIESO are determined.
The iteration processes of RIES and MGs are provided in Figure 10. It takes 104 learning episodes for all the look-up tables to converge. As the iterative process progressed, RIESO's operating costs gradually reduces to the minimum level, with a total computing time of 561.87 s.
The spatial clearing price variation of the electricity market is depicted in Figure 11. The clearing price at bus 32 is relatively low because the CHP 1 unit that integrates into this bus has lower C e,m S (t). Furthermore, as transmission loss increases along the radiant network, the clearing price increases spatially. The intervals when the price of electric energy is higher also yield a consistent trend with the intervals when the load peak arises in terms of temporal variation.
When the joint market achieves equilibrium, the provisions of the electric energy, heat energy, and FRP are demonstrated in Figure 12. The results in the 39th-48th bidding intervals (period 1) and the 60th-75th bidding intervals (period 2) are chosen for analysis to verify the optimal clearing results conducted by the joint operation market mechanism. During these two typical periods, RIES has high demands for FRP, as shown in Figure 12C, due to the obvious fluctuation of renewable Storage equipment --BT 3 BT 4 energy generation and electrical load profile within the system. Considering the IDR technology and the DHN's heat storage characteristic, CHP 2 is able to be shut down for 15 bidding intervals to increase the consumption of renewable energy resources, rather than being kept online to ensure the supply of heat load. RIESO also benefits from it to reduce the overall energy cost for the marginal cost of CHP 2 is relatively high. In period 1, the requirement for downward FRP increases. Both CHP 1 and CHP 2 run in the expanded areas in operating domains, realizing the flexibility exploitation of the DHN. In period 2, the system's demand for the upward FRP rises, CHP 2 starts up to provide FRP. However, after the 71st bidding interval, CHP 2 keeps running at the boundary of the operating domain to maintain its heat output, making it impossible to provide upward FRP. The same situation occurs in CHP 1 at the 75th bidding interval, and thus IDR in MG 5 wins more FRP provision at the same time.
As depicted in Figure 12A,B, since the thermoelectric ratio of each CHP unit is adjustable within its operating domain, in most bidding intervals, MGs manage to maintain the heat demand-supply balance by optimizing multi-energy generation. In the 76th-84th bidding intervals, both the electric and heat loads are relatively high. To maintain the heat output, the CHP units are not able to increase the power output. Therefore, more electricity F I G U R E 1 1 Spatial and temporal variation of the electric price is purchased from the upstream system to maintain the balance.
The revenue of each bidder in the bidding when the market equilibrium is reached is displayed in Table 2. CHP 1 is not cleared as the marginal unit because of its lower marginal generation cost. Therefore, MG 1 to which CHP 1 belongs wins the higher electric and heat power in the energy market and thus yields the higher revenue. Due to the high marginal generation cost of CHP 2, it will be cleared as the marginal unit when it is online. However, CHP 2's fast start-up speed and ramping characteristics make MG 2 inclined to participate in the FRP market to increase revenue. Compared to the CHP units, MG 3, MG 4, and MG 5 have relatively strong competitiveness in the FRP market because they utilize BT or IDR, respectively, which can provide FRP with lower opportunity costs.

| Analysis of operational flexibility improvement
Four scenarios are used to validate the proposed model's improved operational flexibility under the joint market mechanism. The comparative analysis is conducted from three aspects: renewable energy curtailment within the system, RIES overall cost, and average cleared energy price in the bidding. Scenario 1: Serving as the benchmark, the FRP market is not incorporated into the joint market, and the DHN operates in the BS mode.
Scenario 2: The FRP market is incorporated compared to the benchmark. The IDR and BT are excluded from the joint market.
Scenario 3: Based on scenario 2, IDR and BT are further implemented to provide FRP.
Scenario 4: Based on scenario 3, the DHN operates in active control mode, and the revenue is reallocated using the Shapley value method.
i. Comparison of the renewable energy curtailment The renewable energy curtailment under the four scenarios is shown in Table 3. In contrast to Scenario 1, renewable energy curtailment fell by 41.17% in Scenario 2. Because the FRP market has been incorporated into the joint market, the mechanism encourages bidders to provide FRP and allows the system to reserve capacity and consume more renewable energy. Compared with Scenario 2, the renewable energy curtailment in Scenarios 3 and 4 further reduce 13.05% and 21.21%, respectively. Due to IDR and BT's participation in the FRP market, load fluctuation flattens to some extent, reducing RIES demand for FRP and increasing renewable energy consumption. While the DHN cooperates with the CHP unit in Scenario 4, CHP's expanded operation domain increases the flexibility that the supply side can provide.
ii. Comparison of the overall energy cost of RIES The overall energy cost of RIES under the proposed scenarios is listed in Table 4. The overall energy cost in Scenario 2 decreases by 1.99% compared with Scenario 1, and that in Scenarios 3 and 4 decreases by 6.34% and 10.13%, respectively. This is because the average energy prices in both electric and heat markets decrease compared with the benchmark. As a result, RIESO's cost to purchase electric and heat energy decreases, and the electric energy purchased from the wholesale market also decreases because more renewable energy resources are consumed. Besides, in Scenario 3, the total energy cost further decreases because the utilization of IDR and BT reduces the FRP procuring cost. Furthermore, in Scenario 4, the CHP-DHN facility's reallocation of the collaborative supply method can reduce RIES overall cost ulteriorly.
iii. Comparison of average energy prices The fluctuations of the average electric and heat energy price in the bidding under the four scenarios are illustrated in Figure 13. The energy price fluctuations in the energy markets reflect the change of energy load. The average heat price indicates a gentler fluctuation, so the improved flexibility has a greater impact on the electric price.
In Figure 13A, there are 7 price spikes under Scenario 1. This is due to the lack of flexibility of RIES in Scenario 1. When RIES's requirement for downward FRP rises, renewable energy sources cannot be fully consumed, and the curtailment penalty raises the average electric price. When RIES's demand for upward FRP increases, the generation does not keep up with the load profile. Although electricity purchased from the wholesale market can save RIESO from the loss of load penalty, the higher wholesale market price still raises the overall cost.
Once the FRP mechanism is incorporated, the price spikes become less frequent. Due to the optimized clearing result by RIESO, both the renewable energy curtailment and electricity purchased from the wholesale market decrease, lowering the average electric price. Meanwhile, because of the optimized unit commitment, the CHP 2 unit with a higher marginal generation cost is shut down. The average electric and heat prices decrease along with it.

| Analysis of the collaborative FRP supply method
To validate the flexibility improvement by the proposed collaborative FRP supply method, the expanded operating  domain of the CHP-DHN facility is shown in Figure 14.
During the bidding intervals when the heat load is rather high, the CHP units will operate at the boundary of the operating domain when the flexibility of DHN is not considered, and thus the electric power output is fixed. As a result, the flexibility of the CHP unit is not always adequate. The equivalent operating state can exceed the original operating domain when the collaborative FRP supply method is applied, and the CHP-DHN facility can offer extra FRP. As a result, the quantified flexibility of DHN is fully utilized. Due to the lower generation cost, the operating points of CHP 1 are compact in the area where there are high power and heat outputs. Compared with CHP 1, CHP 2 is inclined to provide FRP for more revenue, so the operating points of it are more scattered.

| Sensitivity analysis
There are influencing factors in the proposed model that may have an impact on operational flexibility, such as the forecasted net load confidence level and the heat load percentage, for which a sensitivity analysis is undertaken.
i. Analysis of the confidence level The confidence level of forecast net load reflects the RIES's requirement for flexibility to cope with uncertainty. Based on Scenario 3, a sensitivity analysis of the overall cost and renewable energy curtailment is undertaken in Figure 15 with the confidence level increasing from 70% to 100% and a step size of 5% to examine its impact on operational flexibility.
The overall cost and renewable energy curtailment decrease at first as the confidence level rises, as shown in Figure 15. The demand for FRP increases as the confidence level rises, and the RIES is more constrained by the FRP requirement. As a result, the overall cost increases. It can be concluded that it is critical to select an appropriate confidence level so that the FRP requirement can be determined accordingly.
ii. Analysis of the heat load percentage The heat load is set to increase from 90% to 130% of the original heat load under Scenario 1 with a step size of 10% to analyze the impact of heat load on the operational flexibility. When the heat load is smaller than the original load, as shown in Figure 16, the extended operating domain allows CHP units to operate at more cost-effective states, whereas renewable energy curtailment only shows a minor change because the CHP units are not operating at the boundaries. When the heat load increases toward 130% of the original load, the CHP units cannot lower the power output due to the heat energy balance. As a result of the increased demand and uneconomical operating states, renewable energy curtailment increases, and the overall cost increases dramatically.

| CONCLUSIONS
In this paper, the authors explore the joint energy and FRP market mechanism and propose a hierarchical equilibrium model to fully utilize FRP for operational flexibility enhancement. The comprehensive leverage is realized by making full use of the flexible resources in MGs and DHNs. The conclusions can be summarized through the case studies: (i) The incorporation of FRP into the joint market mechanism can effectively alleviate the negative effects of net load fluctuation and uncertainty, thus enhancing the operational flexibility of RIES. (ii) The collaborative FRP supply method is able to availably quantify and marketize the flexibility of DHNs, and the revenue reallocation method further reduces the overall cost of RIES. (iii) The established hierarchical model is competent to achieve market equilibrium with efficiency. The clearing results show that the joint market mechanism can maximize the synergetic effect of multiple resources through economic incentives.
Future works may include incorporating other types of energy resources to realize a more generalized integrated energy market. Moreover, advanced algorithms are worth investigation to better simulate the bidders' decisionmaking with a continuous action space.