Potential for waste heat utilization of hot‐water‐cooled data centers: A case study

The electric energy demand of data centers in Germany has grown rapidly from 10.5 TWh/a in 2010 to 13.2 TWh/a in 2017, an average of 25% of which are used to fulfill the data centers' cooling demand. In order to increase its energy efficiency, TU Darmstadt applies a new cooling concept in the next generation of its high‐performance computing data center “Lichtenberg II.” Instead of the current air‐cooled servers with water‐cooled rear doors at 17‐24°C, the new data center will be equipped with direct hot‐water cooling for the high‐performance computer, supplying heat at a temperature of 45°C. The high‐temperature waste heat is used for heating purposes on the university's campus Lichtwiese. For waste heat utilization, two concepts are presented, either integrating the heat in the return line of the district heating network or using it locally in buildings located near the data center. Reductions in CO2 emission and annuity are generated both by decreased compression cooling demand for the data center and by decreased heat generation due to waste heat utilization. Depending on the scenario, a total of 20%‐50% of the waste heat emitted by the high‐performance computer can be used for heating purposes, while the remaining heat is dissipated efficiently via free cooling without additional energy demand for mechanical chillers. CO2 emission can be decreased by up to 720 tCO2/a , representing a reduction of about 4% of the total emission at campus Lichtwiese. TU Darmstadt is currently implementing the waste heat integration into its district heating network and will benefit from this concept starting in 2020.

is expected to keep growing to reach up to 13% in 2030. 4 Due to its high energy consumption, the data center industry emits about as much CO 2 as the airline industry. 3 Data centers operate at high and growing energy densities up to 100 times higher than for office accommodations, 5 and the server electricity demand is almost entirely transformed into heat. Therefore, cooling represents about 40% of the total energy demand of a data center. 6 Reducing the energy demand for cooling is therefore a high priority for data center operators.
This paper shows a new approach to improve the energy efficiency of data centers, combining hot-water server cooling with waste heat utilization outside the data center building. For the purpose of this study, the term high-performance computer (HPC) is used to describe the heat-emitting server racks themselves; the term data center describes the facility as a whole, including all auxiliary installations such as cooling and the building.

| Cooling technologies
Higher overall data center energy demands have led to significantly increased energy densities in data center facilities, reaching up to 35 kW per rack 7 compared to about 1 kW per rack in the 1980s. 8 This development makes efficient cooling for data centers an ever more important and challenging task. 9 While the American Society of Heating, Refrigerating and Air-Conditioning Engineers (ASHRAE) recommends a cooling temperature range between 18°C and 27°C for air-cooled data centers, 10 most publications consider temperatures at the processor level of up to 85°C as reasonable. [11][12][13] This makes it clear that higher cooling temperatures for data centers would be possible if the heat transfer between the processors and the cooling unit was improved.

| Air cooling
Currently, most data centers use air-based cooling solutions to remove the heat from its IT equipment via computer room air conditioners (CRACs) or computer room air handlers (CRAHs). 14,15 Modern air-cooled data centers are divided into hot and cold aisles, and various studies have been conducted on improving the air flow within the data center, eliminating hot air recirculation or cold air bypasses. [16][17][18][19] Other factors impacting the performance of data center cooling are ceiling height, where heat traps may occur, raised floor height and air flow direction. 20 A recent review article additionally identifies leakages, over-or underprovisioned air supply as well as a nonuniformity of airflow and air temperature as major inefficiencies in airflow management. 21 Many data centers are supplied with a constant air volume flow based on the design heat load, resulting in high electricity demands of circulation fans. 22 Variable air volume flow can decrease this demand.
An important measure to improve the data centers' cooling performance is an optimized design of the room layout. 21,23,24 Traditionally, equipment is split among different racks according to the maximum allowed power and heat capacity of each rack. 25 To achieve minimal thermal impact of increased power densities, Siriwardana et al 26 couple CFD simulation with particle swarm optimization in order to determine an optimal data center layout. One widely accepted and applied measure is the containment of the hot or the cold aisle in order to achieve a better separation of airflows in the different areas of the data center. 27,28 Another approach to increase the energy efficiency of air-cooled racks is rear door water cooling, where the hot air passing through the rack is cooled down again using cold water running through the rack's back door. 29

| Free cooling
In cold climate areas, free cooling using cold ambient air is becoming more common, although many data center operators fear contamination possibly introduced by allowing ambient air inside their facility. 30 An overview of free cooling strategies can be found in Oró et al. 5

| Water cooling
In order to cope with the ever-increasing data center energy density, new cooling technologies such as two-phase or water cooling will have to become more important in the future. 31 Direct hot-water cooling profits from higher heat capacity and less thermal resistance than air cooling. 9 Based on these advantages, water-cooled data centers are able to deal with heat densities at least 10 times higher than air-cooled ones, 9 making important savings in cooling energy demand possible. 32 At the same time, water cooling allows for high cooling inlet temperatures of up to 60°C 33 and opens new opportunities for waste heat utilization, since the temperature of the available heat is the key factor for such an application. 15 Ellsworth et al 34 note that water cooling increases not only the energy efficiency, but also the processor performance. Zimmermann et al 35 demonstrate the feasibility of water cooling in a prototype hot-water-cooled data center. Chi et al 36 present the case of a fully immersed liquid-cooled data center, reaching a cooling system temperature of about 50°C. Due to the temperature increase possible with water cooling, operating times of free cooling can also be increased in a water-cooled data center. 37

| Waste heat utilization
Utilizing waste heat means to recycle the heat emitted by the computing process as a valuable resource instead of simply discarding it into the environment in the most efficient way. Possible waste heat recovery technologies applicable to data centers depend on the temperature available and include the use for local space or hot-water heating at low temperatures (30-40°C), district heating supply (50-60°C), or the supply of cooling energy via absorption chillers (70-90°C). 9,15,38 Liu et al 39 propose placing small cloud servers in individual homes to be used as a heating alternative. If the waste heat temperature of the data center is not high enough for direct use, its temperature can be upgraded using a heat pump. 15 For an efficient operation of the heat pump, the temperature difference between the waste heat temperature and the desired output temperature must be as low as possible. Barriers preventing the utilization of data center waste heat are listed in Ref. 40 and include unfitting temperature levels, lack of heat demand, high investment costs, differing interests of data center operators, and district heating companies as well as inadequate business models.
Several examples for data center waste heat integration in district heating systems exist, mainly in Nordic countries. A recent study presents an overview of different waste heat utilization projects from large data centers in Denmark, Sweden, and Finland, 40 making it clear that energy efficiency is a pressing issue for data center operators nowadays. In Stockholm, the local district heating company Stockholm Exergi has launched a program called Open District Heating, offering data center operators to buy their waste heat at a price depending on the respective outdoor temperature. 41 Wahlroos et al 33 carried out a study on the impact of data center waste heat integration in the district heating system of the city of Espoo, Finland. They conclude that waste heat utilization in district heating is favorable even in large quantities, as operational costs can be decreased considerably. In 2018, the Scandinavian telecommunications company Telia opened a 24 MW data center in Helsinki, Finland, supplying 200 GWh/a of heat to the nearby city of Espoo. 42 Already in 2014, the Russian Internet company Yandex set up a data center in Mäntsälä, Finland. The current power demand of this facility is 10 MW of which one third is being utilized in the local district heating system. In the future, this demand is expected to grow up to 40 MW and a utilization rate of 50% of the available waste heat. 43 In this facility, the cooling air heats up water to 30-45°C, which is then boosted to 55-60°C by a heat pump in order to become useful in the nearby district heating network. 44 In Braunschweig, Germany, the utility company Veolia integrates the heat from a data center in a low-temperature district heating network supplying heat to a recently developed residential and commercial area with 600 homes. 45 Examples for waste heat utilization in a single building are the data center of Eurotheum in Frankfurt am Main, Germany, which supplies office spaces and a hotel within the building 46,47 and Wallotstraße in Dresden, Germany, where 56 residential units in an apartment building are supplied by 20 racks with a total thermal power of 30 kW. 48

| Unique features of this paper
All known implementation projects regarding waste heat integration in district heating have in common that they still rely on traditional air cooling for the data center itself, resulting in rather low waste heat temperatures and a high-temperature difference between waste heat and district heating that has to be overcome by a heat pump. Also in the case of an in-building application, low temperatures represent a constraint for the waste heat utilization. In this article, we present a case study for data center waste heat utilization combined with hot-water server cooling, making it possible to extract the heat at higher temperatures than in the case of traditional air cooling. This approach yields high efficiencies for the heat pump in the case of a waste heat utilization in district heating and makes it possible to supply waste heat without a heat pump in the case of a single building application when combining the waste heat utilization with the installation of a low-temperature (LT) heating system. It is shown that this concept works even for existing buildings with low energetic standards. Both concepts are compared to each other based on the amount of utilized waste heat as well as possible CO 2 emission savings and economic performance in terms of overall annuity. The first scenario, a waste heat integration into the district heating network, was selected as an implementation project and will go into operation at TU Darmstadt in 2020.

| Background for the case study at TU Darmstadt
In accordance with the German national climate protection goals, TU Darmstadt has resolved to decrease its specific CO 2 emission by 80% compared to the 1990 level until 2050. These goals will only be reached if ambitious measures to improve the performance of the energy system are being implemented.
One of these measures is the integration of the heat generated by the university-owned HPC data center "Lichtenberg II" into the university's heat supply. The current generation of the HPC was installed in the years 2013-2015 and uses air-cooling and rear door water-cooled heat exchangers supplied via compression chillers with cooling water at a temperature between 17°C and 24°C. Additionally, free cooling via a hybrid cooler is available in times when outdoor temperatures are low enough to allow for free cooling.
In 2020/2021, a new generation of the HPC will be installed using direct water cooling at the server level providing a water output temperature of 45°C. The installation process of the new HPC servers will be carried out in two stages. The electric power demand of the first stage will be about 400 kW, which is considered to be constant throughout the entire year. The electric input power is almost entirely converted into heat, of which 90%, that is, 360 kW, can be recovered by means of an on-server water cooling unit. The rest of the heat goes into the server room and continues to be removed by the current air-cooling system. The hot-water alternative for server cooling generates two advantages: On the one hand, the hot-water waste heat can be utilized for heating purposes, either in the context of the campus' district heating network or locally in the adjacent buildings. On the other hand, compression cooling demand is decreased significantly, and the servers are cooled either by the building heat demand or via free cooling. Free cooling is possible all year round when the heat is available at 45°C. When calculating the ecologic and economic savings generated by hot-water cooling and waste heat utilization, we consider a reduction in compression cooling demand by 360 kWth as one of the advantages of this setup. To calculate the resulting energy savings, we take into account that the data center runs all year round (8760 h/a).
The data center is located at TU Darmstadt's campus "Lichtwiese," a typical university campus on the outskirts of Darmstadt built since the 1960s. Nowadays, it comprises 40 buildings and a total net floor area of 150 000 m 2 for different purposes such as lecture halls, offices, and laboratories. Most buildings were constructed in the 1960s and 1970s, but due to recent construction activities in the last decade, also new buildings with higher energy efficiency and lower temperature demands can be found. To supply its buildings with heat, cooling, and electric energy, a heat and power station is located on campus including three combined heat and power (CHP) plants with a total capacity of 7.25 MW el and 7 MW th , as well as six heat-only boilers (9.3 MW th each). 49 Heat is distributed on campus Lichtwiese and to TU Darmstadt's other sites via a district heating network. The annual average district heating supply temperature for campus Lichtwiese is 88°C, the annual average return temperature 58°C. For cooling purposes, a district cooling network and a 1 MW th absorption chiller were installed in 2017 and 2018. At the site of the data center and in the chemistry subdistrict, compression chillers (about 3 MW th ) are connected to the system. The supply temperature in the district cooling network is 6°C, the return temperature 12°C. All infrastructures for generation and distribution of heat, cooling, and electric energy are owned by the university but operated by an external consortium in the context of a contracting agreement. Figure 1 shows the generation and distribution infrastructure at TU Darmstadt Lichtwiese.

| METHODOLOGY
In this case study, two scenarios for the utilization of HPC waste heat are being compared to a reference scenario. Scenario 1 investigates the integration of the heat into the adjacent district heating network. Due to the high temperatures currently present in the district heating network, a direct use of the waste heat is not feasible, but its temperature has to be upgraded using a heat pump. In order to keep the temperature gradient between the output temperature of the HPC and the input temperature into the district heating network as low as possible, the heat will be integrated in the return flow of the district heating network. In scenario 2, the waste heat is not used in the district heating network on campus level but serves to supply heat to the civil engineering experimental halls located near the data center. Even though the considered buildings have a low energetic standard, they can be supplied with low heating temperatures after the installation of surface heating systems. Thereby, the heat can be used directly, and no heat pump is needed in this scenario.

| Reductions in CO 2 emission and amount of usable waste heat
To be able to quantify the potential reduction in CO 2 emission through the HPC waste heat utilization, the emission savings generated by each of the different scenarios compared to the low-temperature air-cooling alternative are calculated and linked to the campus' total CO 2 emission. For electricity, the German 2018 grid electricity CO 2 emission factor of 0.472 t CO 2 ∕MWh 50 is taken into account; for gas, a CO 2 emission factor of 0.202 t CO 2 ∕MWh 51 is considered.
Both a reduction in electricity demand for compression cooling and a reduction in lower heat demand from boilers result in CO 2 emission savings. Considering that the temperatures in the hot-water cooling system are above outdoor temperatures all year round, compression cooling can be eliminated and heat can be transferred to the environment via free cooling in case the waste heat utilization is not in operation. The reduction in compression cooling demand due to hot-water server cooling compared to the reference case is approximately constant throughout the year and amounts to ΔQ . CC = 360 kW. In the reference scenario, cold water is supplied by a compression chiller with | 1797 OLTMANNS eT AL. a seasonal energy efficiency ratio of SEER = 3.61 including free cooling. The annual savings in CO 2 emission from compression cooling ΔCO 2,CC can be calculated using the following equation: The amount of usable waste heat and consequently the CO 2 emission savings from waste heat utilization depend on the system the data center is connected to. In case of TU Darmstadt Lichtwiese, the purpose of the waste heat utilization is to substitute heat from heat-only boilers, not heat from CHP plants. If CHP heat was substituted by waste heat, this would also lead to a decrease in CHP electricity generation. The electricity no longer generated locally would have to be bought from the grid. Since the current energy mix of grid electricity still relies on a significant portion of fossil fuels, this would lead to an increase in CO 2 emission. Therefore, waste heat is only integrated into the district heating network when it does not replace CHP heat. In the summer months, when the heat flow demand of the university is below the cumulated thermal power of the installed CHP plants, the data center waste heat has to be dissipated via free cooling ( Figure 2). The CO 2 emission savings from waste heat utilization ΔCO 2,WH,1 in scenario 1 are simulated using a model of the entire district heating network, considering the utilized waste heat Q WH,1 , the CO 2 emission factor for gas, and the efficiency of the boilers η HOB . The thermal efficiency of the network is not taken into account in this case, because the savings are calculated at the feed-in point of the heat into the district heating system. As long as the heat demand of the network is higher than the cumulated thermal power of the CHP plants, the CO 2 emission savings due to waste heat utilization for each time step can be calculated using Equation (2): To calculate the CO 2 emission savings ΔCO 2,WH,2 for scenario 2, a dynamic CO 2 emission factor for the replaced district heat at the building boundary is calculated and then multiplied with the utilized waste heat Q WH,2 in every time step. This factor is based on the CO 2 emission for heat in the current energy system at Campus Lichtwiese, considering the gas boilers as heat source, as well as the losses in the district heating network. The average CO 2 emission factor for replaced district heat is 0.275 t CO 2 ∕MWh.
In the case of scenario 1, additional CO 2 emission due to the electric energy demand of the heat pump have to be considered. The COP of the heat pump COP HP is calculated within the simulation according to the temperatures at both the evaporator and the condenser of the heat pump. For the (1) Lichtwiese district heating and district cooling system operation of the heat pump, the CO 2 emission factor for grid electricity is taken into account, assuring that the university's share of CHP electricity in its electricity mix is not decreased by this setup.

| Economic evaluation
An economic evaluation of the waste heat utilization is done calculating the average annual savings or additional costs over a period of 10 years (annuity). The annuities for capitalrelated costs A C and energy demand-related costs A D are calculated with the help of the guidelines of VDI 2067, 52 using the annuity factor a and the price-dynamic cash value factor b. A 0 stands for the initial investment, and A 1 stands for the energy demand-related costs in year 1. The interest rate is defined at q = 5%, and the price change factor is considered to be r = 2%, which represents the desired inflation rate in the Euro zone. 53 The observation period is τ = 10 years: For the economic evaluation of the waste heat utilization, we present a generic case study for a typical German district heating network rather than the specific economic case of TU Darmstadt. The district heating network at TU Darmstadt is owned by the university but currently operated by an external contractor. This makes it hard to determine the university's specific energy costs, because instead of paying for the gas consumed, TU Darmstadt pays the contractor for the heat supplied to the individual buildings. The price for this heat consists not only of demand-related expenses for generation and distribution of the heat, but also of investment-related costs for the modernization of the system. Using the price for heat paid by the university would therefore overestimate the benefit of the waste heat utilization. Instead, typical prices for gas (32 € 2019 /MWh 54 ) and grid electricity (160 € 2019 /MWh 54 ) are considered. The total investment cost for the waste heat utilization was estimated with the help of external specialists and represents approximately 1.15 Mio. € in scenario 1 and 1.05 Mio. € in scenario 2. The investment cost includes planning expenses which are estimated to be about 15% of the gross investment for necessary installations to realize the waste heat utilization.

| Energy efficiency metrics
In order to evaluate and compare the energetic performance of different data centers, energy efficiency metrics can be a very helpful tool. A multitude of metrics to characterize the energy efficiency of data centers has been used in the past. 33 They address different aspects of data centers from different perspectives, including the server itself 55 ; the heating, ventilation, and air-conditioning (HVAC) system 56 ; or the data center as a whole. 57 In the context of this study, we concentrate on the overall efficiency of the data center. The most common generic data center energy efficiency metric is the Power Usage Effectiveness (PUE), which compares the total annual energy demand of the data center to the annual energy demand of servers and necessarily needs to be equal to or greater than one. 58 As stated in, 40 this metric is not useful in a data center with waste heat utilization, since the value of the reused energy is not being taken into account. A better metric for data centers including waste heat utilization is the energy reuse effectiveness (ERE) as proposed in Ref. 40 : The disadvantage of this approach is that no difference is made between the different energy forms in terms of their quality. While the energy input of the data center is electric energy, the reuse energy leaves the system as heat, creating a system with energy flows of different qualities. To account for these differences, an exergetic metric that specifies the quality of the energy in addition to the available quantity is more suitable. 35 Based on a methodology developed for an analysis of the exergy destruction of an air-cooled data center, 35 we calculate the exergy reuse effectiveness (ExRE), taking into account the differences in quality of the different energy flows in the system: Both metrics are useful when both the amount of used energy and the energy quality-related factors such as CO 2 emission play a role for the assessment of different concepts for waste heat utilization. In this study, we present the ERE and ExRE for two waste heat utilization alternatives and compare them to the ERE of the traditional low-temperature air cooling formerly installed in the data center. The lower the value of the ERE and ExRE parameter becomes, the better the performance of the considered scenario. The ERE and the ExRE of the current state (low-temperature air cooling) are the same, because all energy flows considered in this case are electric energy flows.

| Description of scenarios
In this chapter, we present the characteristics of the two possible scenarios for waste heat utilization at TU Darmstadt and a reference scenario for comparison. For all scenarios, the campus level is considered as system boundary. This way, it becomes possible to compare the results presented in chapter 5. Table 1 gives an overview of the scenarios for waste heat utilization which are described in more detail below.

| Reference scenario and assumptions
The reference scenario 0 represents a business as usual scenario, in which the new generation of TU Darmstadt's HPC "Lichtenberg II" is cooled using air cooling as well as rear door water heat exchangers at 17-24°C and free cooling. This corresponds to the cooling of the current generation of the HPC. Table 2 shows the assumptions made for the two waste heat utilization scenarios. Those will be presented in more detail below.

| Scenario 1: "Campus"
When heat is integrated in a district heating network, the conditions of the heat flow need to be adapted to the requirements of the district heating network. The minimum temperature of the heat is determined by the district heating return temperature and must always be higher than the latter. Since the temperature of the district heating return flow at Campus Lichtwiese varies between 50°C and 70°C throughout the year, and the temperature of the waste heat is only 45°C, a heat pump is necessary at all times to upgrade the waste heat temperature. On the other hand, the district heating return temperature is limited to a maximum of 70°C in order to guarantee the proper functioning of the CHP plants. Figure 3 shows the setup for the integration of the HPC waste heat into the district heating return flow. The HPC is cooled via water cooling at the server level. The water used for the server cooling circuit has special quality requirements; therefore, this circuit must be disconnected hydraulically from the rest of the cooling system using a heat exchanger (HX). The manufacturer of the HPC guarantees that the waste heat is released at a temperature not lower than 45°C after the heat exchanger. To ensure the removal of the heat from the system even when it cannot be used in the district heating network, the hybrid cooler (HC) already in place at the data center is connected between the servers and a buffer storage (BS). Since the temperature at which the heat is emitted from the HPC in this scenario is a lot higher than it was in the low-temperature air-cooling case, heat can be dissipated through the hybrid cooler all year round and no mechanical cooling is necessary, not even in the summer. In addition to this, the stratified buffer storage (BS) situated between the HPC and the heat pump serves to smooth out the cooling power demand of the HPC that potentially fluctuates over time, thereby decreasing the amount of load shifts for the heat pump. The heat pump is the main component of this setup. It receives waste heat at the evaporator (cold side) at 45°C. The output temperature at the heat pump condenser (hot side) depends on the district heating return temperature. Due to the high waste heat temperature entering the heat pump evaporator, this setup is very efficient, yielding an average COP of COP HP = 6.8. The hot circuit of the heat pump cannot be connected directly to the district heating return flow but must be separated hydraulically using another heat exchanger. For the purpose of this study, the mean temperature difference over all heat exchangers is considered to be ΔT m = 5 K.

| Model setup Scenario 1
A model of the data center cooling system including the heat pump is integrated in a greater model of the campus energy Similar to the reference scenario 0 but including HPC waste heat integration into the district heating return flow via a heat pump. No compression cooling is necessary due to high waste heat temperatures, but some of the waste heat needs to be dissipated via free cooling when waste heat utilization is not possible. system which was developed in MATLAB Simulink using the CARNOT Blockset. 59 This model consists of a detailed representation of the heat and power station, the buildings as consumers of heating, cooling and electric energy as well as the thermal networks. It was validated for another district heating system in Ref. 60 The model uses the buildings' heat demand and return temperature as well as the supply temperature at the heat and power station as input data. On this basis, it calculates the mass flow in each network section and the resulting temperatures in the supply and the return line as well as the network losses. For this purpose, the network is modeled as a set of pipe sections between the different buildings, considering the diameter of the pipe itself as well as the diameter and the thermal conductivity of the insulation. For the calculation of the network losses, the impact of the supply and return line on each other's surrounding temperature are taken into account. 61 Figure 4 presents the calculation flowchart for scenario 1. With the help of this model, the amount of useful waste heat can be computed, respecting the limitations in terms of the priority for CHP heat. At the same time, the model serves to calculate the impact of the waste heat integration on the network return temperature. The data center is located at the end of the return line on the western side of the campus with no other building connected in between the data center and the heat and power station. Therefore, the return flow at the data center can be heated up to the maximum CHP inlet temperature of 70°C without risking overshooting the maximum return flow temperature by other buildings.
The electric energy demand for the circulation pumps necessary in each circuit of the data center cooling system is very low compared to the electric energy demand of the HPC and the heat pump. Therefore, the electric energy demand for pumping is neglected in the context of this study. The MATLAB Simulink model is used to calculate the amount of heat from heat-only boilers replaced by waste heat, the grid electric energy saved due to reduced demand in compression cooling, and the additional electric energy demand for the heat pump.
To be able to quantify the Exergy Reuse Effectiveness (ExRE), the exergy of the utilized waste heat must be calculated. In scenario 1, this is done by calculating the difference between the exergy of the mass flow leaving the district heating network and the one returning to the district heating network. The Test Reference Year outdoor temperature 62 serves as reference temperature for this calculation.

| Scenario 2: "Building"
The waste heat integration into the district heating network via a heat pump (scenario 1) is compared to a direct use in the civil engineering experimental halls ( Figure 5) which consist of three smaller and one large building with a total floor area of 5422 m 2 . They are located in close vicinity of the data center. The buildings considered were constructed in the 1970s and have poor physical properties, such as singleglazing, low air-tightness, poor insulation, and substantial thermal bridges.
To be able to use the HPC center waste heat in these buildings without a heat pump, low-temperature heating systems must be installed. Therefore, a standard renovation with minimally invasive measures including the roof surfaces and all windows in compliance with the German legal requirements (Energieeinsparverordnung EnEV 63 ) is part of scenario 2. These measures fulfill only minimal legal requirements but are very efficient to improve indoor comfort and cut down energy consumption. All simulations have been calculated using Test Reference Year climate data. 62 The model has been validated comparing simulated energy demand based on site-specific climate data of a weather station with measured heat consumption data.
To be able to use the HPC waste heat most efficiently, a LT heating system of ceiling mounted heating panels with a total surface area of 1075 m 2 and a total heating power of 313 kW is considered in the renovation scenario. Additionally, the existing radiators remain in place and serve as backup in case the HPC is not running or the waste heat supply does not reach the expected amount. Figure 6 shows that space heating in scenario 2 is covered by a bivalent system with two separated circuits. In this case, the ceiling panels serve as "primary" heating system fed by waste heat from the HPC. As "secondary" heating system, the existing radiators cover the peak-load demand.
As in scenario 1, waste heat can only be used to replace heat from the gas boilers and must not decrease the amount of heat generated via CHP. Therefore, even though the heat flow emitted by the HPC is considered constant throughout the year, it cannot always be used and has to be dissipated via free cooling in the summer months. The usable waste heat for the "primary" heating system is shown in Figure 2.

| Model setup Scenario 2
The building-related calculations such as load profiles, indoor thermal climate, plant temperatures, and mass flows have been done with IDA ICE version 4.8. The software has been chosen because it is a multizone simulation application which allows a detailed investigation of the interaction between building-physics, plant systems, and their controllers. The software has been validated using measured data according to ASHRAE 140 and CEN Standards of EN 15255, 15265, and 13791. 64
In scenario 2, the "primary" system (heating panels) covers the base load. The usable waste heat from the HPC ( Figure 5) limits the available power for the LT heating system. The ceiling panels are supplied with HPC waste heat at a constant temperature of 43°C (waste heat temperature minus temperature loss between data center and buildings), and the existing radiators are supplied with district heating. The radiators' supply temperature is defined by the buildings' heating curve. The heating power of all heating units is controlled using a variable mass flow. It is limited to the maximum mass flow under design conditions in order to keep pressure losses in pipes low and to prevent high electricity demands for circulating pumps. The heating panels are switched on at a heating threshold air temperature of 18°C and switched off after reaching 20°C. As soon as the heating capacity of the ceiling panels does not cover the entire heat demand of the buildings, the radiators operate as peak supply units. The set point is 20°C indoor air temperature, and no night setback is considered in order to minimize peak loads. Figure 7 illustrates the calculation process. In scenario 2, the exergy of the utilized waste heat needed to quantify the ExRE is calculated as the exergy of the waste heat flow leaving the data center at a constant temperature of 45°C. As in the case of scenario 1, the Test Reference Year outdoor temperature 62 serves as reference temperature.

| RESULTS
The goal of this study is to quantify how much waste heat can be used and what amount of CO 2 emission reductions result in the different scenarios presented above. Additionally, a comparison of the annuity savings or additional expenses in the different scenarios is presented. Figure 8 shows available and useful waste heat in the two scenarios on a monthly basis. In general, waste heat use is high in the winter and no heat can be consumed in the summer due to the constraint that waste heat is not allowed to replace CHP heat, as presented in Figure 2. The highest share is used when the waste heat is integrated in the district heating network, in which case almost the entire heat emitted by the HPC can be used in wintertime (about 50% or 1600 MWh/a considering the entire year). For the direct reuse scenario in the civil engineering experimental halls, the share of used waste heat is about 20% or 630 MWh/a and much lower than in scenario 1. The comparatively low usage potential in this scenario is mainly due to the fact that the total heat demand of the considered buildings is low compared to the available waste heat from the HPC. If it were possible to increase the low-temperature heating potential in the area and to triple the demand, the same amount of waste heat as in scenario 1 could be used in scenario 2.

| CO 2 emission savings
The potential CO 2 emission savings are generated on one hand by using waste heat either in the district heating network or in the civil engineering experimental halls, on the other hand by reducing the electricity demand for compression cooling possible through the change from cold to hotwater cooling of the HPC servers. While the savings from the waste heat itself depend on the potential of waste heat utilization, which changes throughout the year, the savings from reduced cooling demand are the same in all scenarios. The comparison of the different scenarios in Figure  9 indicates that the CO 2 emission savings from waste heat utilization are highest when the heat is integrated in the district heating network, but the heat pump necessary in this case generates additional emission. For scenario 2, potential savings in CO 2 emission not only depend on waste heat utilization and reduced electricity demand for compression cooling, but also depend on savings due to building renovations. These renovations are necessary in order to reduce the heat supply temperature inside the buildings and to make it possible to use the HPC waste heat without increasing its temperature via a heat pump. However, the savings resulting from the renovations are not directly linked to the HPC waste heat utilization. Therefore, they are indicated separately in Figure 9. Figure 10 shows the total monthly CO 2 emission savings in the different scenarios. The yearly savings are compared to the total CO 2 emission emitted to supply the energy demand of Campus Lichtwiese (17 000 t CO 2 in 2018). The savings amount to 720 t CO 2 ∕a in scenario 1 (4.2%) and 670 t CO 2 ∕a in scenario 2 (4.0%) and therefore represent a significant amount of the total CO 2 emission of Campus Lichtwiese in both scenarios. If only CO 2 emission reductions directly related to the data center (waste heat utilization and reduced compression cooling demand) are considered, the savings in scenario 2 drop to 570 t CO 2 ∕a (3.4%). The difference between the two scenarios is small. On one hand, this is due to the fact that the CO 2 emission savings generated by the reduction in compression cooling demand are the same for both scenarios. On the other hand, the higher share of waste heat utilization in scenario 1 is leveled out by the additional emission for heat

Comparison of annuities
To determine the economic impact of the waste heat utilization, savings due to lower electricity costs for compression cooling and decreased gas costs created by the waste heat integration are taken into account. On the other hand, costs are increased by the additional electric energy demand to supply the heat pump and by the annuity of the investment. In scenario 2, only costs for investments exceeding the minimum energetic renovations required under German legislation are taken into account.
In Figure 11, a comparison of the change in annuity for the different scenarios is presented. As in the case of the CO 2 emission, savings are higher in the district heating integration scenario 1, but also additional costs are highest in this scenario. Figure 12 shows the total monthly annuity of savings in both scenarios. Due to the high costs for electricity to supply the heat pump, savings are higher in scenario 2 than in scenario 1 although the used waste heat is a lot higher in scenario 1. Both scenarios represent slight reductions in the overall annuity of 28 000 €/a and 64 000 €/a, respectively, compared to the reference scenario 0. Compared to the total investment costs, these savings are small, and a small change in energy costs can affect them significantly. Conservatively speaking, it can therefore only be stated that the waste heat utilization will most probably not increase energy costs in either of the two scenarios.

Exergy Reuse Effectiveness (ExRE)
A comparison of all scenarios in terms of the Energy and Exergy Reuse Effectiveness is presented in Table 3. In both scenarios 1 and 2, hot-water cooling and waste heat utilization lead to a considerable reduction in ERE and ExRE. The comparison of both scenarios shows that the ERE is lowest when integrating the heat in the district heating system while in terms of the ExRE scenario 2 performs better. The reason for the differences in the ranking of the scenarios is that the amount of utilized energy is highest in scenario 1, but the greater waste heat utilization potential comes with the disadvantage of an additional electric energy demand for the heat pump. That makes this scenario less favorable in terms of its ExRE performance.

| Summary of results
The comparison of the results for all parameters presented above shows that no single scenario performs best in all relevant aspects. While the integration of waste heat into the district heating yields the highest share of used waste heat and the highest savings in CO 2 emission, the economic performance of the direct waste heat utilization in adjacent buildings can be better, due to the additional costs of the heat pump electric energy in the district integration case. The integration of LT heating systems in the experimental halls serves as a "door-opener" for a direct use of waste heat from the HPC. Scenario 2 yields significant cuts in CO 2 emission applying relatively simple renovation of the buildings' envelope using standard insulation levels.

| CONCLUSION AND OUTLOOK
The goal of this case study was to proof the ecologic and economic feasibility of a connection of hot-water cooling and waste heat utilization from a HPC, considering the example of "Lichtenberg II" at TU Darmstadt Lichtwiese. For this purpose, two scenarios were set up, one considering an integration of the HPC waste heat into the return flow of the district heating network via a heat pump and one scenario investigating the direct use in buildings in the data center's vicinity. The results show that both scenarios yield considerable reductions in CO 2 emission and the investments for the heat pump and other equipment will be repaid through reduced energy costs over a period of 10 years. At the same time, it becomes evident that most waste heat can be used when it is integrated in the central district heating network rather than being used on the local level at a specific complex of buildings. On the other hand, the district heating integration is only possible after increasing the waste heat temperature via a heat pump that generates additional costs and CO 2 emission.
Currently, TU Darmstadt is realizing a waste heat integration into its district heating network as described in scenario 1. This option is the least complex to implement, and it uses the greatest amount of HPC waste heat and yields the highest CO 2 emission savings. Furthermore, it is not necessary that available waste heat and heat demand are locally balanced. In the future, it will be interesting to investigate how the modeling results presented here compare to the CO 2 emission savings achieved in the actual operation of the system. The university is also currently working on decreasing the temperatures in its district heating network in order to decrease network losses and facilitate the use of renewable and waste heat. Lower temperatures in the district heating network would reduce the electricity demand of the heat pump for the HPC waste heat utilization. How the change in network temperature affects the performance of the waste heat utilization will also be subject of future research.

ACKNOWLEDGMENT
Technische Universität Darmstadt is funded for this research by the German Federal Ministry of Economic Affairs and Energy (BMWi) in the project "EnEff:Stadt Campus Lichtwiese" (03ET1638). Further financial support comes from the DFG in the framework of the Graduate School of Excellence Energy Science and Engineering (GSC 1070).