Document jBpXrVNo3m2Xk8w1NgB19xyg2

Challenges of data center thermal management The need for more performance from computer equipment in data centers has driven the power consumed to levels that are straining thermal management in the centers. When the computer industry switched from bipolar to CMOS transistors in the early 1990s, low-power CMOS technology was expected to resolve all problems associated with power and heat. However, equipment power consumption with CMOS has been rising at a rapid rate during the past 10 years and has surpassed power consumption from equipment installed with the bipolar technologies 10 to 15 years ago. Data centers are being designed with 15-20-year life spans, and customers must know how to plan for the power and cooling within these data centers. This paper provides an overview of some of the ongoing work to operate within the thermal environment of a data center. Some of the factors that affect the environmental conditions of data-communication (datacom) equipment within a data center are described. Since high-density racks clustered within a data center are of most concern, measurements are presented along with the conditions necessary to meet the datacom equipment environmental requirements. A number of numerical modeling experiments have been performed in order to describe the governing thermo-fluid mechanisms, and an attempt is made to quantify these processes through performance metrics. R. R. Schmidt E. E. Cruz M. K. Iyengar Introduction Because of technology compaction, the information technology (IT) industry has experienced a large decrease in the floor space required to achieve a constant quantity of computing and storage capability. However, power density and heat dissipation within the footprint of computer and telecommunications hardware have increased significantly. The heat dissipated in these systems is exhausted to the room, which must be maintained at acceptable temperatures for reliable operation of the equipment. Data center equipment may comprise several hundred to several thousand microprocessors. The cooling of computer and telecommunications equipment rooms has thus become a major challenge. Background Considering the trends of increasing heat loads and heat fluxes, the focus for customers is in providing adequate airflow to the equipment at a temperature that meets the manufacturers' requirements. This is a very complex problem considering the dynamics of a data center, and it is just beginning to be addressed [1-8]. There are many opportunities for improving the thermal environment of data centers and for improving the efficiency of the cooling techniques applied to data centers [8-10]. Airflow direction in the room has a profound effect on the cooling of computer rooms, where a major requirement is control of the air temperature at the computer inlets. Cooling concepts with a few different airflow directions are discussed in [11]. A number of papers have focused on airflow distribution related to whether the air should be delivered overhead or from under a raised floor [12-14], ceiling height requirements to eliminate ``heat traps'' or hot air stratification [12, 15], raised-floor heights [16], and proper distribution of the computer equipment in the data center [13, 17] such that hot spots or high temperatures would not exist. Different air distribution configurations, including those described in subsequent sections, have Copyright 2005 by International Business Machines Corporation. Copying in printed form for private use is permitted without payment of royalty provided that (1) each reproduction is done without alteration and (2) the Journal reference and IBM copyright notice are included on the first page. The title and abstract, but no other portions, of this paper may be copied or distributed royalty free without further permission by computer-based and other information-service systems. Permission to republish any other portion of this paper must be obtained from the Editor. 0018-8646/05/$5.00 2005 IBM 709 IBM J. RES. & DEV. VOL. 49 NO. 4/5 JULY/SEPTEMBER 2005 R. R. SCHMIDT ET AL. Heat load per product footprint (W/ft2) 1992 1994 1996 1998 2000 2002 2004 2006 2008 2010 Heat load per product footprint (W/m2) 10,000 8,000 6,000 4,000 2,000 1,000 800 600 400 200 100 80 60 100,000 80,000 60,000 40,000 20,000 10,000 8,000 6,000 4,000 2,000 1,000 800 formed a consortium1 in 1998 to address common issues related to thermal management of data centers and telecommunications rooms. Since power used by datacom equipment was increasing rapidly, the group's first priority was to create the trend chart shown in Figure 1 on power density of the industry's datacom equipment to aid in planning data centers for the future. This chart, which has been widely referenced, was published in collaboration with the Uptime Institute [20]. Since publication of the chart, rack powers have exceeded 28,000 W, leading to heat flux based on a rack footprint in excess of 20,000 W/m2. Year of product announcement Communication equipment (frames) Servers and disk storage systems (1.8-2.2 m tall) Workstations (standalone) Tape storage systems Figure 1 Power trends in data center equipment. Reprinted from [20], with permission. Data center heat flux (facilities sizing) Zonal heat flux (thermal focus) Rack heat flux (manufacturer's metric) Hot aisle Cold aisle Hot aisle Cold aisle Computer racks Computer room air conditioning (CRAC) Recent activities of the American Society of Heating, Refrigerating, and Air-Conditioning Engineers (ASHRAE) In January 2002 ASHRAE was approached to create an independent committee to specifically address highdensity electronic heat loads. ASHRAE accepted the proposal and eventually formed the technical committee known as TC9.9 Mission Critical Facilities, Technology Spaces, and Electronic Equipment. The first priority of TC9.9 was to create a Thermal Guidelines document that would help to standardize the designs of equipment manufacturers and help data center facility designers to create efficient, fault-tolerant operation within the data center. The resulting document, ``Thermal Guidelines for Data Processing Environments,'' was published in January 2004 [21]. More recently, another publication entitled ``Datacom Equipment Power Trends and Application,'' published in February 2005 [22], provided an update to the power trends documented in [20] and shown here in Figure 1. The heat load of some datacom equipment was found to be increasing at an even faster rate than that documented in the original chart published by The Uptime Institute [20]. Other ASHRAE publications will follow on data center thermal management, with one planned for the end of 2005 that will provide information on air and liquid cooling of data centers, energy conservation, contamination, acoustics, etc. Figure 2 Organization of computer racks in a hot aisle-cold aisle layout. been compared with respect to cooling effectiveness in [14, 16, 18, 19]. Trends--Thermal management consortium for data Physical design of data center systems Layout of computer rack equipment Data centers are typically arranged into hot and cold aisles, as shown in Figure 2. This arrangement accommodates most rack designs, which typically employ front-to-back cooling, and partially separates the cold air exiting the perforated tiles (for raised-floor designs) and overhead chilled airflow (for non-raised-floor designs) from the hot air exhausting from the rear of the racks. 710 centers and telecommunications rooms Since many of the data center thermal management issues span the industry, a number of equipment manufacturers 1The following companies formed the Thermal Management Consortium for Data Centers and Telecommunication Rooms: Amdahl, Cisco Systems, Compaq, Cray, Inc., Dell Computer, EMC, HP, IBM, Intel, Lucent Technologies, Motorola, Nokia, Nortel Networks, Sun Microsystems, and Unisys. R. R. SCHMIDT ET AL. IBM J. RES. & DEV. VOL. 49 NO. 4/5 JULY/SEPTEMBER 2005 Zonal heat flux (W/ft2) Rack power (kW) 500 40 Extreme thermal High-performance problem 400 computing 30 300 20 200 A Commercial 10 (a) 100 B computing Creeping 0 0 1996 1998 2000 2002 2004 2006 2008 thermal problem Year Figure 3 Trends for zonal data center heat flux. (b) The racks are positioned so that the fronts of the racks face the cold aisle. Similarly, the rear sides of the racks face each other and provide a hot-air exhaust region. This layout allows the chilled air to wash over the front sides of the data processing (DP) equipment, while the hot air from the racks exits into the hot-air aisle as it returns to the inlets of the air conditioning (A/C) units. With the arrangement of DP equipment in rows within a data center, zones may exist in which all of the equipment contained in a zone dissipates very high heat loads. This arrangement of equipment may be required in order to achieve the desired performance; however, such high-performance zones (Figure 3) may create significant challenges to maintaining an environment within the manufacturers' specifications. This figure shows the trend (high-performance computing) for these high-heat-flux zones based on the equipment power trends shown in Figure 1. In contrast, the trend toward data centers containing a mix of computer equipment that employs lower-power racks (commercial computing) is also shown in Figure 3. Many data centers currently follow this lower trend line. (c) (d) Figure 4 Some data center airflow configurations: (a) raised-floor supply; (b) raised-floor supply/ceiling return; (c) raised-floor supply/ceiling supply; (d) non-raised floor/ceiling supply. types of air distribution configurations are shown in Figure 4. Air distribution configurations Airflow distribution within a data center has a major Raised-floor room cooling impact on the thermal environment of the data processing Figure 4(a) shows a raised-floor arrangement in which equipment located in these rooms. A key requirement of chilled air enters the room via perforated tiles in the floor manufacturers is that the inlet temperature and humidity and exits the room into air conditioning units known to the electronic equipment be maintained within their as CRAC (computer room air conditioning) units. specifications. To provide an environment of controlled Computers typically have numerous cables connecting temperature and humidity, two types of air distribution the components within a rack and between racks. To configurations are commonly utilized for such equipment: maintain an organized layout, a raised floor (also known raised-floor and non-raised-floor layouts. These and other as a false floor or double floor) is used, with other 711 IBM J. RES. & DEV. VOL. 49 NO. 4/5 JULY/SEPTEMBER 2005 R. R. SCHMIDT ET AL. Chilled air entry Hot exhaust air recirculation and humidity requirements for the electronic equipment housed within the data center. For example, one large computer manufacturer has a 42U rack2 configured for front-to-back air cooling and requires that the temperature of the air supplied to the front of the rack be maintained between 108C and 328C for elevations up to 1,287 m (4,250 feet). Higher elevations require lowering of the maximum dry bulb temperature 18C for every 218 m (720 feet) above an elevation of 1,287 m (4,250 feet) up to a maximum elevation of 3,028 m (10,000 feet). These temperature requirements are to be maintained over the entire front of the 2-m height of the rack where air is drawn into the system. Figure 5 Hot rack exhaust air recirculation--A major cooling Computer-based simulation of hot-air recirculation in a raised-floor data center. problem In such air-cooled racks, with airflow nominally front-to- back, the chilled-air supply, whether from a raised- floor tile [Figure 4(a)] or via diffusers from the ceiling interconnect cabling connected beneath the raised floor. [Figure 4(d)], is typically only a fraction of the rack The space under the raised floor is also used as an air- airflow rate. This is due to the limitation of tile or diffuser supply plenum, with perforated tiles exhausting chilled flow rate. The remaining fraction of the supply-side air air. Similarly, it is possible for the room to have a false is made up by ambient room air through recirculation. ceiling (also called a dropped ceiling), with the space This recirculating flow is often extremely complex in above the false ceiling used as the air supply or the return nature and can lead to rack inlet temperatures that are plenum, as shown respectively in Figures 4(b) and 4(c). significantly higher than expected. Figure 5 shows the complex air circulation patterns resulting from a Non-raised-floor room cooling computer-based simulation of a raised-floor design such Figure 4(d) shows another configuration, in which the as the one depicted in Figure 4(a). As may be seen in chilled air enters the room from diffusers in the ceiling Figure 5, there is significant recirculation of hot exhaust and exits the room via vents that may be placed at air, which can be detrimental to the performance and different locations. Cooling air can be supplied from the reliability of computer equipment. Elimination of this ceiling in the center of the room, where computers are recirculation with barriers above the datacom racks is located, with exhausts located near the walls. Short typically not a viable alternative because of fire code partitions are installed around the supply openings restrictions. Data processing equipment is typically to minimize short-circuiting of supply air to returns designed to operate for rack air inlet temperatures in the (short-circuiting occurs when air takes the path of least 108C to 358C range. Because of recirculation, there could resistance). Similarly, cool air from outlets distributed be a wide range of inlet air temperatures across the rack. across a ceiling can be supplied, with the exhaust located For a raised-floor layout [Figure 4(a)], the inlet air around the perimeter or a return in the floor, as shown in Figure 4(d). Alternatively, a design employed by the telecommunications industry and more recently adopted by the computer industry utilizes heat exchangers located above the racks near the ceiling. The racks are arranged using the hot and cold aisle concept in which hot air is collected in the heat exchanger and, once cooled, is forced down into the cold aisles, using the fans mounted at the bottom of the heat exchanger. temperature can range from 108C-158C at the bottom of the rack close to the chilled air supply to as much as 308C-408C at the top end of the rack, where the hot air can form a self-heating recirculation loop. Since the rack heat load will be limited by the rack inlet air temperature at the ``hot'' part, this temperature distribution correlates to inefficient utilization of available chilled air. Also, since data center equipment almost always represents a large capital investment, it is of paramount importance Factors influencing rack air inlet temperatures from a product reliability, performance, and customer satisfaction standpoint that the temperature of the inlet Temperature and humidity specifications As previously stated, the primary focus in thermal air to a rack be within the desirable range. The efficient cooling of such computer systems and the amelioration of 712 management for data centers is to meet the temperature 2U is equal to 1.75 in. R. R. SCHMIDT ET AL. IBM J. RES. & DEV. VOL. 49 NO. 4/5 JULY/SEPTEMBER 2005 localized recirculation currents of hot air returned to the rack constitute a focus item of the present paper. Table 1 Key characteristics of the NCEP data center. Parameter Measurement Impact of high financial investment in data centers In addition to a significant increase in the power density of the equipment in the data center, other factors affect Perforated tile flow rate Cable opening flow rate 21.4 m3/s (61% of total flow) 9.5 m3/s (27% of total flow) data center thermal management. Information Other opening flow rate 2.7 m3/s (8% of total flow) technology equipment must be deployed quickly to obtain maximum use of a large financial asset; this may mean that minimal time can be spent on site preparation, which can result in thermal issues after the equipment is Tile leakage flow rate Total power dissipated Heat flux in region of p690 1.5 m3/s (4% of total flow) 534.4 kW 1,830 W/m2 (170 W/ft2) installed. The construction cost of a data center exceeds $15,000/m2 in some metropolitan areas, with an annual operating cost of $500 to $1,500/m2. For these reasons, the aim is to obtain the most from data center space Average perforated tile flow rate Average temperature rise to top portion of rack 0.17 m3/s (363 ft3/min) 10.58C and maximize the utilization of the infrastructure. Unfortunately, the current situation in many data centers does not allow for this optimization. The equipment installed in a data center may have been obtained from many different manufacturers, each having a different environmental specification. With these varying requirements, the data center must be overcooled to compensate for the equipment with the tightest requirements. racks. Finally, the inlet air temperatures for the racks at a height of about 1.8 m (6 ft) were measured. Preliminary temperature measurements showed that the location near the top of the rack exhibited one of the highest inlet air temperatures. Measurements of power and airflow rate were also obtained from each CRAC unit located around the perimeter of the room; these measurements were Thermal profiling of high-power-density data centers To aid in the advancement of data center thermal management, Schmidt published the first paper that presented a complete thermal profile of a high-powerdensity data center [23]. The paper provided basic information on the thermal/flow data collected from such a data center, and it also provided a methodology for others to follow in collecting thermal and airflow data from data centers so that the information could be deemed critical to thermal profiling of the data center. Some of the parameters of this data center are given in Table 1. The data indicated that the total airflow rate, including airflow through the perforated tiles adjacent to the racks and the cable openings associated with a rack, did not equal the rack flow rate. For the p690 racks, the airflow through the perforated tiles adjacent to the rack was approximately one fourth to one half of the airflow through the rack in order to maintain system inlet air assimilated for comparison. This database could then temperatures within the specifications. The perforated-tile provide the basis for future data center air cooling design and aid in the understanding of deploying racks with higher heat loads in the future. The data center profiled was the National Center for Environmental Prediction (NCEP), located in Bethesda, Maryland. All of the equipment was located on a raised floor in an enclosed area of 22.6 m 3 25.6 m (74 ft 3 84 ft). A plan view of the data center, indicating the location of the electronic equipment, power distribution units (PDUs), CRAC units, and perforated floor tiles, is shown in Figure 6. Most of the servers (51 racks) are IBM Model 7040 (pSeries* 690). The other systems were a mix of switching, communications, and storage equipment. The key parameters measured within the data center in order to calculate the total airflow rate into the room airflow rate plus the cable cutout flow rate was about one half to two thirds of the total airflow through the rack. Even though the chilled airflow adjacent to the server racks did not appear adequate, the total room flow rate was adequate to handle the data center heat load. The convection currents that occurred at the room level were sufficient to bring the local air temperatures for the highpowered racks within the temperature specifications. More recent data center measurements were conducted on a cluster of 134 server racks in the IBM Poughkeepsie Data Center, as shown in Figure 7. The machines were mainly IBM p690 and p655 servers. The area of the data center studied was 688 m2 (7,400 ft2), and the average heat flux was approximately 1,615 W/m2 (150 W/ft2). were flow rates from perforated tile openings, cable However, there existed a high heat flux region spanning openings, and other openings. The total power entering 33.8 m2 (364 ft2) and encompassing two rows of servers the room was also measured using PDUs and selected that reached a heat flux level of 6,290 W/m2 (585 W/ft2) 713 IBM J. RES. & DEV. VOL. 49 NO. 4/5 JULY/SEPTEMBER 2005 R. R. SCHMIDT ET AL. Future PDU Future CRAC Future PDU 92 kW PDU B2 - 37 9,330 CFM 24.0C inlet CRAC 47 42 41 40 39 38 37 36 35 34 33 32 31 30 29 28 27 26 25 24 23 22 21 20 19 18 17 16 15 14 13 12 11 10 9 8 7 6 5 4 3 2 1 CRAC 44 10,050 CFM 22.5C inlet PDU A2 - 36 86 kW CRAC 48 10,390 CFM 22.1C inlet Future CRAC CRAC 46 9,880 CFM 23.0C inlet PDU A3 - 38 89 kW Future CRAC Future growth Future CRAC Future PDU Future PDU Storage Storage Perforated tiles: 25% (typical) 17.2C average aisle exhaust 11.6C average aisle exhaust IBM p655 11.3C average aisle exhaust 86 kW PDU A1 - 34 10,600 CFM 27.1C inlet CRAC 42 9,600 CFM 25.4C inlet CRAC 43 90 kW PDU B1 - 35 40 kW PDU B3 - 39 10,550 CFM 24.4C inlet CRAC 45 A B C D E F G H I J K L M N O P Q R S T U V W X Y Z AABBCCDDEE FFGGHH II JJ KK LL Figure 6 Detailed layout for thermal profiling of high-power-density data center (NCEP.) (CFM cubic feet per minute.) From [23], reprinted with permission. (see the area in Figure 7 outlined by the dashed box). system (inputs), and those that reveal system performance Results obtained for this data center were similar to those (outputs). for the one described above. More significantly, the inlet rack air temperatures measured at a height of 1.8 m were Description metrics within the equipment specifications even with the very high power densities experienced. The most frequently used description metric is the heat flux of the data center (W/m2), which is defined as the ratio of the total heat load to the footprint area of the Metrics for data center characterization layout under consideration. By extending the heat flux to Two kinds of metrics are commonly employed in 714 describing a data center--those that help describe the include the volume of the facility, a volumetric heat load (W/m3) can also be used. R. R. SCHMIDT ET AL. IBM J. RES. & DEV. VOL. 49 NO. 4/5 JULY/SEPTEMBER 2005 Norota et al. [24] utilized a statistical metric known as the Mahalanobis generalized distance, D2, in an innovative manner to characterize the nonuniformity in rack heat load throughout a data center facility. In the data center context, D2 can be computed as 2 X x Y y S1 X xave ; 1 D ave ave Y yave where X and Y are the central coordinates of the CRAC units, xave and yave are the central coordinates of the heatgenerating racks, and S 1 is the inverse of the variance- covariance matrix S. In the rack heat loads across the entire layout, a large D2 value suggests significant nonuniformity, and a small D2 value suggests uniformity. Norota et al. conducted experiments for several different rack layouts that yielded a range of D2 values (3-9) showing a clear correlation between the energy efficiency of the particular layout and its corresponding Mahalanobis Generalized Distance [24]. High-heat-flux region CRAC 2 CRAC 4 CRAC 6 CRAC 8 CRAC 10 CRAC 12 CRAC 1 CRAC 3 CRAC 5 CRAC 7 CRAC 9 CRAC 11 Figure 7 High-heat-flux data center. Performance metrics The temperature differential between the rack inlet temperature at a specific vertical height and the chilled air entering the data center environment, either through a perforated tile (raised floor) or through an overhead perfectly efficient with respect to its flow configuration. Hot-air recirculation in the data center also adversely affects the cooling efficiency by increasing the value of diffuser (non-raised floor), has been used by Schmidt [1] SHI. However, the SHI and RHI are largely ``global'' and by Schmidt and Cruz [2-6] to identify localized hot metrics, and do not characterize the localized nature of spots. data center hot spots. A data center could display very Dimensionless metrics that characterize data center global cooling efficiency can be found in [25]. These metrics, namely the supply heat index (SHI) and the return heat index (RHI), help evaluate the design of hot and cold aisles. The SHI for a data center is expressed as favorable values for SHI (low) and RHI (high), while possessing significant local hot spots which would result in device failure. This local inefficiency can be captured by using a ratio of the temperature differentials, b, as given by SHI DDhhtcotahl ; 2 b DDTTirnalcekt ; 4 where Dhch is the air enthalpy rise due to the heat transfer between the cold and the hot aisles, and Dhtotal is the total enthalpy rise in the air that passes through the racks. The RHI for a data center is calculated using RHI Dhcrac ; 3 Dhtotal where DTinlet is the temperature differential between the rack inlet air and the chilled-air entry, and DTrack is the temperature rise in the air as it passes through the racks. The common range for b is between 0 and 1. The numerator for Equation (4) that defines b represents a local value, while the denominator is an average rack value. Therefore, the value of b is different for different where Dhcrac is the enthalpy removed from the hot exhaust air that passes through the chilled air conditioning units (CRACs). The sum of the SHI and the RHI is equal to unity. A higher value of SHI indicates poor data center design due to significant superheating of the chilled air in the cold aisle, caused by mixing of the cold- and hot-aisle air streams. This results in low data center cooling efficiency. locations in front of the rack (as well as for different racks). Some parts of the rack may show a b value of 1 or possibly greater than 1, but the averaged value will always be between 0 and 1. A value of 0 for b indicates no impact of hot-air recirculation at that location, and a value of 1 indicates that the local inlet temperature is the same as the average rack exit temperature. A value for b greater than 1 means that there exists a local self-heating loop If the hot and cold aisles are perfectly insulated from each that raises the air temperature locally more than if it had other, the value of SHI is 0, and the data center is passed just once through the rack. 715 IBM J. RES. & DEV. VOL. 49 NO. 4/5 JULY/SEPTEMBER 2005 R. R. SCHMIDT ET AL. Rack airflow direction papers are presented in order to provide some guidance on the design and layout of a data center. A plan view of the two data center layouts is shown in Perforated floor tiles Racks Figure 8. Two symmetrical arrangements of the CRAC units were investigated, thereby requiring modeling of only half of the data center. Figure 8(a) shows the Air conditioning units CRAC units perpendicular to the rows of racks, while Figure 8(b) shows the CRAC units in the parallel (a) Computational domain (b) configuration. Each of the four CRAC units was sized to provide the necessary heat removal capability for one Figure 8 quarter of the heat dissipated by the electronic equipment inside the DP units installed on the raised floor. The CFD model data center layout: (a) A/C units perpendicular to rows; (b) A/C units parallel to rows. From [2], reprinted with permission. physical size of the CRAC units (890 mm wide by 2,440 mm long by 1,800 mm high) was consistent with those offered by manufacturers in the industry. The DP equipment, which used front-to-back cooling, was Norota et al. [24] defined a heat-source energy consumption coefficient, eCOM, that is calculated using X qrack Wblower eCOM Xn X ; 5 qrack Wcrac n m arranged in rows so that the inlets faced each other, creating the cold aisle, and the backs faced each other, creating the hot aisle. This type of layout allows the chilled air to exhaust from the cold-aisle perforated tile openings and wash the fronts of the DP racks while the hot air exhausting from the DP racks exits into the hot aisle before returning to the CRAC unit inlets. All of the where qrack is the rack heat load, n is the number of racks, DP racks measured 610 mm wide by 1,220 mm deep by Wblower and Wcrac are the work performed respectively by 2,000 mm high. The spacing between the rows of DP the blower and the air conditioners, and m is the number racks was 1,220 mm, so that two 610-mm by 610-mm of computer room air conditioning (CRAC) units. square raised-floor tiles separated the rows. Metrics similar to eCOM are commonly used to evaluate Owing to the flow rates and sizes involved in a data the energy efficiency of the data center infrastructure. center model, the flow was turbulent and was modeled Data center numerical modeling using the two-equation k-e turbulence model. The k-e model is the most appropriate, given the large scales and open spaces of the data center. In the numerical model, Raised-floor configuration the entire volume that is being modeled is discretized into Using numerical modeling techniques, Schmidt [1] and Schmidt and Cruz [2-6] studied the effects of various a large number of cells. The k-e model is used to calculate the fluid (air) viscosity parameter for each cell for the parameters on the inlet temperature of a raised-floor data turbulent flow regime. The steady-state form of the center. The numerical models were run using Flotherm 4.2 [26], a commercially available finite control volume computer code. The computational fluid dynamics (CFD) code used a k-e turbulence model3 in the flow solver. The data center consisted of 40 racks of data processing (DP) equipment, with front-to-back cooling, which were placed in rows in an alternating hot aisle/cold aisle arrangement located in the center of the floor with four CRAC units at the perimeter of the 12.1-m-wide by 13.4-m-long room. The chilled cooling air provided by the CRAC units was delivered through perforated tile openings via the air plenum under the floor. Only the above-floor (raisedfloor) flow and temperature distributions were analyzed in order to reduce the computation domain and time. By taking advantage of the symmetry in the model, only half of the data center had to be modeled. The results of these governing equations is standard and can be found in [28]. In each of the papers [1-6], the same nonuniform 135,000cell grid was used. An appropriate grid sensitivity study on the nonuniform grid can be found in [1]. The typical temperature difference in DP equipment and CRAC units is 108C. In order to maintain the 108C air temperature difference as the power dissipation of the DP racks increases, the flow rate through the rack must increase in direct proportion. This is not always the case in practice, since many DP racks with higher heat loads are now experiencing temperature differences as high as 158C to 208C. For a rack heat load of 4 kW, the resulting flow rate was 20 m3/min to maintain the 108C temperature difference across the DP rack. In this closed system, the mass flow rate under the floor had to match the CRAC floor rate. For the case in which the floor-tile 716 3In the k-e model, k is the turbulence kinetic energy and e is the turbulence dissipation rate [27]; this is the most popular numerical model used in commercial solvers. flow rate matched the rack flow rate, the CRAC units delivered 200 m3/min each. R. R. SCHMIDT ET AL. IBM J. RES. & DEV. VOL. 49 NO. 4/5 JULY/SEPTEMBER 2005 Three temperature monitor points were added to the study was that once the chilled air from the perforated inlets of each of the DP racks. One point was placed at tiles was drawn in by the lower portions of the rack, each of three levels above the raised floor: 1,000 mm, satisfying conservation of mass, the upper region of the 1,500 mm, and 1,750 mm. The papers [1-6] presented rack required that air be drawn in from other portions of data on the rise of the air temperature at the inlets of the the room, primarily heated exhaust air from the rear of DP racks over the perforated-tile air temperature. This the same rack. metric allowed for comparison of the effects of different Schmidt and Cruz [2] examined the effects of shifting changes to the data center and its DP equipment on some of the chilled air from the cold aisle to the hot aisle. the inlet temperature of the DP racks. DP equipment This was done in order to examine the effects of cable manufacturers require that certain inlet air temperature cutouts at the backs of the DP equipment and also to specifications be met in order for the DP equipment to determine whether the injection of cool air into the hot run reliably. Considering the amount of capital invested aisle helped reduce the recirculated air temperature at the in the DP equipment and the critical business applications inlets of the DP racks. The variables altered in this paper that are run on it, keeping it running reliably is a major consisted of three rack powers (4 kW, 8 kW, and 12 kW), concern for IT and facilities personnel. two CRAC unit locations (perpendicular and parallel to In [1], Schmidt investigated two DP rack heat the rows of DP racks), and nine variations of chilled loads (4 kW and 8 kW), two CRAC unit locations airflow rates exhausting through the perforated tiles. (perpendicular and parallel to the rows of DP racks), A fixed ceiling height of 2.74 m was chosen for all three ceiling heights (2.44 m, 2.74 m, and 3.05 m), and simulation runs. As in [1], the total chilled air provided to three variations of chilled airflow rates exhausting the data center by the CRAC units was varied to 80%, through the perforated tiles. The boundary conditions in 100%, and 120% of the DP rack flow rate, but it was the numerical model for the three variations in chilled distributed in three configurations to both the cold and airflow rates from the perforated tiles were achieved by hot aisles. In the first case, 100% of the chilled air exited adjusting these flow values in the model to be equal to in the cold aisle, while the second case delivered 75% of three fractions (80%, 100%, and 120%) of the airflow rate the flow to the cold aisle, with the remaining 25% exiting through the rack that results in a 108C increase in air in the hot aisle. In the third case, the chilled air was split temperature through the DP equipment rack. For evenly between the cold and hot aisles so that 50% example, in the 4-kW DP rack heat-load cases, each of the DP racks moved 20 m3/min of air of maintain a went to the cold aisle and 50% exited in the hot aisle. Exhausting the chilled air into the hot aisle did not help 108C temperature difference across the DP rack, and the reduce the DP rack inlet temperatures, and, in all cases, it perforated tile directly in front of each DP rack produced was better to have all of the available chilled air exit in the 16, 20, and 24 m3/min of chilled air for the 80%, 100%, cold aisle. and 120% cases, respectively. The flow through each of Schmidt and Cruz [5] studied the effect of situating the DP racks was left unchanged for this study. It was higher-powered (12-kW) DP racks among lower-powered discovered that hot-air recirculation created by two (4-kW) DP racks. Twelve high-powered DP racks were different means was the main cause of increased inlet arranged with half of them having a single high-powered temperatures to the DP racks. The first case was DP rack and the other half having two high-powered DP recirculation of hot exhaust air over the top of the DP racks. A fixed ceiling height of 2.74 m and a fixed total rack back into the front of the same rack, as may be perforated-tile flow of 80% of the total DP rack flow seen in Figure 6. In the second case, hot exhaust air rate were chosen for all simulation runs. Two different recirculated around the sides of the end DP racks. perforated-tile flow arrangements were studied; the first These recirculation cells were present in each case, but had an unrealistic increase in the perforated-tile flow in the recirculation over the tops of the DP racks was only the tile directly in front of the high-powered DP substantially more pronounced with increased ceiling racks, while the second arrangement distributed the height and in the perpendicular CRAC unit layout. DP additional flow across the cold aisle with the high- rack inlet temperatures increased as much as 98C when powered DP racks. In the first case, the inlet temperatures the ceiling height was increased from 2.44 m to 3.05 m. of the high-powered DP racks did not exceed 48C above The second means of hot-air recirculation was created the under-floor air temperature. However, compared from the hot exhaust of neighboring rows of DP racks. with the baseline case of a fully populated low-powered Decreasing the under-floor airflow rate also increased the DP data center, the temperatures of the low-powered inlet temperatures by increasing the amount of air that DP racks increased significantly at an inlet height of each DP rack pulled from the room. Increases in DP rack 1,500 mm, with most increases in the range of 58C to inlet temperatures were in the range of 108C to 158C in 108C and the maximum being 218C. At a height of both the 4-kW and 8-kW cases. The primary result of the 1,750 mm, the low-powered DP racks showed minimal 717 IBM J. RES. & DEV. VOL. 49 NO. 4/5 JULY/SEPTEMBER 2005 R. R. SCHMIDT ET AL. Table 2 Effect of removed racks on inlet air temperatures into high-powered racks (at height of 1,750 mm)--temperature reductions compared to baseline case (8C) [4]. No. of adjacent removed racks 1 2 3 4 One high-powered rack Average DT inlet (8C) Standard deviation in DT inlet (8C) 8.9 2.8 12.0 0.8 14.8 0.5 Two high-powered racks Average DT inlet (8C) Standard deviation in DT inlet (8C) 7.1 3.1 12.8 1.9 13.8 1.9 16.5 1.5 Table 3 Perforated tile flow rates (m3/min) to maintain an temperatures of not only the high-powered DP rack, but average inlet air temperature of 108C [3]. also all of the neighboring low-powered racks. The effects of removing DP racks on air inlet temperature into the Rack heat load (kW) Perforated-tile flow rate (m3/min) For DT 20.88C For DT 108C Increase in flow rate from tile (%) high-powered DP rack can be seen in Table 2. These results show that, as more and more DP racks are removed close to the high-powered DP rack, the inlet air temperatures of the high-powered DP rack decrease. The across rack across rack greatest improvement in inlet air temperatures to the 4 12.5 18 44 8 18 20.5 14 high-powered DP racks occurred when only one DP rack was removed adjacent to the high-powered DP rack. The low-powered DP racks situated on the same cold aisle 12 20 25.5 28 as the removed DP racks experienced reduced inlet air temperatures, with the DP racks closest to the removed changes compared with the baseline case. The inlet temperatures in the second case, where the additional chilled airflow was distributed along the cold aisle, showed a high dependency on the location of the highpowered DP racks. When the high-powered DP racks were placed near the symmetry line, the inlet temperatures at 1,750 mm increased as much as 248C over the under-floor air temperature. However, when the highpowered DP racks were near the ends of the row, the temperature rise was on average only 138C above the under-floor air temperature. At the height of 1,750 mm, DP racks experiencing the greatest reduction in inlet air temperature. Schmidt and Cruz [3] also examined the effects of reducing the airflow rate through the DP rack for a higher temperature rise across the DP rack. Although the airflow rate exiting the floor tiles matched the DP rack airflow rate, decreasing DP rack airflow rates [or increasing the change in temperature (DT) through the DP rack] increased DP rack air inlet temperatures. This result was unexpected, since recent studies predict that the air temperatures into the DP racks would have not varied much if the chilled airflow rate exiting the perforated tiles the temperature of the high-powered DP racks ranged from 38C to 58C above that of the low-powered DP matched the DP rack flow rate. However, for the same perforated-tile flow rate, decreasing the DP rack flow rate racks at the same height. This is a minimal difference resulted in decreased DP rack air inlet temperatures. For considering a factor of 3 increase in airflow and power example, for clusters of uniformly powered 8-kW DP dissipated. Low-powered DP racks located near the racks, the average inlet temperature of all DP racks at high-powered DP racks showed an increase in inlet a height of 1,750 mm decreased by 128C when the DP temperature at the 1,500-mm height, while those rack flow rate was cut in half. To maintain a given inlet farther away from the high-powered DP racks showed temperature, the chilled airflow rate exhausting from the a decrease. When the high-powered DP racks were near perforated tiles should increase with increased rack flow the ends of the rows, the inlet temperatures of the low- rate (see Table 3). powered DP racks decreased at a height of 1,750 mm, in Schmidt and Cruz [6] explored the effects of under-floor some cases by as much as 168C. On the basis of these air distribution on the above-floor DP equipment air inlet results, the best position for the high-powered DP racks temperatures. Earlier papers had focused only on the is near the end of the row. above-floor air and assumed a uniform perforated-tile air Schmidt and Cruz [4] found that removing DP racks distribution. The space underneath the raised floor was 718 adjacent to a high-powered DP rack reduced the air inlet modeled using a commercially available CFD code called R. R. SCHMIDT ET AL. IBM J. RES. & DEV. VOL. 49 NO. 4/5 JULY/SEPTEMBER 2005 Table 4 Average air inlet temperature rise (8C) for DP rack powers, different perforated-tile open areas, and raised-floor heights [6]. Rack power 4 kW Variation in rack air inlet height (mm) 1,750 1,500 1,000 Uniform 16.1 6.6 0.0 Rack power 8 kW Variation in rack air inlet height (mm) 1,750 1,500 1,000 Uniform 7.7 3.5 1.2 Rack power 12 kW Variation in rack air inlet height (mm) 1,750 1,500 1,000 Uniform 6.9 4.2 1.8 Average air inlet temperature for different perforated-tile free areas and different raised-floor plenum heights (8C) 25% free area in perforated tile 60% free area in perforated tile For height of 0.15 m For height of 0.30 m For height of 0.60 m For height of 0.15 m For height of 0.30 m For height of 0.60 m 15.3 15.8 16.0 9.8 13.4 14.2 6.1 6.5 6.5 5.9 6.3 6.2 0.0 0.0 0.0 1.0 0.2 0.0 Average air inlet temperature for different perforated-tile free areas and different raised-floor plenum heights (8C) 25% free area in perforated tile 60% free area in perforated tile For height of 0.15 m For height of 0.30 m For height of 0.60 m For height of 0.15 m For height of 0.30 m For height of 0.60 m 3.2 3.5 3.5 2.3 3.0 3.2 1.9 2.1 2.1 1.9 2.1 2.0 1.2 1.0 1.0 1.8 1.3 1.0 Average air inlet temperature for different perforated-tile free areas and different raised-floor plenum heights (8C) 25% free area in perforated tile 60% free area in perforated tile For height of 0.15 m For height of 0.30 m For height of 0.60 m For height of 0.15 m For height of 0.30 m For height of 0.60 m 2.3 2.3 2.3 1.5 1.9 2.3 1.7 1.7 1.7 1.4 1.8 1.9 1.2 1.2 1.2 1.7 1.4 1.2 Tileflow**. The CFD calculation requires the solution of For the 4-kW case, the more poorly distributed the the three-dimensional turbulent flow in the under-floor airflow, the lower were the average DP rack inlet space. As in the above-floor CFD model, the turbulence is temperatures (see Table 4). Slight variations in poorly characterized by the k-e turbulence model. The inflow to distributed tile airflows (less than 8%) resulted in the calculation domain is created by the CRAC units, significant inlet temperature reductions of up to 88C while the outflow is governed by the local pressure under for the 8-kW case and 108C for the 12-kW case. the perforated tiles and the resistance of the perforated tiles. The computation takes into account the flow Non-raised-floor configuration resistance due to under-floor obstructions (such as cables A representative non-raised-floor data center system, and pipes) and the resistance of the support structure for depicted in Figure 4(d), has been modeled using the the raised floor. Three DP rack powers (4 kW, 8 kW, and commercial software tool Flotherm. The rack layout for 12 kW), two perforated-tile percentage-open areas (25% this analysis is identical to that for the raised-floor studies and 60% open), and three raised-floor heights (0.15 m, as conducted [1-6] and illustrated in Figure 8. The non- 0.30 m, and 0.60 m) were examined. For the same data raised-floor model, comprising a half-symmetry section of center total airflow rate, the smaller the plenum height a 40-rack data center arranged in a cold-aisle/hot-aisle and the greater the perforated-tile percentage open, the fashion, was organized as two rows of six racks and two more poorly distributed was the airflow exiting the tiles. rows of four racks, respectively. Overhead ducts that feed 719 IBM J. RES. & DEV. VOL. 49 NO. 4/5 JULY/SEPTEMBER 2005 R. R. SCHMIDT ET AL. Diffuser ducts Return vents for hot exhaust air Cold aisles Computer (chilled-air supply) equipment racks Figure 9 Schematic of non-raised-floor model for computer-based simulations. chilled air to the diffusers were constructed as part of the model. Return air vents were placed at either the bottom or the top of the bounding walls of the model and were located on the wall perpendicular to the hot and cold aisles, or on the walls parallel to the aisles. Figure 9 illustrates the results of a simulation using the non-raised-floor CFD model developed for this study. Symmetry boundary conditions were enforced on the center wall that is in contact with the racks. The fluid flow was numerically solved using the k-e turbulence model. The size of the model ranged from 150,000 to 200,000 elements. Satisfactory convergence for temperature and continuity was typically achieved within 1,000 iterations, and required approximately two to four hours on a high-end PC. Temperature data was recorded for the numerical simulations at five specific locations in front of the rack, namely at vertical heights of 330, 660, 1,000, 1,333, and 1,666 mm. Average temperatures for five volumetric regions at the inlet to the rack were also ascertained. A parametric study is currently being conducted to characterize the effect on the temperature of the inlet air to the rack. Table 5 provides detailed information regarding variables that were considered for the study. As seen in the table, the input variables to the design of experiments are the rack heat load (32), rack flow rate, air temperature rise through the rack (31), diffuser flow rate (36), diffuser location (height) (33), diffuser pitch and arrangement (34), angle of air exiting the diffuser (35), room ceiling height (37), and return vent location for the exhausting air (38). In addition to these variables, special configurations that characterize zero power racks (39 alternate zeros) as well as nonuniform rack heat load (39 alternate halves) are incorporated into the study. Statistical techniques are to be utilized to analyze the data, and the results are to be used to formulate quantitative and qualitative design guidelines. Data center energy requirements The amount of electric power drawn by computer hardware can be very large, and the power required for the cooling can add an additional 30%-50%. For example, the NCEP data center described above and depicted in Figure 6 requires approximately 0.5 MW of power for the computer equipment installed within the Table 5 Parametric variables for non-raised-floor study. Variable description Variation Name Description Units 1 2 3 4 5 31 Rack temperature rise 8C 10 15 20 NA NA 32 Rack heat load kW 4 12 20 28 36 33 Diffuser height m 3.05 At ceiling NA NA NA 34 Diffuser location -- Alternate 1 Alternate 2 NA NA NA 35 Diffuser angle degrees 0 30 NA NA NA 36 Total diffuser flow rate % of rack 100 80 flow rate 60 NA NA 37 Ceiling height m 3.66 5.18 6.71 NA NA 38 Return air vent location -- Bottom of Top of Bottom of Top of NA perpendicular perpendicular parallel wall parallel wall wall wall 39 Rack heat load nonuniformity -- Uniform Alternate zeros Alternate halves NA NA 720 NA: Less than five variations considered. R. R. SCHMIDT ET AL. IBM J. RES. & DEV. VOL. 49 NO. 4/5 JULY/SEPTEMBER 2005 data center. Although a detailed study was not performed on the energy required for cooling, it is estimated that this energy was in the range of 150 kW to 250 kW. On an annual basis, at a rate of 10 cents per kilowatt-hour, this results in an annual energy expense of $130,000 to $220,000 per year just for cooling. This is a large expense devoted to cooling the equipment and, considering the number of data centers in the world, requires that a significant effort be put into optimizing the use of this energy. Such an effort requires a detailed examination of the various inefficiencies that occur in a data center in order to determine how to decrease the amount of energy spent. The design and layout of data centers and their cooling infrastructure are inherently wasteful of energy. Bash et al. [10] outlined three key factors in lowering energy usage and thereby maximizing performance: 1. Minimize the infiltration of hot air into the rack inlets. 2. Minimize the mixing of hot return air with cold-air streams prior to return to the CRAC units. 3. Minimize the short-circuiting of cold air to the CRAC inlet. They postulated that research in the area of airflow optimization in the data center space was lacking. They maintained that in the absence of airflow research data, air-handling infrastructure in data centers has suffered from overdesign and extravagant redundancy. A methodology for understanding this was outlined and tested on a simple data center model in [29]. From this analysis, Bash et al. [10] focused on the following areas: 1. Three-dimensional numerical modeling of fluid mechanics and heat transfer within data centers, with emphasis on optimization and real-time solutions. 2. Distributed sensing and control of cooling resources within data centers. 3. Development of metrics and correlations to guide data center design and predict performance. on the basis of the most energy-efficient availability of cooling resources. In addition to the optimization that should occur within the data center, solutions should also be pursued to reduce the energy that is supplied to the facilities. Felver et al. [30] showed that a new data center under design for Sandia National Laboratories in Livermore, California, will use variable-air-volume air-handling units that provide backup evaporative cooling. This proposed two-stage evaporative cooling design offers a major benefit to both utilities and data centers by shifting the peak electrical demand of the HVAC system to an offpeak period. This minimizes the burden placed on them by the data centers while allowing electrical power to be purchased by the data center at a reasonable rate during periods of peak demand. The use of this technique, through the elimination of CRAC units, allows for a recapturing of valuable computer-center floor space and a reduced risk of water damage in computer rooms. Future data centers Each data center is unique and has its own limitations; for example, data center power density limits vary greatly. To resolve the environmental issues, manufacturers of HVAC equipment have begun to offer liquid cooling solutions to aid in data center thermal management. The objective of these new approaches is to move the liquid cooling closer to the electronic equipment that is producing the heat. Placing cooling near the source of heat shortens the distance that air must be moved and minimizes the required static pressure. This increases the capacity, flexibility, efficiency, and scalability of the cooling solutions. Several viable options based on this strategy have been developed: 1) rear-mounted fin and tube heat exchangers; 2) internal fin and tube heat exchangers, either at the bottom of a rack of electronic equipment or mounted on the side of a rack; and 3) overhead fin and tube heat exchangers. Although each solution involves liquid cooling adjacent to the air-cooled rack, the liquid can be either water-based or refrigerantbased. New approaches such as these and others will continue to be promoted as heat loads of data communication equipment continue to increase. Although much work remains to be done in this area, their view of future progress was outlined in [9]. A cooling system comprising variable-capacity computer room air conditioning units, variable air-moving devices, Summary With the increased performance requirements of computer equipment and the resulting heat dissipated by this equipment, a significant strain is placed on the data adjustable vents, etc., which are used to dynamically center and its environment. Air cooling for some data allocate air conditioning resources where and when centers has reached its limit; racks of equipment with needed could be installed. A distributed metrology layer high-density loads must be distributed among other racks could be used to sense environment variables such as with less dense equipment loads in order to spread out the temperature, pressure, and power. The data center energy heat and make the data center ``coolable.'' The value manager would then redistribute the computer workloads proposition of packing the racks as close as possible 721 IBM J. RES. & DEV. VOL. 49 NO. 4/5 JULY/SEPTEMBER 2005 R. R. SCHMIDT ET AL. for maximum performance is significantly diminished Rates Are Reduced,'' presented at the Interpack Conference, because the air conditioning capacity within a data center is not sufficient to permit such an arrangement. Maui, Hawaii, July 6-13, 2003. 4. R. Schmidt and E. Cruz, ``Raised Floor Computer Data Center: Effect on Rack Inlet Temperatures When Adjacent These cases require closer examination and a better understanding of the flow and temperature dynamics within each data center. Because of the extreme Racks Are Removed,'' presented at the Interpack Conference, Maui, Hawaii, July 6-13, 2003. 5. R. Schmidt and E. Cruz, ``Raised Floor Computer Data Center: Effect on Rack Inlet Temperatures When High complexity of this problem and the uniqueness of every data center, this effort is only just beginning. Much more thermal data from data centers must be collected, Powered Racks Are Situated Amongst Lower Powered Racks,'' Proceedings of the ASME International Mechanical Engineering Congress (IMECE), New Orleans, November 2002, pp. 297-309. examined, correlated, and understood, along with detailed thermal and fluid models. Non-dimensional metrics must be identified to provide guidance for data 6. R. Schmidt and E. Cruz, ``Clusters of High Powered Racks Within a Raised Floor Computer Data Center: Effect of Perforated Tile Flow Distribution on Rack Inlet Air Temperatures,'' Proceedings of the ASME International center thermal management. As more is understood of data center thermal behavior, energy optimization and savings may follow. Since data centers typically require Mechanical Engineering Congress (IMECE), Washington, DC, November 15-21, 2003, pp. 245-262. 7. C. Patel, C. Bash, C. Belady, L. Stahl, and D. Sullivan, ``Computational Fluid Dynamics Modeling of High Compute an additional 30% to 50% of the data processing energy for cooling, it may be possible to save a large amount of energy in the thousands of data centers around the world. Density Data Centers to Assure System Inlet Air Specifications,'' Advances in Electronic Packaging, Proceedings of the Pacific Rim/ASME International Electronic Packaging Technical Conference and Exhibition (IPACK'01), Kauai, Significant savings can be obtained if an improvement of as little as 5% is achieved in energy-related expenditures. As equipment power densities increase along with Hawaii, July 8-13, 2001, Vol. 2, Paper IPACK2001-15662, pp. 821-829. 8. C. Patel, R. Sharma, C. Bash, and A. Beitelmal, ``Thermal Considerations in Cooling Large Scale Compute Density Data expenditures for energy to power this equipment, the HVAC industry as well as the server manufacturers may have to embark on liquid cooling solutions in order to Centers,'' Proceedings of the Eight Inter-Society Conference on Thermal and Thermomechanical Phenomena in Electronic Systems (ITherm), San Diego, June 2002, pp. 767-776. 9. C. Patel, C. Bash, R. Sharma, M. Beitelmal, and R. Friedrich, resolve some of the future temperature problems that will occur within the server racks and the environment in which they reside. ``Smart Cooling of Data Centers,'' Advances in Electronic Packaging--Proceedings of the Pacific Rim/ASME International Electronic Packaging Technical Conference and Exhibition (IPACK'03), Maui, Hawaii, July 6-11, 2003, pp. 129-137. Note added in proofs [31] A water-cooled heat exchanger attached to the rear door 10. C. Bash, C. Patel, and R. Sharma, ``Efficient Thermal Management of Data Centers--Immediate and Long Term Research Needs,'' HVAC&R Res. J. 9, No. 2, 137-152 (2003). of the rack reduces the effect of the hot-air recirculation. Extraction of heat at the point of generation is much more efficient than with CRACs positioned around the 11. H. Obler, ``Energy Efficient Computer Cooling,'' Heating Piping, & Air Cond. 54, No. 1, 107-111 (1982). 12. H. F. Levy, ``Computer Room Air Conditioning: How To Prevent a Catastrophe,'' Building Syst. Design 69, No. 11, perimeter of a data center. A total cost-of-ownership analysis was performed which showed a significant advantage in both first-time cost and annual cost when 18-22 (1972). 13. R. W. Goes, ``Design Electronic Data Processing Installations for Reliability,'' Heating, Piping, & Air Cond. 31, No. 9, 118-120 (1959). this attached heat exchanger approach is used. 14. W. A. Di Giacomo, ``Computer Room Environmental Systems,'' Heating, Piping, & Air Cond. 45, No. 11, 76-80 (1973). *Trademark or registered trademark of International Business 15. J. M. Ayres, ``Air Conditioning Needs of Computers Pose Machines Corporation. Problems for New Office Building,'' Heating, Piping, & Air Cond. 34, No. 8, 107-112 (1962). **Trademark or registered trademark of Innovative Research, Inc. 16. M. N. Birken, ``Cooling Computers,'' Heating, Piping, & Air Cond. 39, No. 6, 125-128 (1967). References 17. F. J. Grande, ``Application of a New Concept in Computer Room Air Conditioning,'' Western Electric Engineer 4, No. 1, 1. R. Schmidt, ``Effect of Data Center Characteristics on Data 32-34 (1960). Processing Equipment Inlet Temperatures,'' Advances in 18. F. Green, ``Computer Room Air Distribution,'' ASHRAE J. 9, Electronic Packaging, Proceedings of the Pacific Rim/ASME No. 2, 63-64 (1967). International Electronic Packaging Technical Conference and 19. H. F. Levy, ``Air Distribution Through Computer Room Exhibition (IPACK'01), Kauai, Hawaii, July 8-13, 2001, Vol. Floors,'' Building Syst. Design 70, No. 7, 16 (1973). 2, Paper IPACK2001-15870, pp. 1097-1106. 20. The Uptime Institute; see the whitepaper,``Heat Density 2. R. Schmidt and E. Cruz, ``Raised Floor Computer Data Trends in Data Processing, Computer Systems and Center: Effect on Rack Inlet Temperatures of Chilled Air Telecommunication Equipment,'' www.uptimeinstitute.org, Exiting Both the Hot and Cold Aisles,'' Proceedings of the 2000. Eight Inter-Society Conference on Thermal and 21. American Society of Heating, Refrigerating and Air- Thermomechanical Phenomena in Electronic Systems Conditioning Engineers (ASHRAE), Thermal Guidelines for (ITherm), San Diego, June 2002, pp. 580-594. Data Processing Environments, 2004; see www.ashraetcs.org. 3. R. Schmidt and E. Cruz, ``Raised Floor Computer Data 22. Datacom Equipment Power Trends and Cooling Application, 722 Center: Effect on Rack Inlet Temperatures When Rack Flow ASHRAE Publication, February 2005. R. R. SCHMIDT ET AL. IBM J. RES. & DEV. VOL. 49 NO. 4/5 JULY/SEPTEMBER 2005 23. R. Schmidt, "Thermal Profile of a High Density Data Center--Methodology to Thermally Characterize a Data Center," Proceedings of the ASHRAE Nashville Conference, May 2004, pp. 604-611. 24. M. Norota, H. Hayama, M. Enai, and M. Kishita, "Research on Efficiency of Air Conditioning System for Data Center," Proceedings of the IEICE/IEEE International Telecommunications Energy Conference (INTELEC), October 19-23, 2003, pp. 147-151. 25. R. Sharma, C. Bash, and C. Patel, "Dimensionless Parameters for Evaluation of Thermal Design and Performance of Large Scale Data Centers," Proceedings of the American Institute of Aeronautics and Astronautics (A/AA), 2002, p. 3091. 26. Flomerics, Flotherm 4.2 Reference Manual; see www.flotherm.com. 27. S. Patankar, Numerical Heat Transfer and Fluid Flow, First Edition, Taylor and Francis Publishers, 1980; ISBN: 0-891-16522-3. 28. R. Schmidt, "Thermal Management of Office Data Processing Centers," Advances in Electronic Packaging--Proceedings of the Pacific Rim/ASME International Electronic Packaging Technical Conference (INTERpack'97), Kohala Coast, Hawaii, June 15-19, 1997, Vol. 2, EEP-Vol 19-2, pp. 1995-2010. 29. A. Shah, V. Carey, C. Bash, and C. Patel, "Energy Analysis of Data Center Thermal Management Systems," Proceedings of the ASME International Mechanical Engineering Congress (IMECE), Washington, DC, November 15-21, 2003, pp. 437-446. 30. T. Felver, M. Scofield, and K. Dunnavant, "Cooling California's Computer Centers," HPAC Eng., pp. 59-63 (2001). 31. R. Schmidt, R. C. Chu, M. Ellsworth, M. Iyengar, D. Porter, V. Kamath, and B. Lehmann, "Maintaining Datacom Rack Inlet Air Temperatures with Water Cooled Heat Exchangers," Proceedings of IPACK2005/ASME InterPACK'05, San Francisco, July 17-22, 2005, in press. Received October 4, 2004; accepted for publication March 2, 2005; Internet publication September 15, 2005 Roger R. Schmidt IBM Systems and Technology Group, 2455 South Road, Poughkeepsie, New York 12601 ( @us.ibm.com). Dr. Schmidt is an IBM Distinguished Engineer, a National Academy of Engineering Member, an IBM Academy of Technology Member, and an ASME Fellow; he has more than 25 years of experience in engineering and engineering management in the thermal design of large-scale IBM computers. He has led development teams in cooling mainframes, client/servers, parallel processors, and test equipment utilizing such cooling mediums as air, water, and refrigerants. He has published more than 70 technical papers and holds 51 patents in the area of electronic cooling. He is a member of the ASME Heat Transfer Division and an active member of the K-16 Electronic Cooling Committee. Dr. Schmidt is chair of the IBM Corporate Cooling Council and the Power Packaging and Cooling Technical Community Council. He has been an Associate Editor of the Journal of Electronic Packaging and is currently an Associate Editor of the Ashrae HVAC&R Research Journal. Over the past 20 years, he has taught many mechanical engineering courses for prospective professional engineers and has given seminars on electronic cooling at a number of universities. Ethan E. Cruz IBM Systems and Technology Group, 2455 South Road, Poughkeepsie, New York 12601 ( @us.ibm.com). Mr. Cruz received his B.S. degree in mechanical engineering from the University of Texas at Austin in 2000. He is a Staff Engineer in the Thermal Engineering and Technologies Department in the IBM Systems and Technology Group. Since joining IBM in 2000, Mr. Cruz has worked as an equipment engineer; he is currently a thermal engineer developing cooling solutions for the high-end pSeries and zSeries* servers. He is responsible for the servers' air- moving systems, bulk power cooling, and thermal-acoustic covers and specializes in the optimization of centrifugal blower designs. Mr. Cruz is a member of ASME and has co-authored several technical publications. Madhusudan K. Iyengar IBM Systems and Technology Group, 2455 South Road, Poughkeepsie, New York 12601 (M@us.ibm.com). Dr. Iyengar works as an Advisory Engineer in the IBM Systems and Technology Group in Poughkeepsie, New York. He joined IBM in October 2003. He received his B.E. degree in mechanical engineering from the University of Pune, India, in 1994, and then worked from 1994 to 1995 at Kirloskar Oil Engines, Pune, India. He received his M.S. and Ph.D. degrees in mechanical engineering from the University of Minnesota in 1998 and 2003, respectively, and then joined the mechanical engineering faculty at Purdue University as a postdoctoral research associate conducting research from April to October of 2003. Dr. Iyengar has ten archival journal publications and nineteen refereed conference papers. His technical interests include thermal design and optimization, cooling of electronics, data center thermal management, mathematical optimization, product design for sustainability, and design for manufacturability. IBM J. RES. & DEV. VOL. 49 NO. 4/5 JULY/SEPTEMBER 2005 R. R. SCHMIDT ET AL.