The École polytechnique fédérale de Lausanne (EPFL) in Switzerland has improved its power usage effectiveness, achieved space savings and has seen a reduction of CAPEX and OPEX, after working with Vertiv.
The École polytechnique fédérale de Lausanne (EPFL) is – in addition to the many traditional universities and universities of applied sciences controlled by the cantons – one of the two Federal Institutes of Technology in Switzerland and is headquartered in Lausanne.
Since its foundation in 1969, it has become one of the best-known institutions for science and technology in Europe. Like its sister institute in Zurich, the ‘ETHZ’, it has three main goals: education, research and technology transfer.
More than 14,000 lecturers, researchers and students work, teach and conduct research on the main campus.
Background
To ensure the many different innovative and future-oriented projects being carried out by researchers and students are supported with the appropriate IT equipment, the EPFL has IT and building infrastructure teams made up of around 200 employees.
Aristide Boisseau, Head of the Datacenter Operations team, plays a particularly important role. Together with his team, he acts as the interface between the two domains, ensuring a regular exchange of information and smooth processes.
High-performance computers are predominantly used at the EPFL to provide the capacity required for scientific calculations.
While the EPFL’s budget for new high-performance computers has remained the same over recent years, providers have nevertheless increased the capacity of their products in the data centre.
As a result, the power density per square meter has risen and the existing ventilation-cooling systems have not been able to keep pace with this growth. A new, intelligent solution capable of adequately cooling the eight-point grid systems was therefore required.
Challenges and objectives
Increasing data and computation capacities require higher power density of high-performance computers, greater energy efficiency and cooling solutions that occupy less space.
The EPFL’s image is one of a university that educates the brightest minds of the future, enabling them to produce creative approaches to many current challenges, in areas such as mobility, urban planning, technology and science etc.
Students not only acquire new knowledge but also manage their own projects together with the professors. Measurements, evaluations and findings – all of this data has to be stored, processed and retained.
Applications, programs and databases enrich and simplify the work of students and professors but at the same time means greater complexity for IT.
The upshot is a massive amount of data which lays the foundation for new discoveries. The EPFL operates its own data centres to meet the growing requirements.
The server capacity of the two existing data centres has been increasingly expanded over recent years which eventually placed too much strain on the cooling solutions.
In 2011, the EPFL decided to build another data centre to keep pace with the increasing volumes of data. Boisseau began to look for a new cooling solution for the two existing data centres, one of which is used for the processing and storage of data and the other as back-up.
He had previously relied on a circulation cooling system. Water from Lake Geneva was used to cool down the air which in turn cooled the racks. However, this old solution only produced eight to ten kW per rack whereas 30 kW per rack was required for high-performance computers. This unwieldy system also took up too much space at the data centre.
Boisseau contacted the experts at Vertiv – which was called Knürr at the time – about the new racks to be integrated into the new data centre. The cooling options were also discussed. It soon became apparent that Vertiv’s racks could provide a space-saving cooling solution with the cooling output required.
The concept: Water cooling of the racks with the Knürr DCD air-water cooling doors from Vertiv
Boisseau initially thought that two different solutions would be needed to meet the individual power density requirements in the racks. This is because, in addition to the high-performance computers, the EPFL uses racks where output amounts to a maximum of 10 kW. The initial plan was to use the CoolTherm solution for high density (up to 30kW) and hot-air containment for lower needs (up to 10kW).
After consulting with Vertiv’s experts in depth, it was decided that the Knürr DCD Water Cooled Rack Doors from Vertiv were the best cooling solution for all the racks.
The air-water cooling door from Vertiv is an air-water heat exchanger integrated in the rear door of the server rack. The cold water is used in a closed heat exchanger system directly behind the servers in the rear door of the rack. All the heat of the integrated IT equipment is dissipated via the water circuit which feeds the rack door with cold water and carries away warmer water. The cooling effect occurs when the warm outgoing air flows through the heat exchanger in the rear section of the server cabinet. The cooling air flow is driven solely by the server fans.
Additional fans in the cooling door are not required. Boisseau initially decided to purchase three Liebert Smart Rack PDUs MPX from Vertiv and to test the DCD air-water cooling doors in a two-month trial period. The result of the proof-of-concept was clear: with 20 to 25 kW per rack, the new solution had still not exceeded the power limit. Another benefit was the huge space saving.
The price was also unbeatable which meant the decision was quickly made and the new data centre was equipped with just under 30 racks.
The results: Higher performance densities for the servers, cost reductions, space saving and greater energy efficiency
The water cooling solution has also resulted in a lower risk of failure, greater system reliability and fewer error sources than with a solution based purely on air cooling. This type of cooling also reduces energy consumption through electrical loss and the cooling process.
The optimised heat exchanger structure and short air paths without diversions also mean there is no need for additional fans in the cooling unit. Cooling via the DCD air-water cooling door achieves power usage effectiveness (PUE) values of 1.06. The same pipes are used for the new water cooling system as those previously deployed for the transport of water from Leman Lake, thus reducing investment costs.
As the water is used directly in the DCD air-water cooling rear door rack exchanger, the EPFL used 30% to 50% less space compared to conventional air cooling systems despite integrating more servers.
Today, up to 90 DCD Cooling Doors carry out the cooling of the 300 m2 data centre. The temperature is regulated via thermostatic valves which control the water flow in the heat exchanger depending on the current load in the rack.
This means the water volume can be adjusted to the power density in the racks and vice-versa – the DCD Cooling Doors can be deployed as the sole cooling solution in the entire data center regardless of the various performance densities in the racks.
“When it comes to cooling servers and racks with different power densities or high-performance servers, Vertiv’s water cooling solution with the DCD air-water cooling doors is an outstanding solution,” said Aristide Boisseau Data Center Architect and Operations at the EPFL.
“It stands out due to its low PUE values, higher energy efficiency, lower costs and easy installation compared to conventional air-cooling systems. If the volumes of data to be processed and power densities at data centers increase, data centre managers should definitely consider this innovative and future-oriented method of cooling.”
Looking ahead
In 2015, Mr Boisseau realised that the capacity of the new data centre commissioned in 2012 was running short. One of the older data centres also had to be completely upgraded to meet the increasing requirements.
He therefore began planning a new data centre which is set to be put into operation by 2021 at the latest and to upgrade the old data centre.
In accordance with WTO guidelines, the EPFL as a government-run university has to put out a new invitation to tender for the construction of the new data centre.
Vertiv will take part in the tender process with its innovative and application-oriented water cooling solutions.
