That maintenance period has finished, and the datacenter temperatures appear to be stable. At this point, we are returning the datacenter to normal operations. If you have any questions or further issues, please contact us at service@discovery.wisc.edu.
Posted Nov 22, 2021 - 14:15 CST
Update
We're now on the temporary chilled water solution that we will be using for the months that campus will be performing maintenance on the chilled water loop, so we've begun bringing up compute resources. At this point, all of the Discovery research compute resources are online, and CHTC has started to bring up their resources in staged.
However, we've been informed that there may be another 20-30 minute window where we may experience chilled water interruption. We will be monitoring cooling in the Datacenter and we expect that we will not have to bring anything down again, but we will update here if that situation changes.
Posted Nov 22, 2021 - 12:15 CST
Update
The steamfitters returned our access to campus chilled water loop temporarily, so we are bringing up some compute resources for the weekend. At the moment, we are turning on wid-submit.discovery.wisc.edu and the CHTC Tiger cluster nodes.
Note that we will still have another outage on Monday morning and any compute services we turn back on now will have to be shut down again before that outage. Again, if you have an urgent need for something to be on over the weekend, let us know at service@discovery.wisc.edu and we can determine if we can bring it back online.
Posted Nov 19, 2021 - 18:19 CST
Update
Due to a change in the steamfitters' plan, we will not be back on a stable cooling solution until Monday. Until they're finished with their maintenance, we do not plan to bring up any services that are currently down.
If there's a server or service that is urgently needed, please contact service@discovery.wisc.edu and we may be able to bring it back online. All other services are expected to return on Monday when the maintenance on the cooling system is completed.
Posted Nov 19, 2021 - 12:37 CST
Update
Temperatures in the datacenter are still rising slowly, so we've started to bring down a few more services.
For now, we're bringing down the rest of CHTC's resources, plus the roy-fs.discovery.wisc.edu file server. We'll continue to update here as things change.
Posted Nov 19, 2021 - 09:26 CST
In progress
Scheduled maintenance is currently in progress. We will provide updates as necessary.
Posted Nov 19, 2021 - 05:01 CST
Scheduled
In the coming weeks, campus will begin a chilled water valve replacement project that impacts many buildings, including Discovery. On Friday, November 19th, a temporary chiller will be activated in Discovery to provide continued support and cooling to critical areas in the building while the valve replacement project is occurring.
The transition from the campus chilled water to the temporary chiller is expected to take about 8-12 hours and will start at 6am on Friday, November 19th.
We will be proactively shutting down Research Compute servers, but Research Storage servers and all other infrastructure will remain up. We will update this space if temperatures rise too much and we need to take down more services.
Posted Nov 12, 2021 - 15:22 CST
This scheduled maintenance affected: Servers, Hardware, and Software (Research Computing), Data Center (CHTC - Computational Clusters), and Data Storage (Research Storage).