CS Department NFS mounts are down
- Posted by Donald Johwa
- on June 9, 2020
The CS Department filesystem mounts are currently unavailable. We are working on the problem, and will update you once we have resolved the issue.
UPDATE - June 9 9:20am
CIS has identified the problem and are working on a solution.
UPDATE - June 10 2:20pm
With this morning's CIS firewall upgrade, which was previously scheduled, the immediate problem affecting our access to CIS file services was resolved by 8am. Most CS linux desktop systems remained unreachable, however, and require a reboot. This includes the compute cluster (grid). Most systems to which remote users connect were rebooted, by hand, later this morning. We will be using the overnight update, which runs at 4am, to reboot the remaining systems. There may be some stragglers and we will reboot those later, by hand. If you need a particular machine rebooted right away, please email problem.
Further Details
In preparation for today's (Wednesday) scheduled CIS firewall upgrade, CIS updated the firewall firmware on Tuesday morning. As soon as they did, the "fastpath" pathway through which we access CIS file services stopped working. According to Cisco, this was due to a software bug - the new firmware and the old firewall software are incompatible. This morning's firewall software upgrade resolved that issue.
The extended outage left our desktop systems inaccessible via ssh. For those machines that were not rebooted by pressing a button, we will use our configuration management system to reboot. It runs once a day at 4am, so overnight tonight is our next opportunity to restart those systems. In some cases a soft reboot is not sufficient, and for those machines someone has to press the reset button. We'll hunt those down tomorrow (Thursday).