During the final weeks of the semester, the technical staff encountered issues with routine maintenance of our GPFS filesystem. IBM suggested that there was corruption in the filesystem, and recommended that we take it offline to repair it. We postponed that work until after commencement, to avoid disrupting the final weeks of classes, a conference deadline and commencement activities. Meanwhile, disk usage has been approaching 100%.
This past weekend, we carried out the recommended filesystem repair procedure. IBM estimated it would take 16 hours, but the process hung, and we stopped it to restore services in time for the business day on Monday, June 3rd. It's not clear what good it did, and we find ourselves in the same critical situation as before. IBM has suggested that we will have to schedule more downtime in the very near future.