Technical Staff Blog

Category archives: Service Issues

Tstaff announcements related to general IT service issues

RSS feed of Service Issues

Last update on .

owncloud.png 08:54: We are experiencing some database issues with our ownCloud service that could affect syncing back to the servers. The database cluster is managed by CIS, so we are working with their DBAs to diagnose and fix the problems.

09:22: We have shut down one of the backend servers to aid in debugging ...

Last update on .

openvpn.png We are continuing to experience some authentication issues with our OpenVPN server. A recent upgrade has introduced a bug that leaks file handles, ultimately leading to system resource starvation. We've filed a bug report, but have yet to hear back from the software maintainers. While we continue to investigate this bug and/or find ...

Last update on .

openvpn.png 09:57: We experienced some authentication issues with our VPN server. The process has been restarted and we are investigatng the root cause.

17:03: It looks like there is a libgcrypt bug that crops up when you use openVPN and authenticate against LDAP over TLS, which is exactly our setup. This bug results in ...

Last update on .

padlock.jpg Google posted to their online security blog earlier today about an interesting discovery one of their engineers made while trying to debug an SSH problem. After some investigation, they determined it was actually due to a glibc blug that could be exploited. The patches rolled out in coordination with this blog post. Unfortunately, it does ...

Last update on .

ibm-gpfs.jpg The department experienced a GPFS blip shortly after 10am this morning. The outage become pregressively worse and lasted about 10 minutes in total.

For those interested in more details, one of the quorum and manager nodes for the cluster experienced a system hard drive failure a few days ago. After replacing the hard drive and ...

Last update on .

Eiki LC-WUL100A.jpg The projector in CIT 316 is experiencing some hardware issues and should be considered unreliable. We have ordered a replacement, but not yet have an ETA for arrival. If you have this room booked in the next week or so and are in need of projection, we encourage you to reach out to CS reception ...

Last update on .

network-2400px.png (10:35) Around 9:15am this morning, a network card in a CIS switch failed that connects our primary firewall's external network interface. This necessitated someone going onsite to manually force our firewalls to failover. Service was restored around 10:30am. We are in the process of checking various services to make sure that ...

Last update on .

logo_sympa_small.png At 07:38 this morning we discovered a dead process on our list server. Upon restarting the process a flurry of delayed emails were sent. We are investigating the root cause and, perhaps more importantly, why automated monitoring did not properly detect and correct the dead process.

Last update on .

xen_project_logo_dualcolor_165x69.png One of our Xen hosts seems to have had a disk failure over the past 12hrs, which resulted in nearly all of it's virtual machines going offline.  We have brought this machine back online temporarily while we work to temporarily move those virtual machines to a new host. 

The affected virtual machines are:

anvil ...