Technical Staff Blog

Category archives: Service Issues

Tstaff announcements related to general IT service issues

RSS feed of Service Issues

Last update on .

network-2400px.png This was supposed to be a trouble free morning... just power up the switch we moved downstairs yesterday and add our redundancy back in. It seems to be a trend with this project, things not going quite as we planned...

07:14: The second distribution switch was powered on at 7am. Unfortuantely, something happened that ...

Last update on .

network-2400px.png 09:53: We are currently experiencing a widespread network outage with any site outside the CS department, including wireless. Tstaff is trying to confirm it's not an issue on our end and we have reached out to the CIS networking team to get their help debugging this.

10:06: We lost the network link ...

Last update on .

postfix.png As part of our ongoing virtualization efforts, we rolled out an upgraded version of our mail relay server yesterday at noon. Unbeknownst to us, the default value for a configuration variable changed in a subtle way that affected delivery to some subdomains. This definitely affected delivery to some of our email lists. We put a ...

Last update on .

owncloud.png 08:54: We are experiencing some database issues with our ownCloud service that could affect syncing back to the servers. The database cluster is managed by CIS, so we are working with their DBAs to diagnose and fix the problems.

09:22: We have shut down one of the backend servers to aid in debugging ...

Last update on .

openvpn.png We are continuing to experience some authentication issues with our OpenVPN server. A recent upgrade has introduced a bug that leaks file handles, ultimately leading to system resource starvation. We've filed a bug report, but have yet to hear back from the software maintainers. While we continue to investigate this bug and/or find ...

Last update on .

openvpn.png 09:57: We experienced some authentication issues with our VPN server. The process has been restarted and we are investigatng the root cause.

17:03: It looks like there is a libgcrypt bug that crops up when you use openVPN and authenticate against LDAP over TLS, which is exactly our setup. This bug results in ...

Last update on .

padlock.jpg Google posted to their online security blog earlier today about an interesting discovery one of their engineers made while trying to debug an SSH problem. After some investigation, they determined it was actually due to a glibc blug that could be exploited. The patches rolled out in coordination with this blog post. Unfortunately, it does ...

Last update on .

ibm-gpfs.jpg The department experienced a GPFS blip shortly after 10am this morning. The outage become pregressively worse and lasted about 10 minutes in total.

For those interested in more details, one of the quorum and manager nodes for the cluster experienced a system hard drive failure a few days ago. After replacing the hard drive and ...

Last update on .

Eiki LC-WUL100A.jpg The projector in CIT 316 is experiencing some hardware issues and should be considered unreliable. We have ordered a replacement, but not yet have an ETA for arrival. If you have this room booked in the next week or so and are in need of projection, we encourage you to reach out to CS reception ...

Last update on .

network-2400px.png (10:35) Around 9:15am this morning, a network card in a CIS switch failed that connects our primary firewall's external network interface. This necessitated someone going onsite to manually force our firewalls to failover. Service was restored around 10:30am. We are in the process of checking various services to make sure that ...