Technical Staff Blog

Last update on .

We had an emergency failover of the CS firewall at 4:45pm because two security updates screwed up the active firewall software configurations.  The fail over was a manually process because some of active firewall software were removed due to a package conflict with the security updates.  There was about a ten minutes outage for the manual failover to complete.  All services should be back to normal. 

18:10 - It was a case of truly unfortunate timing. We tried to update our server at precisely the same time that the security mirror was being updated. The amd64 version of a security update made it to the mirror before the i386 version, which resulted in only one version being updated and a whole slew of dependency problems. The mirror is now in a better state and future updates appear to be going off without a hitch.

19:19 - We've now checked out all machines that might have been affected by this update skew. All have been pre-emptively updated, so we should not be at risk of complications during our nightly maintenance run.