We had an emergency failover of the CS firewall at 4:45pm because two security updates screwed up the active firewall software configurations. The fail over was a manually process because some of active firewall software were removed due to a package conflict with the security updates. There was about a ten minutes outage for the manual failover to complete. All services should be back to normal.
18:10 - It was a case of truly unfortunate timing. We tried to update our server at precisely the same time that the security mirror was being updated. The amd64 version of a security update made it to the mirror before the i386 version, which resulted in only one version being updated and a whole slew of dependency problems. The mirror is now in a better state and future updates appear to be going off without a hitch.
19:19 - We've now checked out all machines that might have been affected by this update skew. All have been pre-emptively updated, so we should not be at risk of complications during our nightly maintenance run.