08:54: We are experiencing some database issues with our ownCloud service that could affect syncing back to the servers. The database cluster is managed by CIS, so we are working with their DBAs to diagnose and fix the problems.
09:22: We have shut down one of the backend servers to aid in debugging this problem.
11:24: CIS restarted the data nodes and things seem better, at least we can log into the service again. We are still seeing errors in the logs that at least one table is still read-only. This should not affect file operations (i.e. uploads/downloads), but will affect changing any of the optional app settings.
12:50: Actually, it turned out to be more than just one table that was read-only. CIS ultimately restarted the SQL nodes as well, which fixed this issue. At this point, we believe the service is fully operational again. CIS is investigate whether they can determine the root cause.
15:42: The second backend node is online again, so we are back at full strength.