Sunday, March 16, 2008

Downtime

Sorry, we had to take the server offline for maintenance. Here is an overview of what happened (copypasted from irc):


[18:31] the ddos used up all available connections
[18:31] so I used a config tool to up the connection limit
[18:31] but the config tool added a newline in a comment in the config file
[18:31] and part of the comment was parsed as a directive
[18:32] this filled the error log with errors
[18:32] and /var was completely full
[18:32] because /var was full
[18:32] mysql crashed
[18:32] and the system wouldn't start except in single user mode
[18:32] so I had to log in with vmware-server-console to get single user mode and delete the logs
[18:32] In the process of restarting mysql, it broke the database
[18:33] the auto-incrementing column in one of the tables broke
[18:33] so I had to dump and reload the whole database
[18:33] :D
[18:33] A nice days work

No comments: