This graph could happen to you if you ever forget to configure munin email alerting:
It only took approximately 1 hour to diagnoses and resolve this issue however most of my web applications hosted on this server were down for about 11 hours. I was lucky that this outage fell on a weekend otherwise I would not have known about the problem till around 6:30pm!
Diagnosis:
Two of my pylons apps had session files that slowly ran away on me. The session files don't consume much capacity however the shear quantity of them caused my inode usage to hit 100%.
Had I properly configured Munin's email alerting this issue would have been identified well before it was a problem.
Want to know what alerted me to the problem? G Webmaster's tools claimed it could not read my robots.txt on a few of my sites... After investigating I learned the site was down. Checking the Apache error logs pointed me to disk space issues. df -ha reported everything was fine, however df -hi reported 100% inode usage! At this point I started looking to cache and log locations to find lots of files, which lead me to my pylons web applications data/sessions directories.
Resolution:
Delete the session cache tree directories and allow the applications to rebuild them.
Todo: move /www off the root disk partition. This issue could have been much worse if I was unable to boot or login to remedy. Moving /www off root should prevent the web server from effecting the systems ability to boot.