Recent Downtime

[Wise1]

Message from the data centre below in regard to the recent downtime.

At approximately 9:45PM your server experienced a kernel panic due to ext3 file system errors. A manual fsck was required on reboot and as it was running we discovered that the home file system was completely destroyed. Here is what we are in the process of doing:

1. Copying backups to an alternative location for additional redundancy on this important data.

2. Although no hardware failure has been indicated, we are going to go ahead and replace the disks in the RAID array just to be safe.

3. We will be reloading the OS and setting up the server once again (cPanel/WHM/etc.).

4. We will be restoring all accounts from system backups that were generated on January 26th starting at approximately 2:00AM.

This entire process will take a while to complete. I will continue to update this thread as we progress.

Unfortunately this sort of problem will inevitably arise from time to time (albeit fairly rarely). There is really no way to fully prevent this from occurring as the drives have shown no signs of hardware problems at all. Fortunately we do have the system backups available that were generated just before this incident occurred so the overall impact will be much less than it could have been.

I do apologize for the inconvenience that this has undoubtedly caused. We're working as quickly as we can to get the server back online and fully functional with all accounts restored.