We’re coming up on 24 hours since the new server was put on-line and starting accepting traffic, and it all looks good. The performance of the new disks are excellent and we are very pleased with it. We will continue to keep you updated on the state of the DNS change propagation.
The CTS Team
On Sunday night, 3/2/2008 around 11:30pm, one of the drives in mirrored disk pair failed in such a way that it locked up the server. The problem was diagnosed quickly and our upstream provider, ServerBeach promptly replaced the failed drive and the server was back up around 4am Monday, 3/3/2008 with no loss of data.
In the days that followed, the CTS team started looking at migrating to more robust hardware, had narrowed it down to two possibilities and were doing preliminary planning.
This past Saturday, 3/8/2008 around 11:30am, the replacement drive used on 3/2 failed in the same fashion, locking up the server. At this time, the CTS team looked at our options and decided to accelerate and expedite the server migration. ServerBeach was asked to expedite the provisioning of a new server with more memory, more disk space and a faster CPU, and they were able to do so in about a third of the time that they guarantee for new servers. Once the new server was up, we started transferring files from the old server to the new server, but due to the speed of the drives and other issues on the old server, the speed was slower than expected and we were looking at a 10 to 12 hour time frame to complete. The file transfer was started and the CTS team finally went to sleep around midnight on Saturday.
On Sunday morning, we discovered that about around 1am Sunday morning, the NEW server locked up. We reboot the server and restart the file transfer, only to have the new server lock up again 4.5 hours later. We decided that this is a case “infant mortality”, call ServerBeach and have them expedite a chassis swap so we don’t loose the 20GB of files we’ve already transferred. The ServerBeach technicians have the chassis swapped in record time, we restart the file transfer, and all goes well. Around 10pm Sunday night (3/9/2008), the migration was completed and the new server was on-line and responding to requests. However, we were unable to move the IP address from the old server to the new server, which means it will take some time for the IP address change to propagate. Until this has happened, we will keep the old server up and have configured it to transparently proxy connections between the two servers, so you should not have any problems connecting to your web sites or getting your email.
It may take a few days for all the mail servers around the internet that has queued up email for delivery to notice that our mail server is back on-line and attempt to re-deliver the email, but it should show up. We apologize for the unexpected downtime the past two weekends and feel that the new server will address this issue.
