On May 29th, 2007, at around 4:00 p.m. (Malaysian time), the web server that hosts all the GIDNetwork sites crashed. Every time the support team at the data centre restarted the dedicated server, it crashed, again and again.

It must have been a computer hardware problem, although I am still not certain what was actually the item that was failing. Looking at the messages in the log file revealed many lines like this:

May 29 05:28:57 yumie kernel: hda: dma_intr: status=0x51 { DriveReady SeekComplete Error }
May 29 05:28:57 yumie kernel: hda: dma_intr: error=0x84 { DriveStatusError BadCRC }
May 29 05:28:57 yumie kernel: ide: failed opcode was: unknown
May 29 05:28:57 yumie kernel: hda: dma_intr: status=0x51 { DriveReady SeekComplete Error }
May 29 05:28:57 yumie kernel: hda: dma_intr: error=0x84 { DriveStatusError BadCRC }
May 29 05:28:57 yumie kernel: ide: failed opcode was: unknown
May 29 05:28:57 yumie kernel: hda: dma_intr: status=0x51 { DriveReady SeekComplete Error }
May 29 05:28:57 yumie kernel: hda: dma_intr: error=0x84 { DriveStatusError BadCRC }
May 29 05:28:57 yumie kernel: ide: failed opcode was: unknown

A quick search online hinted that this could be an issue related to a Hard Disk Drive (HDD) failing, or something as simple as a faulty IDE cable, neither of which we could figure out for sure. So the kind people at WholesaleInternet decided to replace everything, from new HDDs, to the processor and main-board of this server.

That took a while, and because the most recent backup I had on my own PC here in KL was over 1 week old, I didn’t want to restore the server with it. I knew there was a current backup on the old HDD and appealed to the support techs to copy the files onto the new HDD. Well, to make a long story short, they could not mount the (old) hard disc drive no matter what they tried. Much of Wednesday (May 30) was lost waiting for them to get files moved.

Later that night, I met with Darrin Smith online and he was kind enough to offer to help. Within minutes of logging into the server he got the HDD/partition mounted and I could finally access the files, move the backups to a safe place on the new HDD, and proceed with the restore, and bringing the sites back online.

I would like to thank all the good people at WholesaleInternet, especially Brian Vlasenko, who stayed up through the night trying to solve the problem, for eventually replacing nearly all the hardware, and reloading the Operating System and required software for the server on a new HDD.

A special thank you to Darrin for saving us all from losing over a week’s worth of data.

May nothing like this happens ever again…

Sometime between yesterday and the day before, something changed and it caused all the sites being hosted here, including GIDForums, to load very slowly. I am very sorry if this “error” caused you some inconvenience. I’ll try to explain what happened.

The first thing I noticed was that accessing GIDForums.com was unusually slow for me. Though distressed, I didn’t panic because I assumed it was just my lousy ISP again, as it is usually the case.

Then I noticed that the popular gzip testing tool was failing, the log file reported that the script wasn’t able to access any of the web sites the users were checking the whole day! Now I panicked…

It took me nearly half a day, but at least I managed to get one issue resolved: the web host had changed nameservers over the weekend, and the nameserver IP addresses I was still using for this server was of course now no longer valid. I made the changes as soon as I was informed, and at least the gzip testing tool was working again, but now it was working unusually slow, just like everything else.

Another few hours and many emails later, Aaron (my web host), suggested I restart the server. The server had been up for 308 days until yesterday, so I agreed it was a good idea. We did that, but nothing, everything was still so slow.

Then in one email he said, “I’ve run some tests and while it doesn’t appear to be the switch I’d still like to move your server.” I wasn’t going to say no, I was just glad we were doing something at this point!

Well, despite the move, there was not much improvement and we had to leave it at that. Now some 24 hours later, all the sites seem to be back to normal and no longer slow for me, I hope it is not for you either.

By the way, it was a Sunday evening where Aaron was at the time. Even so, he went all the way to the datacenter, after his supper, to help me with the reboot and the ’switch’ move, and to see if he could resolve this issue for me — what a guy!

Today, I think I know what the real problem was, but until I hear from Aaron again, I won’t be able to comment on it yet.

I registered GIDBlog.com yesterday - and as soon as I finished setting it up in my web server, I realised that I couldn’t get to the web site at all.

I was certain that I had done everything correctly from within my web hosting control panel after many tries, but still, nothing.

> more <