What a week it has been! We returned to Kent from visiting family in Derbyshire, unaware of the events in store…
It all started Monday morning with an ominous email reading “The server has disappeared!”. Never something you want to see, never mind on a Monday! The usual ping test confirmed the host was down, though the hosting companies routers confirmed that the host was unreachable, so at least the DNS was not to blame.
Checking the providers news feed revealed they’d had a server outage the previous day, confirmation something had gone awry on their end. After finding someone with the password for the dashboard, they raised a support ticket and got the server running again! Unfortunately that was only the start of the problems!
As the server had unexpectedly gone off line, the file system had a few errors and so had been mounted read only until we could run FSCK over it to correct any problems. Being a hosted server, this had to be requested from the ever helpful support team. That done, we managed approximately half an hour before the server switched the file system back to read only mode! After several attempts to stabilise things, the decision was made to request a new server image and perform an emergency migration… Not something to be taken lightly, given the server was still running Ubuntu 10.04!
With the new server freshly running 14.04 and updated, the long slow process of installing the LAMP stack commenced, followed by the reinstatement of the databases and web site pages. By Tuesday evening, everything appeared to be running. Or so it seemed! It turns out the website was relying on features disabled in the currently installed version of PHP! After trying a few code kludges to work around the matter, time was pressing on and so a downgrade looked the best option. But how to carry it out? A good old compile from source! Which always takes forever in a server environment, given they aren’t geared for development and so package after package had to be added before it was finally compiled and installed and back up and running!
So, by Wednesday evening, the majority of the functionality had been reinstated… A problem reading created Excel spreadsheets was traced to bugs in a no longer supported PHP module (http://pear.php.net/bugs/bug.php?id=19284&edit=3) and usability issues relating to cookie usage across domains were rectified (rather than respond to several addresses, the site now forwards requests to one domain) and so we’re up and ready for the weekend!
So what did I learn from the week?
- Always keep up to date backups of the /etc/ directory! Luckily I had a copy I’d made last year available, but it would have been much easier if it had been kept up to date!
- PHP is evil…
- And PHP code from last decade isn’t going to be happy on a fresh server.
- Ask for passwords for the control panel of the server! I hadn’t asked as I didn’t forsee a need for it…
So here’s to the weekend, and the 10 mile run on Sunday! (http://canterbury10.co.uk/)
With thanks to:
The support staff at Hosting UK (https://hostinguk.net/) who did all they could to ensure we were back up and running as quickly as possible.
My colleagues at AUKWEB (http://www.aukweb.net/) all volunteers who set up the old server, coded the site and assisted where they could in the migration.
Here’s to another decade of uptime!