All posts by terrykay79

An update! And interesting algorithms…

Hello! Has it really been 8 months since I last posted? My intent to keep the blog updated certainly didn’t fan out over the summer! Though now the cooler weather is here, I have more time on my hands so I intend on changing my habbits!

Firstly, I apologise for the new look! I’m still getting the hand of WordPress themes. I’m a fan of the minimalist look, though this “Twenty fourteen” certainly looks a little too spartan! I’ve never been great at the “Making it pretty” part of web design, so stick with me an hopefully things will improve!

The thing I am better at is back end coding. I particularly enjoy problem solving, so when the team behind the cycling club came to me with a suggestion that one of the site functions could do with some improvements in speed,  I jumped at the chance!

The function in question relates to validating GPS trails – the majority of the rides organised are attended my masses of riders who start at a given point, visit certain “Checkpoints” to get a receipt, signature, stamp or some other proof of visit, and then (though not always) return to the start. These are the types of ride I attend, though another way of gaining points is to ride a “DIY Permanent” whereby you track your progress using a GPS device and then submit the trail, which is checked against known records of the trail.

Using libraries of functions, this part of the task is fairly straightforward and is carried out by visiting each point of the expected ride, finding the closes point from the riders trail and comparing that to within a certain level tolerance. The problem comes when there are several of these GPS trails to be checked and the riders have recorded a huge number of points along the way! One example I used was a 200km trail that had a suggested GPS trail of just over 1000 points.

The user had submitted 67,000 points along their journey! Although running locally, the software managed the impressive task within 40 seconds, I’m told that wasn’t the biggest that has been seen!

So how could the times be improved? The task is currently pretty much sequential, so only one processing core is being used so one option would be to break up the task into smaller chunks to allow for some parallel processing of the trail. Though given the cost of servers with multiple accessible cores over our current offering, I didn’t give this much consideration.

So how to simplify the task? Researching “Line simplification” may lead you to Ramer, Douglas & Peucker’s algorithm. Specifically wikipedia’s entry: https://en.wikipedia.org/wiki/Ramer-Douglas-Peucker_algorithm.

So how does it work? (For people not wanting to look just yet…) You take the start and finish points, draw a line between, and then find the point that is furthest from this line… If this point is outside of the tolerance required, mark this point as a mid point and create two new lines meeting here. Repeat this process for the new lines created, et voila! You eventually get lines representing the original route, with slightly decreased resolution but potentially a huge compression rate!

And the results of carrying out the filtering? The trail was reduced from 67,000 points to an impressed 1,200! Running the same path validation algorithm over this new track resulted in an execution speed of just 10 seconds! a 400% improvement!

Unfortunately it wasn’t all good news. As you can imagine, comparing all these points is still a computing intensive activity and took a full 20 seconds to carry out. Still, 30 seconds total compared to 37 was a 24% improvement.

So, the simplest option that was considered, of simply “missing out points” from the control trail currently looks the best option in terms of implementation ease and improvement (Using 1 in 4 points on the control trail and comparing against the full path of 67,00 user points took 17 seconds, incidentally) though we are trying one last ditch attempt to get our riders to tone down the data sampling!

 

And just in case anyone is interested in the Java implementation of the algorithm (you’ll need your own implementation of lineToPoint()):

 

private List <Coordinate> trim(List <Coordinate> coords, int epsilon) {
 double dmax = 0;
 int index = 0;
 int end = coords.size() - 1;

// Find furthest point from line start to end
 List<Coordinate> theLine = new ArrayList();
 theLine.add(coords.get(0));
 theLine.add(coords.get(end));
for (int i = 1; i < end - 1; i++) {
 double dist = lineToPoint(theLine, coords.get(i));
 if (dist > dmax) {
 dmax = dist;
 index = i;
 }
 }

 List <Coordinate> resultList = new ArrayList();
 // If max distance is greater than epsilon, recursively simplify
 if ( dmax > epsilon ) {
 List <Coordinate> recResults1 = trim(coords.subList(0, index), epsilon);
 List <Coordinate> recResults2 = trim(coords.subList(index, end), epsilon);
 
 // Build the result list
 resultList.addAll(recResults1);
 resultList.remove(resultList.size()-1);
 resultList.addAll(recResults2);
 } else {
 resultList.add(coords.get(0));
 resultList.add(coords.get(coords.size()-1));
 }
 return resultList;
 }

Look before you leap, think before you adjust the Apache settings!

I’m sure not everyone will have the same problem – several different URLs pointing at one server, all served by one Apache server? Maybe you’ve bought a few similar domains? Maybe tezk.co.uk, tezk.uk and tezk.home? Though through slightly sloppy web page creation, some of the links will direct you to one of the other sites… Not a problem! Until you realise the cookies you’ve created on one domain wont necessarily work on another domain, resulting in logged in users being forced to log in several times! A good argument for never specifying the host in links, only the relative path.

Anyways, how to force people to use just one domain? Here’s where Apache’s redirect feature comes into play! Where as previously you might have had the following in your sites configuration file:

<VirtualHost *:80>
           ServerName www.tezk.co.uk
           ServerAlias  tezk.home
           <directory /var/www/html/tezk>
           </directory>
</VirtualHost>

So the server responds to a host request on any of those addresses, simply use the redirect feature!

<VirtualHost *:80>
           ServerName tezk.home
           Redirect "/" "http://www.tezk.co.uk/"
</VirtualHost>

<VirtualHost *:80>
           ServerName www.tezk.co.uk
           <directory /var/www/html/tezk>
           </directory>
</VirtualHost>

Simple! Now, any request to tezk.home will result in a 3xx code with a redirect! The browser simply requests the tezk.co.uk web page and the user is none the wiser, other than not having cookie related login issues!

Though there is one problem this setup does create… What happens when we request the website using the ip address? Or use a script or curl to pull data from the site? Due to the ordering here, if the site name isn’t specified, we match the first listed, and so we get redirected! Unless the script is configured to follow redirects (And how many are?) the script will fail and you’ll be left scratching your head!

So how to resolve it? Listing the redirects after the main page definition will set the “default” page to our main site, which does fix the script issue, but does mean anyone requesting another site from us (Using the “Host” http header) will be served up our main site. But then again maybe that isn’t such a problem? The other way would be to ensure the “host” http header is set correctly! This is what Apache looks at when checking the virtual hosts!

It seems a related server and an App in the Google play store had been broken since the migration due to them not specifying the “host” properly…

AWS, a new comer’s views

As an IT professional it can be a daunting task trying to keep up with new technologies. A lot of job specifications I’ve seen have asked for Amazon or other cloud experience, so I decided it was time I had a look!

Previously I’ve administered several virtual servers hosted by hosting companies, all of which can be classed as cloud computing, given that the machine doesn’t actually exist and all of the processing is done “in the cloud”, so what are my initial thoughts having delved into AWS for the first time?

Mainly how easy it is to get started! The interface certainly looks daunting, though I managed to get a virtual server up and running on the EC2 (Elastic cloud computing) service in less than 10 minutes! And what’s better, they provide a modest machine known as a t2.micro, absolutely free of charge for the first year!

Delving a little deeper, following Ryan Kroonenberg’s “AWS Certified Developer 2016” course on Udacity, (https://www.udemy.com/aws-certified-developer-associate/) I was surprised by how easy it all was! I even manged to get a load balancer set up, linked to an EC2 server, and managed to Link to the Tomcat server seamlessly from Netbeans! And it was still free! (So long as you pick the right machine…) Admittedly the choice of course was biased towards getting certified rather than actually learning AWS, but it provides an excellent starting point!

Having developed an idea for a future Android app, the environment looks like the perfect place to host the back end processing given it’s scalable nature and high availability. Plus it will earn me some brownie points for the CV! Watch this space for details of the App in the future!

What a week!

What a week it has been! We returned to Kent from visiting family in Derbyshire, unaware of the events in store…

It all started Monday morning with an ominous email reading “The server has disappeared!”. Never something you want to see, never mind on a Monday! The usual ping test confirmed the host was down, though the hosting companies routers confirmed that the host was unreachable, so at least the DNS was not to blame.

Checking the providers news feed revealed they’d had a server outage the previous day, confirmation something had gone awry on their end. After finding someone with the password for the dashboard, they raised a support ticket and got the server running again! Unfortunately that was only the start of the problems!

As the server had unexpectedly gone off line, the file system had a few errors and so had been mounted read only until we could run FSCK over it to correct any problems. Being a hosted server, this had to be requested from the ever helpful support team. That done, we managed approximately half an hour before the server switched the file system back to read only mode! After several attempts to stabilise things, the decision was made to request a new server image and perform an emergency migration… Not something to be taken lightly, given the server was still running Ubuntu 10.04!

With the new server freshly running 14.04 and updated, the long slow process of installing the LAMP stack commenced, followed by the reinstatement of the databases and web site pages. By Tuesday evening, everything appeared to be running. Or so it seemed! It turns out the website was relying on features disabled in the currently installed version of PHP! After trying a few code kludges to work around the matter, time was pressing on and so a downgrade looked the best option. But how to carry it out? A good old compile from source! Which always takes forever in a server environment, given they aren’t geared for development and so package after package had to be added before it was finally compiled and installed and back up and running!

So, by Wednesday evening, the majority of the functionality had been reinstated…  A problem reading created Excel spreadsheets was traced to bugs in a no longer supported PHP module (http://pear.php.net/bugs/bug.php?id=19284&edit=3) and usability issues relating to cookie usage across domains were rectified (rather than respond to several addresses, the site now forwards requests to one domain) and so we’re up and ready for the weekend!

So what did I learn from the week?

  • Always keep up to date backups of the /etc/ directory! Luckily I had a copy I’d made last year available, but it would have been much easier if it had been kept up to date!
  • PHP is evil…
  • And PHP code from last decade isn’t going to be happy on a fresh server.
  • Ask for passwords for the control panel of the server! I hadn’t asked as I didn’t forsee a need for it…

So here’s to the weekend, and the 10 mile run on Sunday! (http://canterbury10.co.uk/)

With thanks to:

The support staff at Hosting UK (https://hostinguk.net/)  who did all they could to ensure we were back up and running as quickly as possible.

My colleagues at AUKWEB (http://www.aukweb.net/) all volunteers who set up the old server, coded the site and assisted where they could in the migration.

Here’s to another decade of uptime!

Trouble with WordPress automatic updates?

Well, I’ve just spent a good half an hour trying to jog my memory on how exactly I got WordPress to update itself the last time I tried. Googling for a solution didn’t really help either, as the general consensus was “Do it manually!”.

Not one to give up on a problem I tried to remember what I had done… I do recall having set up something via FTP… Did I need a server running for the update? No… All I would get was an error saying the download had timed out. This told me my file permissions, that do seem to cause a lot of issues, were good. But why was the download failing?

The file it was trying to down load was http://downloads.wordpress.org/release/wordpress-4.4.1-new-bundled.zip. I checked the firewall, outbound HTTP traffic was fine, attested to the fact I’d successfully updated the rest of the system only minutes earlier. What was the problem? One sure fire way to find out – WGET, a command line function that will download a file from a web server… And what did it tell me? http:// was no longer supported on the server, trying https:// Ah, at last! I’d opened port 80, but SSL uses port 443! I opened the outgoing port et voila! Update successful!

So just remember, if you do get any error messages, try recreating the error outside of wordpress, it can help immensely!

 

 

100 days until the big day!

So I had some spare time last night and thought I’d better get the donation page updated and wouldn’t you know there’s a whacking big countdown timer on there? Only 100 days to go until Marathon day!

So how am I doing? Ideally I’d be out every day, clocking up the miles and bomb proofing the legs! Though as it stands, my legs are still sore from the 10k I did Sunday… 4 days ago! Though I did manage the full 10k without tiring after 5k, which had been happening. I just need to be able to do it 4 times over! So keep an eye on this space, and follow me on strava (https://www.strava.com/athletes/terry_kay) to follow my slow painful progress…

Before I go, it might be a coincidence but my supporter pack came through today!

IMAG0089

And what is that exactly? My superhero pack! Yes, a running vest, two sweat bands and a cape… We all know how hard it is to raise sponsorship via social media, so I’ve promised on my donation page (http://uk.virginmoneygiving.com/TerryKay) that if I can raise £100, I’ll run the marathon wearing the cape… Wish me luck!

Another two weeks?

So, another two weeks have passed… And what have I achieved? I did attend my graduation ceremony!

12002875_10206991169994256_6474005596193399581_n

I’ve also registered on a Cisco Network Security course with the OU (here) but more pressingly, we’re moving! After spending the summer in a caravan and awning, we’re moving back in to bricks and mortar next week! So what have I learned from a summer under canvas? The costs are certainly appealing… 6 months living back in Derbyshire would have cost us around £5000, accounting for mortgage payments, council tax, water rates, energy costs, etc…. That cost in greater London? We’re paying £6000 for rent alone, so I’ll let you know the total… Our running costs for the summer? Not counting the expense of the van, around £1500… And for that we’ve got on site sports facilities, sauna, swimming pool and surrounded by woodland!

Unfortunately we aren’t allowed to stay year round. Though with the dropping temperatures, I’m not sure I’d want to! I’m certainly looking forward to running water, plumbed drainage and central heating again!

Fingers crossed and see you on the other side!

Two weeks to update?

So what are my thoughts on WordPress now I’ve had it installed for two weeks? Make sure you set the ownership of the installation correctly!

Given the need to ensure any software visible to the internet at large is as secure as possible, updates should be applied as soon as they are available to ensure any bugs and vulnerabilities are patched.

So, having installed my copy of WordPress, one of the first things I looked at was how to make sure I could easily update the software. Thankfully, WordPress provide an easy way to update the software through the integrated dashboard. Though clicking revealed I required an FTP server running on the web server to perform the updates? Not keen on setting up and FTP server on my production server, I put off the update until I had an afternoon free to install a server and get the updates carried out.

Following the Ubuntu Documentation’s suggestion of installing vsftpd (https://help.ubuntu.com/lts/serverguide/ftp-server.html) I went on to add a user whose home account was set to the directory I’d installed WordPress, to enable the files to be directed to the right place. Changing the owner of the files, I carried out the updates and then wondered why exactly they felt the need to force users to install an FTP server to carry out the updates?

Needless to day, A quick search lead me to a topic that had already discussed the issue… http://stackoverflow.com/questions/640409/can-i-install-update-wordpress-plugins-without-providing-ftp-access revealed that WordPress will only attempt to upgrade over FTP when the web server tries and fails to write to the /wp-content directory! I had already disabled the FTP server and blocked the ports again, though lesson learned: Make sure file permissions and ownership are set properly from the start!

Hello world!

maxresdefault

Here I am! An absolute age after deciding I was going to set up a blog, I’ve finally got around to putting it into practice! Why has it taken so long? Well, after last year and completing my last year with the Open University by taking 4 modules, upping sticks selling our property in Derbyshire to live in a temporary home in the outskirts of London, then taking the Linux Foundations Certified Sysadmin and Certified Engineer exams after carrying out my own studies (http://training.linuxfoundation.org/certification) and then working on a few projects to flex my coding skills to get a some websites up and running to showcase my skills, I’ve struggled to get around to it!

So who am I? I still remember my dad buying us a Sinclair ZX Spectrum 48k from our local Co-op superstore back in the 80’s. After my initial awe and wonder, I got around to talking my dad into buying the Input magazine series that promised to introduce it’s readers to the world of programming! After wetting my toes with the early 8bit machines, then moving on to the Commodore Amiga (I’ve still got a full set of the original Reference manuals! Though I never managed much programming..) I moved on to studying for my NVQ level 3 in programming with a local company which involved coding on 386 machines running windows 3.11 for work groups! After failing to get in to the industry I eventually took an administration job with a local company that saw me settle there for the next 13 years!

During my time there, I came across the Open University and their degree in computer science. I started with my first module in 2005 and have enjoyed carrying out study ever since! Most recently, I finished my degree by taking redundancy from the firm I was working with to carry out 4 modules in one year, the equivalent of a year of full time study, which resulted in me achieving an upper second degree in computing and opened my eyes to Java EE and MySQL as I used these in my end project.

Having moved to London, I took the opportunity to further my studies and now have the Linux Foundation’s Sysadmin and Engineer certifications under my belt, and I’m now using my experience to work on a website implementing the technologies I’ve learned.

So where next? A major fan of Java, Linux, the open source world and Android, I hope to start a few Android projects. Being an ardent runner I’ve a few ideas in mind!

Wish me luck!