Website issue 28th July 2013

[TransAmDan]

Well last night the website was down for a little, I first noticed problems when i was uploading the Worthing photos. It seems to be missing images getting in a muddle, then having a server time out.
There was a few people on the website, the server is more than capable of handling that small amount. so I done a server restart, this was back online in 20mins, but over night, it seems to only serve blank pages, which isn't much use to anyone. Don't know what that problem started, could have been in the early hours of this morning.
So i restarted the 'http' services on the server and its now back,.

I've been racking my brains to what's changed, we had many more visitors over Wheels day time. Although I have installed APC caching services since then, which has dramatically speeded things up. I also done a load test, simulating 100 users all viewing 4 or 5 pages per minute. It handles that very well.

I regularly check the servers speed, and noticed google were saying my pages could be faster and to installed Google Pagespeed which sits on the linux server. I've not enabled this yet, however after last nights reboot it would have enabled since then. I'm gonna certainly disable that until i have time to play with it. I will also run some lead speed tests to see how well it copes. Pagespeed may have been the culprit although it could be one or more of many other things.

 

[sarah]

Just can't get the staff these days!

 

[TransAmDan]

Just can't get the staff these days!

I can train you up.

 

[sarah]

I've seen the stuff you do, I'm okay thanks haha!

 

[Rob63]

I did find it was struggling to load pages yesterday lunchtime.
Hope you all had a good day yesterday at the show.

 

[TransAmDan]

I did find it was struggling to load pages yesterday lunchtime.
Hope you all had a good day yesterday at the show.

That's good to know in a way. I was hoping it was all running smoothly while I was at Worthing. Looks like it wasn't just the evening.
The show was great, a good turn out.

The good thing is I can make the website fall over by asking for 100 pages per second. Which is a lot, but would be nice for it not to go wrong.

 

[TransAmDan]

Still getting temperamental issues, usually the problem occurs when its running out of RAM, however with 60% of ram used and access some part of the gallery I can get a server error, which knocks out the rest of the website for about 20 seconds while the web services restart
Its gonna take a bit of narrowing down.

 

[TransAmDan]

Well I think its all sussed now. There were some Linux addons like mod_fastcgi and mod_cgid and a few others which when I disabled the site still worked. Also disabled mod_suexec, after finding what I could get rid of in the linux build I found a lot less processes were generated while viewing a web page.
I found when uploading images it would no fail though, but putting mod_suexec back in, it all worked once again, and the extra processes didn't return.
Also while I was there I made sure the APC chaching was disabled and read into the google mod_pagespeed this seems quite clever in the way that it sort out the css, and combines images. Works very well on google chrome now, hoping it seems to give good performance in other browsers too. It also caches pages. Even when running a a couple of days its only created a 54mb cache, I've set the limit to 200mb.
So I'll be keeping an eye on things. I've done many performance load tests, it would fall over with 20 simulated users and pages would slow down to 15 seconds to download before then, not I've loaded it with 50 users, and pages still come in in under 3 seconds. So things have certainly improved.

 

[mummyjan]

Jeez Im impressed Dan, its all gobblygook to me, but well done x

 

[TransAmDan]

Thanks Jan.
A lot goes on behind the scenes of the website, especially to make it run fast, the original programming for the website isn't very well optimised, but its too massive to rewrite it all. So by moving it onto a faster computer, and putting in various caches and having various items of the website served be servers around the world, help make it whizzy, but also more complex.
Great fun though. Keeps me out of trouble I suppose. lol

 

[TransAmDan]

Well beeing tinkering with the website some more, and got another chap to take a look. It looks like the amount of RAM the server has ins't quite enough. It had 1.5GB of ram which is usially enough for Linux, but occationally it used it all, locks up and restarts various services.

So I have been on the hun for an upgrade option. VPS mean that the hardware is shared, so although it loks like a server on its own, I suppose their could be bottle necks, anyway, there was a 2GB option, but increased the hard drive space too which we didn't really need. Well for a little bit more code there was the Dedicated Server route. A dedicated server is what it say it is, a server no one else can use the resources of. It has a Dual core processor, 4GB of RAM 190Gb of web space, plumbed into the fast internet backbone. Its gonna be far quicker than the current one.

Anyway, the new server is in place, just need to copy the website and database across (been here before) then I will switch between the old website and the new. Hopefully the copy will happen over night.

 

[TransAmDan]

Well looks like the website move was pretty seamless. Had a small issue with uploading images, but that's all fixed now. It was down to directory protection.
The website seems far more stable at the moment. I stick 50 users on here, downloading from one page many times per second. The website would have collapsed before at around 20 users as the memory usage would creep up and then fall over. Now he memory stayed static, website remained the same speed, so basically the web server didn't even batter an eyelid at 50 users downloading many pages fast.