Excessive website load

[TransAmDan]

Since the weekend about 19th Jan 2014, I've noticed excessive CPU load on the web sites server. Its the only website hosted on the server, so the problem should be all under my control. However tracking down the issue hasn't been easy. Spent a bit of time on it yesterday, thought I found the solution but the extra CPU load comes back from time to time. It may be fine for the 10 mins you are watching it, then check it again later and its back high for a few mins.
When i mean high, we are usually under 30% cpu load averaged across the two CPU's. It used to be as low as 8%. I don't mind it being under 70% but it does go over 300% when many pages are requested.

So when did this start to happen? well after the 7th Jan 2014 as I done that speed test in another thread. So it basically opened around 8 selected pages on the website and simulated 50 users doing that. It would load every page in under 1 second previously. However now it takes 20 seconds plus when 50 users are loaded. So something is seriously wrong. However the website didn't crash, just got very slow.

All since I installed this DBSEO plug in, http://www.solent-renegades.co.uk/recent-changes/10373-new-seo-plugin.html

However I disabled this plug in last night for about 5 mins (probably not long enough) and CPU load appeared to be better but probably my imagination as it did jump over 100% at times. The trouble is with disabling this plug in, is all the nice web page names it creates. If we turn it off all links to this website will no longer work.

Anyway sit tight, I'm looking into it.....

 

[TransAmDan]

Okay, made some progress....

Zapped it though another load test, load times between 0.9 and 1.2 seconds when simulating 50 users. That's pretty darn good, considering it was taking 20 seconds+ for the same test yesterday.
so whats different?

I've disabled many plug-ins, but kept the DBSEO running(keeping page names the same), brelow is a list of all the add-ons, only the ones in bold are currently enabled.



Advanced BBCode Permissions 3.0 Lets you set Usergroup Permissions for BBCode.

DragonByte Tech: Advanced Post Thanks / Like (Lite 3.1.7 Advanced Post Thanks / Like lets members quickly voice their opinion on a post.

DragonByte Tech: Advanced User Tagging (Lite) 3.0.8pl1 Advanced User Tagging lets your members quickly alert eachother to threads they might be interested in.

DragonByte Tech: SEO (Pro) 1.0.0 DragonByte SEO is an advanced Search Engine Optimisation tool to help increase your forum visibility on search engines.

DragonByte Tech: vB Optimise (Pro) 2.6.0b4 Optimises vBulletin by using cached content to reduce MySQL usage.

DragonByte Tech: vBActivity & Awards (Lite) 3.1.4pl1 vBActivity by DragonByte Technologies: Easily track and reward the activity of your members, including in-depth statistical analysis of forum activity in various areas.

DragonByte Tech: vBAnalytics (Lite) 1.2.0 vBAnalytics by DragonByte Technologies: Extended forum analytical stats

DragonByte Tech: vBMail (Pro) 1.2.0 DragonByte Tech: vBMail is a professional mailing list creation and management mod.

DragonByte Tech: vBNavTabs (Pro) (vB 4.2.0+) 1.3.1 Enhances the Navigation Manager adding more control over navigation items.

DragonByte Tech: vBQuiz (Pro) 2.0.1 vBQuiz by DragonByte Technologies: Allow your forum members to submit and answer vBulletin based quizzes!

DragonByte Tech: vBSecurity (Lite) 1.1.1pl1 vBSecurity by DragonByte Technologies: Take back control over Contact Us

IWT - Time Spent Online 1.2.6 This will add a counter in that shows how much time your users have spent online on the site.

Mark Thread As 'Sold' 2.0.1 This hack will allow members to mark their threads sold.

Navbar menu for Gallery 1.00 Adds a navbar Gallery drop-down menu

Navtab menu for Gallery 1.00 Adds a navtab Gallery drop-down menu

Picture and Album Gallery 1.08 Picture and Album Gallery for vB4

PostRelease 4.2.1 Official PostRelease plugin for vBulletin

VB Pro Garage 3.0.5 An advanced garage system for VBulletin

VB Pro Garage Featured Widget 1.0.1 vB CMS Widget to show the VB Pro Garage Featured Vehicle

VB Pro Garage Random Image Widget 1.0.3 vB CMS Widget to show VB Pro Garage Images

vBExperience 4.1.1 Calculate activity of your users

vbStopForumSpam 0.61 This plugin allows you to check new registrations against www.StopForumSpam.com

vBTubePRO 2.6.8_vB4 Video, Photo and MP3 Gallery by www.vBTube.com

vBulletin Blog 4.2.1 Personal web log, integrated with vBulletin.

vBulletin CMS 4.2.1 Content Management System

vBulletin-CMS Links 1.0.0 vBulletin-CMS Links let your add a link to a section instead of jumping right to the section.

VSa - Advanced Forum Statistics 7.0.7 VSa - Advanced Forum Statistics

VSa - Auto Birthday Greeter 2.0.6 VSa - Auto Birthday Greeter

VSa - ChatBox 3.1.8 VSa - ChatBox

 

[TransAmDan]

Still with these items disabled, I've been watching the process window.

Average over the last min is 0.00, average over 5 mins 0.03 and average over 15mins is 0.22
Numbers get multiplied by 100, so 0.22 is actually 22% CPU time.

The website seems to be working well, we have a loss of a few minor bits of functionality, like clicking on 'Like' on a post, the rest of the website it turned on and optimisation turned off.

So its just a case of watching it, to check the problem has gone, and then i will start turning bits back on to see what is really causing this problem.

Its not really a problem the end user will see, until we get many people at the same time then it will slow down. In its current state it would have to be hundreds of people all loading 3 pages per second for it to cause a problem.

At 14:40, the Average has come down to 9% on a 15min average
So at 14:45, i have enabled VBOptimise, and will leave it a good 30mins to see what the average is then. I've done many forum and article page refreshes, and server load is looking good so far (although I've had this before, looks good when your watching then all goes to pot).
15:10 enabled DragonByte Tech: vBNavTabs (Pro) (vB 4.2.0+), server load is around 20% on average, a few users online too.
15:38 enabled Advanced BBCode Permissions, DragonByte Tech: Advanced Post Thanks / Like and DragonByte Tech: Advanced User Tagging (Lite)

16:13 problem back, 4.82, 3.17, 2.34 processing time, disabled Advanced BBCode Permissions, DragonByte Tech: Advanced Post Thanks / Like and DragonByte Tech: Advanced User Tagging (Lite) and still high.

 

[saleen 192]

Don't know what you are talking about but good luck yes pages have been very slow

 

[TransAmDan]

Don't know what you are talking about but good luck yes pages have been very slow

Thanks for the feedback, the load times should be okay at the moment.
I may have been doing some load testing when they were really very slow before. I aim to improve. Gotta be prepared for the rush of visitors when the show season is well on its way.

 

[TransAmDan]

Still looking into this. I've disabled some addons, and re-enabled the ones that don't seem to effect the CPU level.

Reading this article, Understanding Linux CPU Load - when should you be worried? we shouldn't need to worry until we get above 1.4 CPU load on a 15min average as we have 2 CPU's and they recommend 70% duty on each, therefore 1.4

I've been running some rests of about 20 pages on the website, loading them many time (created a reload random script on the works server) loading about 88 pages per minute. The CPU load is 1.07 on a 5 min average, so it appears to be holding up fine.

We had 2079 visitors on 29th March last year, the bulk of them between 6am and 1pm. Thats about 182 visitors per hour. Or 3 pages per minute. So we should have plenty of resources. So if its not page loading causing the CPU to stumble, then what is it...... Its only the HTTPD process that seems to be most demanding on the process list. This process serves web pages.

Anyway, during looking at the Server Specs, I've decided up upgrade from the E4500 - Core 2 Duo 2.2Ghz to a nice E3-1230 Quad Core 3.2Ghz. So this gives us double the processor cores and an increased of 1/3rd in CPU speed. So effectively 2.9 times faster (of course some over heads for more cores) Its gonna be whizzy. :whoo:

 

[TransAmDan]

New web server is online. Its just basically an operating system and admin panel at the moment.
Need to whizz over the complete website, that I will schedule in for tonight to minimize time offline.

 

[saleen 192]

Yes system / pages very slow even tryed on my phone very slow, Good luck with upgrades tonite

 

[TransAmDan]

Well we are on the new server now, the transfer was pretty painless. The new server will be idle most of the time, as its going to deliver the pages with ease.

We are running with no speed optimisation at the moment. I need to install and configure various cache systems in the OS to do that. So at the moment as its wokring fine i will do the cache stuff later on.

 

[TransAmDan]

Memcached has been installed. This is a create program to pass data though, as it saves access time to disks if the data can be stored in RAM.

However just noticed a glitch, when the Memcached filled up, i couldn't load pages. I may have to tweak that one.

 

[TransAmDan]

I've turned off the Memcached, its a 3rd party plugin which seems a bit ropey, they say in their latest version if can cause crashes. humm, well I stayed away from the latest. Anyway, the website is pretty darn fast without the cache, I will leave it as it as its all working.

Processor usage currently is 0.01 on a 15min average, lol. Basically 0.25% (as 4 CPU's) I got it to respond to 70 pages per minute, he went to to 0.10, which is about 2.5%

So overall, very pleased with the switch. The new server is on a special price offer, so basically costing the same as the old server but with a far better spec, so it was well worth switching.

 

[an88fiero]

Nice going Dan. Didn't even notice the switch over. Well done!

 

[TransAmDan]

I was hoping everyone would be switched. Its being a bit off. At home I'm on the old website, at work I'm on the new one. This is down to the internet provider not updating the DNS pointer to the correct location. This happens within 24 hours.I started this about 7am. So hopefully tommorow morning we should all be on the new one.

Atleast if people are looking for info etc.. the will be on either the new or old so will see the info. However its just the posting, hopefully no one posts on the old one. Still not much longer to wait.

It feels quicker, but that could be my imagination. Actually there is a time counter at the bottom of the pages says how long the page took to generate. Its about 0.3seconds on the new one, about 1.2 seconds on the old one for the same page. So its certainly better.

I must look into that Arcade at some point.

 

[TransAmDan]

Well everything should all be totally switched over now.

I'm hoping everyone can still get online. It should all be fine. I've seen a few members log on to browse, is posting working for everyone?

 

[an88fiero]

Is for me