18
Mar 11

WordPress Simple Graph Plugin

One of the downsides of working for IBM (and frankly, there aren’t that many), is that most of the work I do is usually under a pretty strict NDA. I’m learning a lot about the innards of database and High Availability and Disaster Recovery, but I can’t actually go blogging about it ūüôĀ However, IBM support a lot of open-source work and so they actually understand it (many companies I’ve worked for don’t). So long as you let your manager know what you’re working on and it isn’t something that competes with IBM products (fair enough, if I’m doing DB2 internals all day, I shouldn’t go work on MySQL internals on weekends for a dozen different reasons), then you can hack away on anything you want in your spare time.

That in mind, I’m finally getting back to some coding in my spare time (what little of it I have), and I’m starting slowly with some maintenance work on a WordPress Plugin called Simple Graph. You can see it in action over on my training blog, it’s what’s making the graph on the right of the page that tracks my weight:

Simple Graph sidebar widget

Simple Graph sidebar widget

It’s just a very simple widget for taking in hand-entered numerical data and plotting it dates.¬† The codebase has stagnated since 2007, so its author has given me permission to maintain it, and I have a list of things I want from it before I’ll call it done:

  • Clean commented code without any PHP warnings or notices
  • Fully WordPress 3.1 compatible
  • All options working as documented
  • Documentation updated to be more accurate and expansive
  • Better SQL query structure
  • More checking of input data
  • Multiple graphs working
  • Shorttag substitution in posts working

Once that’s done, I have a small list of features I want to add to it:

  • CSV import and export of data
  • Area graphs
  • More options to control graphics – line thickness, grids, background options, etc.
  • Use more GD features to give better looking end results
  • Use more Google Chart API features for better looking end results

Once that lot is done, we should be up to version 1.10 or thereabouts. I’ll post up a roadmap on the Simple Graph page I’m putting up later on. If you’re using the plugin and have any features you’d like to see, let me know using the comments below.

So far, I’m about a third of the way into cleaning up the code and commenting as I go. Feels good to finally be working on something for fun again..


10
Mar 10

Performance tuning a server in less than three minutes while being slashdotted

So you wrote a blog post about something that seemed fairly innocuous, but for whatever reason, it caught the attention of one of the major sites and now your server load is at 110 and climbing, the ssh command line session is taking thirty seconds to respond to anything at all, and given that your post is on the front page of slashdot at primetime, this doesn’t look like it’s a temporary blip. What do you do?

Burning computer

Okay, first things first. You don’t have time to do a proper fine-tuning session. You need a quick & dirty tuneup here. Proper fine tuning you leave till after the traffic spike, and you can then come back at it with a plan and decent tools like siege and so on – but this is the “fix it in three minutes” version. So if you see stuff here that looks crude to you, despair not, everything looks crude when you’re trying to do it in that kind of timeframe.

First, you have to be able to actually use the server command line. The load’s at 110 – that’s not coming down anytime soon and until it does you can’t do the rest of the work you need to since it takes so much time for even a shell to respond. The load is being caused by Apache and MySQL taking on more work than the server can handle, causing the server to swap excessively; and you’ve got to dump that work or shut off the incoming new work to recover. You can try sudo killall -9 apache2 if you can get an open ssh terminal in to do the job (and it’s the first thing you should try), but the odds are that that server has to be reset. Whether that means a phone call to the data centre, or a walk down the hall to the server room, or just clicking a button in a web interface, that’s the first thing to do. Don’t hold off, because unless everyone stops reading your page right now, that load’s not coming down.

Once the box has rebooted (and I mean immediately – sit there watching ping until it comes back), ssh in and shut down apache. MySQL is okay for now, but the work is coming from Apache, so that has to get shut down until everything’s ready to go again. You’re going to lose a few minutes of service, yes, but that’s recoverable from for a blog (and if this is for something more serious than a blog, you’re going to be in trouble for not properly spec’ing the hardware to begin with anyway, so limit the damage now and take your licks later).

At this point — whether you’ve just logged in or whether you managed to run killall successfully —¬† if it’s a debian server you should run sudo /etc/init.d/apache2 stop (did I mention I love debian for having scripts like that? No man-paging apachectl, just an easy to remember standard interface for every service on the box. Wonderful). It’ll tidy up from killall, or it’ll shut down apache cleanly, depending on how you got here.

I’m going to use my server as the example here, by the way, since it’s what got burned last night and it’s what prompted me to write this refresher — it’s been two or three years since I last had to maintain a server that was near its capacity, the experience was a bit of a flashback ūüôā So, some background on my server – I moved my blog from wordpress.com to here, on a Hetzner server. It’s their entry-level dedicated server offering 2Gb of RAM, a 64-bit Athlon, 160Gb of hard drive in a hardware RAID-1 array and a 1Gbit NIC) — all running Debian Lenny (and no, I’ve no relationship with Hetzner, they were just the cheapest of the places various friends recommended). WordPress is up to date on my server (2.9.2 at the time of writing), as is the Lenny install — if you don’t have the latest security fixes and such in place, or your WordPress is outdated, then that’s probably adding to your problem, but for a quick fix like this, that’s too big a job. Get through the traffic spike and deal with it later.

And yes, that server spec is overkill for my needs really – but I had a bunch of side projects like RangeClerk (don’t bother, not much is up yet) and the blog for Herself Indoors and her book and some other things I wanted to run as well that would be using wierd php and python modules and libraries and the like; and I just hate cpanel and not being able to install anything I wanted. Plus, it was cheap ūüėÄ

Right, back to it.

The first thing we need to do is to sort out MySQL’s configuration. Open up the my.cnf file, whereever you’ve put it (it’ll be /etc/mysql/my.cnf for a stock Debian install). We need to tweak just a few settings. First off, key_buffer. This is probably the most critical variable here because by default all the tables will be MyISAM tables (if they’re not, then this isn’t so critical). It’s set to about 16Mb by default; we’re going to turn that up quite a bit. On a dedicated database box, this would be set very high – anything up to 50% of the total available memory. In this case, with a full stack on the one box, we set it a bit lower since Apache’s going to want a lot of RAM too – 256Mb will do for a starting value.

Next we’re going to disable the InnoDB engine completely to cut down on MySQL’s footprint. Again, WordPress by default isn’t using it. Just ensure skip-innodb is either uncommented or inserted into my.cnf.

Lastly, we’re going to enable the query cache. The thing is, MySQL’s query cache is a fairly blunt instrument. If a query is precisely the same the second time it comes in, it’ll hit the cache – but any change at all, no matter how small, and it misses the cache. So it’s not as enormously useful as you’d first imagine. However, it does help, so we’ll increase its size modestly (48Mb of RAM is sufficient here). So our changes to my.cnf look like so:

[cc lang=”ini”]
key_buffer = 256M
query_cache_limit = 16M
query_cache_size = 48M
skip-innodb[/cc]

Once those changes are made, sudo /etc/init.d/mysql restart will get the MySQL server up and running with the new setup.¬† Once that’s done, let’s look to the next level in the stack – Apache. Under debian the config files are arranged differently than normal; the configuration changes we’ll make will be in /etc/apache2/apache2.conf but in other installations they would be in httpd.conf or elsewhere.

The default Apache install uses the prefork MPM setup – one thread per process. It’s older, slower, less efficient, but less buggy than the worker MPM which isn’t threadsafe. So find the prefork MPM config settings in apache2.conf. They should look like this in a default install:

[cc escaped=”true” lang=”apache”]
<IfModule mpm_prefork_module>
StartServers          5
MinSpareServers       5
MaxSpareServers      10
MaxClients          150
MaxRequestsPerChild   0
</IfModule>[/cc]

We’re going to cut down a lot on how much work Apache takes on at once here. Yes, some users will have to wait a few seconds to see your page – but right now, with the load at 110 and climbing, they could wait until their browser timed out and they’d never see anything. So we reduce slightly the number of servers that Apache will farm off to handle requests at any one time from 5 to 4; we’ll increase the number of spare servers it’ll keep around to hand off requests to (we want to reduce the overhead of starting and stopping those processes) from 10 to 12. We’ll set an upper limit on how many we can have though, and we’ll keep it to just under 100. This works on my system, which is an entry-level system; you might get away with more, but for now use these settings and it’ll get you up and running and you can increase a bit and check again as you go (and this guide really isn’t aimed at big sites anyway, just small ones like mine which were caught on the hop). We’re also going to ensure no apache process takes on too much at once by creating a limit of how many requests any process can take on – we’ll keep it low for now (3), but it can be increased later. So our changed config settings now look like this:

[cc escaped=”true” lang=”apache”]<IfModule mpm_prefork_module>
StartServers          4
MinSpareServers       4
MaxSpareServers       12
ServerLimit           96
MaxClients            96
MaxRequestsPerChild   3
</IfModule>
[/cc]

Okay. At this stage, you have two options. The first is to start Apache up again and get back to work. Odds are, this will hold up pretty well – but you want to keep a window open with htop running in the background to keep an eye on things (and mainly you’re watching the swap space usage and the load. The former’s critical, the latter indicative that a problem’s arising – if either go sideways, kill apache and edit apache2.conf setting even lower values for ServerLimit, MaxClients and MaxRequestsPerChild before restarting apache). If that’s your preferred option, skip to the end of this post.

However, if you want to take that extra step, we could install memcached quickly here. It’s a very effective load reducer and under debian, it’s far easier than you’d expect:

[cc lang=”bash”]sudo aptitude install build-essential php5-devel php-pear memcached[/cc]

And let that haul in whatever other libraries it needs, then:

[cc lang=”bash”]pecl install memcached[/cc]

And once that’s done, edit the php.ini file (in Debian, that’ll be /etc/php5/apache2/php.ini ) and insert this (anywhere in the file will do, but the extensions section is the tidiest):

[cc lang=”ini”]extension=memcache.so[/cc]

That should be memcached installed and running in a default configuration (we can finetune later). We now need to drop in the backend that WordPress uses to take advantage of memcached. Download object-cache.php and copy it into the wp-content directory of your website and change the permissions and ownership of the file:

[cc lang=”bash”]cd [insert your www/wp-content directory here]
sudo wget http://plugins.trac.wordpress.org/export/215933/memcached/trunk/object-cache.php
sudo chown www-data.www-data object-cache.php
sudo chmod 644 object-cache.php[/cc]

And that’s it done. Quick, dirty, and everything at default, but that’s a three-minute setup for you (well, maybe five if you do the memcached setup as well, and I am assuming you have a fast net connection for the aptitude step, but still).

Now, restart apache and everything should fire up with memcached caching a lot of requests and keeping the server load to a managable level.

[cc lang=”bash”]sudo /etc/init.d/apache2 force-reload
sudo /etc/init.d/apache2 restart[/cc]

And once that traffic spike is past… take the time to tune it properly!


02
Nov 07

Reaching the limits of PHP

First off, I rather like PHP. PHP 5 at least, I think PHP 4 was – and remains – a dog’s dinner of a thing at best. But for 90% (at least) of webpages, PHP is a pretty decent solution. Retrieval of data from databases, display of that data, no worries. Minimal learning curve, support for high-level constructs like objects and exceptions, several frameworks available – PHP’s a damn good choice in the vast majority of cases.

Thing is, right now I’m trying to put together a bit of code to act as a proof of concept, and – more fool me – I tried to use PHP to do it, figuring I’d save some time. No such luck. Turns out, trying to create data in PHP is a lot harder than trying to manipulate or display it. Right now, I’m in the middle of implementing a recognition system using a variable order markov model approach, and just generating the PPM-C prefix trees is a total pain in the fundament. Even the basic data structure I’m using – a trie – has no standard implementation in PEAR or the SPL or anywhere else I could find. So first I have to write that. Then I have to write the code to do all the other stuff. I mean, granted, hoping to find a pre-written PPM library was a bit hopeful, but prefix trees are basic data structures (and yes, I eventually found some sample code for one, but I’m still having to kick that about a fair bit to get what I need from it). But there was other stuff that I tried and ran into problems with. For example, PEAR::Math_Matrix seems to be PHP4 only, and the PEAR version doesn’t want to even parse in PHP5, let alone let me look for determinents or jacobians or eigenvalues. Which put yet another crimp in my plans earlier this week.

Anyway, for the actual production version of this project, PHP’s obviously out and never was in the running. This piece of code has to run as fast as possible – so I’m looking at C++ or C or something of that nature. But for proof of concept, I was hoping PHP was the way to go. Nope. This horse does not do well on this course. I’m thinking it’s time to start playing a bit more with Python or Perl or something for this sort of thing in the future. Which isn’t horrible – better to have more strings to your bow if at all possible. But it does rather shoot the whole fanboy attitude in the PHP-vs-Ruby-vs-Python-vs-Whatever language wars. There’s simply no place for it in a professional environment, and it’s not of any real use anywhere else either.

Still gonna do that rifle club management system project in PHP with Symfony or Prado (if they get the menu component done and released in time) though.

(And someone needs to tell Sean Coates that Microsoft is still evil ūüôā )