Nov 12

Using VimWiki as a distributed, encrypted lab notebook for programming

So what’s the most important coding tool for a software engineer in the long-term? There’s a pretty decent argument that it’s a lab notebook. I’ve kept one for years and it’s one of the more useful tools I have, but I found that my notebook, apart from not quite measuring up to how a lab notebook should be kept, wasn’t quite as useful as it could be because when you have five or six notebooks covering a bit over a decade’s worth of notes and no search function, then how do you use them daily?

I’ve been looking around for a while for a software version of a paper notebook without much success (this isn’t ludditism – we just don’t have very much that can match what paper and pen can do for this kind of task). I’ve used a simple text editor on a cron job for a few years as a way to track what I do during the day – every 30 minutes, up pops a vim window with a ready-inserted line listing the date and time, I scribble in what I’ve done since the last time the window popped up and I save and quit – it takes about 15 seconds and it gives me a file listing everything I’ve done at the end of the day/week/month/year which is handy for things like preparing reports, doing job reviews, that sort of thing. It’s simplicity itself to set up, just call a simple script using a simpler cronjob entry:

[cc escaped=”true” lang=”bash”]


# Cron script for 30-min activity journal

export DISPLAY=:0
echo -n -e “\n[” `date` “] :\n” >> ~/.journal
/usr/bin/gvim -U ~/.journal.gvimrc -geometry 100×40+512+400 + ~/.journal


[cc escaped=”true” lang=”bash”]

# m h dom mon dow command
0,30 8-20 * * mon-fri ~/.journal.sh


Simple and effective… but not a lab notebook. It does help test a few criteria for what I need that notebook program to do to fit in with my workflow though:

  • It has to be *fast* and familiar to use. ’nuff said, really. 
  • It has to be distributed. I work on lots of machines; I need it to be available on them, or else I’d have to have it on something like a netbook that I’d carry everywhere and that’s just not practical for me (besides, what if I forgot it or it got stolen?)
  • It has to be encrypted. Everything I work on is NDA’d at one level or another; and design/debugging notes would definitely be too sensitive to leave lying around in plaintext format. With a physical notebook, I keep it locked away; but this is supposed to be better than that option…

Colleagues have used various products for this over the years – mindmapping software; emacs in one mode or another; and various other software. But none of those really appealed. Mediawiki did seem to be as good a fit as I could find, but something that depends on an entire LAMP stack to run is hardly lightweight; and while I could host it somewhere public, that’s not really very secure (I’d spend more time making sure the full LAMP stack was up to date and mediawiki too than I want to). Besides, I’d rather this be console-accessible if possible (yes, some of us are still happier that way 🙂 ).

I’ve been using vim since around 1993 or so; at this point it’s wired into my fingers. So when I saw vimwiki, it seemed ideal. For those who’ve not encountered vimwiki before, it creates a directory, and then every file in that directory becomes part of a rudimentary text-based wiki (which it can turn into a set of HTML pages so it can handle images and so forth, but you can also navigate it from within vim). It also has a diary function which works in a sub-directory of the wiki directory.

It doesn’t have any support for encryption or distribution. But that’s quite solvable.

The encryption is easy enough – you could use the blowfish encryption in (post-v7.3) vim but that proved a bit awkward as you had to reenter the password every time you navigated down a link (and I don’t always have post-7.3 vim available). This password entering every minute or so broke up my workflow, so no thanks. My netbook and work laptops all have whole-disk-encryption, so I just left the vimwiki directory as normal on those laptops, and on the machines where I don’t have whole-disk-encryption, I use eCryptFS to create an encrypted directory and put the wiki under that. Very simple indeed, but quite effective. Now even theft of the physical hard drive isn’t a major concern.

The distribution was equally simple; you could use any DVCS, but I’m fond of mercurial, so I decided to use that. You have to tweak the vimwiki script ( .vim/ftplugin/vimwiki.vim ) to call it:

[cc escaped=”true” lang=”vim”]

augroup vimwiki
au! BufRead /home/mdennehy/vimwiki/index.wiki  !hg pull;hg update
au! BufWritePost /home/mdennehy/vimwiki/*  !hg add <afile>;hg commit -m ” “;hg push
augroup END


But that’s a simple tweak at best. And you want to have ssh setup with keys for the easiest workflow, but you have that already, right? 😀

Then just modify the crontab script:

[cc escaped=”true” lang=”bash”]

# Cron script for 30-min activity journal

export DISPLAY=:0
/usr/bin/gvim -U ~/.journal.gvimrc -geometry 100×40+512+400 -c “call vimwiki#diary#make_note(v:count1)” + -c “r !date +’\%n= \%H\%Mh =\%n'”[/cc]

And add an Awesome keybinding and menu entry:

[cc escaped=”true” lang=”lua”]

vimwiki_cmd = “/usr/bin/gvim -U /home/mdennehy/vimwiki/.gvimrc -c ‘call vimwiki#base#goto_index(v:count1)'”

mymainmenu = awful.menu({ items = { { “awesome”, myawesomemenu, beautiful.awesome_icon },

{ “VimWiki”,vimwiki_cmd }

awful.key({ modkey, }, “w”, function () awful.util.spawn(vimwiki_cmd) end),


And now whenever I hit <Mod4>-w from within Awesome, it pops up a gVim window open at the root of the wiki; every 30 minutes it pops up a gVim window in today’s diary page with the time inserted automatically for a log entry; and whenever I hit save or switch buffers, it syncs the files up to a central server’s encrypted area.

Distributed, encrypted, fast and useful. I’ve been using it in the job for the last few months now and it does almost everything I need. I do still keep around the paper notebook though – no matter how good the program, we still don’t have anything that can do everything paper can do (doodle, take cornell format notes, sketch diagrams easily for later capture, that sort of thing), but vimwiki’s search function alone is making it the day-to-day workhorse and it’s making my life a lot easier. Notes on development, patent ideas, job review reports, sysadmin notes, notes on papers I’m writing, and a daily log, all in one easy-to-use package. Damn useful tool.

Mar 10

Performance tuning a server in less than three minutes while being slashdotted

So you wrote a blog post about something that seemed fairly innocuous, but for whatever reason, it caught the attention of one of the major sites and now your server load is at 110 and climbing, the ssh command line session is taking thirty seconds to respond to anything at all, and given that your post is on the front page of slashdot at primetime, this doesn’t look like it’s a temporary blip. What do you do?

Burning computer

Okay, first things first. You don’t have time to do a proper fine-tuning session. You need a quick & dirty tuneup here. Proper fine tuning you leave till after the traffic spike, and you can then come back at it with a plan and decent tools like siege and so on – but this is the “fix it in three minutes” version. So if you see stuff here that looks crude to you, despair not, everything looks crude when you’re trying to do it in that kind of timeframe.

First, you have to be able to actually use the server command line. The load’s at 110 – that’s not coming down anytime soon and until it does you can’t do the rest of the work you need to since it takes so much time for even a shell to respond. The load is being caused by Apache and MySQL taking on more work than the server can handle, causing the server to swap excessively; and you’ve got to dump that work or shut off the incoming new work to recover. You can try sudo killall -9 apache2 if you can get an open ssh terminal in to do the job (and it’s the first thing you should try), but the odds are that that server has to be reset. Whether that means a phone call to the data centre, or a walk down the hall to the server room, or just clicking a button in a web interface, that’s the first thing to do. Don’t hold off, because unless everyone stops reading your page right now, that load’s not coming down.

Once the box has rebooted (and I mean immediately – sit there watching ping until it comes back), ssh in and shut down apache. MySQL is okay for now, but the work is coming from Apache, so that has to get shut down until everything’s ready to go again. You’re going to lose a few minutes of service, yes, but that’s recoverable from for a blog (and if this is for something more serious than a blog, you’re going to be in trouble for not properly spec’ing the hardware to begin with anyway, so limit the damage now and take your licks later).

At this point — whether you’ve just logged in or whether you managed to run killall successfully —  if it’s a debian server you should run sudo /etc/init.d/apache2 stop (did I mention I love debian for having scripts like that? No man-paging apachectl, just an easy to remember standard interface for every service on the box. Wonderful). It’ll tidy up from killall, or it’ll shut down apache cleanly, depending on how you got here.

I’m going to use my server as the example here, by the way, since it’s what got burned last night and it’s what prompted me to write this refresher — it’s been two or three years since I last had to maintain a server that was near its capacity, the experience was a bit of a flashback 🙂 So, some background on my server – I moved my blog from wordpress.com to here, on a Hetzner server. It’s their entry-level dedicated server offering 2Gb of RAM, a 64-bit Athlon, 160Gb of hard drive in a hardware RAID-1 array and a 1Gbit NIC) — all running Debian Lenny (and no, I’ve no relationship with Hetzner, they were just the cheapest of the places various friends recommended). WordPress is up to date on my server (2.9.2 at the time of writing), as is the Lenny install — if you don’t have the latest security fixes and such in place, or your WordPress is outdated, then that’s probably adding to your problem, but for a quick fix like this, that’s too big a job. Get through the traffic spike and deal with it later.

And yes, that server spec is overkill for my needs really – but I had a bunch of side projects like RangeClerk (don’t bother, not much is up yet) and the blog for Herself Indoors and her book and some other things I wanted to run as well that would be using wierd php and python modules and libraries and the like; and I just hate cpanel and not being able to install anything I wanted. Plus, it was cheap 😀

Right, back to it.

The first thing we need to do is to sort out MySQL’s configuration. Open up the my.cnf file, whereever you’ve put it (it’ll be /etc/mysql/my.cnf for a stock Debian install). We need to tweak just a few settings. First off, key_buffer. This is probably the most critical variable here because by default all the tables will be MyISAM tables (if they’re not, then this isn’t so critical). It’s set to about 16Mb by default; we’re going to turn that up quite a bit. On a dedicated database box, this would be set very high – anything up to 50% of the total available memory. In this case, with a full stack on the one box, we set it a bit lower since Apache’s going to want a lot of RAM too – 256Mb will do for a starting value.

Next we’re going to disable the InnoDB engine completely to cut down on MySQL’s footprint. Again, WordPress by default isn’t using it. Just ensure skip-innodb is either uncommented or inserted into my.cnf.

Lastly, we’re going to enable the query cache. The thing is, MySQL’s query cache is a fairly blunt instrument. If a query is precisely the same the second time it comes in, it’ll hit the cache – but any change at all, no matter how small, and it misses the cache. So it’s not as enormously useful as you’d first imagine. However, it does help, so we’ll increase its size modestly (48Mb of RAM is sufficient here). So our changes to my.cnf look like so:

[cc lang=”ini”]
key_buffer = 256M
query_cache_limit = 16M
query_cache_size = 48M

Once those changes are made, sudo /etc/init.d/mysql restart will get the MySQL server up and running with the new setup.  Once that’s done, let’s look to the next level in the stack – Apache. Under debian the config files are arranged differently than normal; the configuration changes we’ll make will be in /etc/apache2/apache2.conf but in other installations they would be in httpd.conf or elsewhere.

The default Apache install uses the prefork MPM setup – one thread per process. It’s older, slower, less efficient, but less buggy than the worker MPM which isn’t threadsafe. So find the prefork MPM config settings in apache2.conf. They should look like this in a default install:

[cc escaped=”true” lang=”apache”]
<IfModule mpm_prefork_module>
StartServers          5
MinSpareServers       5
MaxSpareServers      10
MaxClients          150
MaxRequestsPerChild   0

We’re going to cut down a lot on how much work Apache takes on at once here. Yes, some users will have to wait a few seconds to see your page – but right now, with the load at 110 and climbing, they could wait until their browser timed out and they’d never see anything. So we reduce slightly the number of servers that Apache will farm off to handle requests at any one time from 5 to 4; we’ll increase the number of spare servers it’ll keep around to hand off requests to (we want to reduce the overhead of starting and stopping those processes) from 10 to 12. We’ll set an upper limit on how many we can have though, and we’ll keep it to just under 100. This works on my system, which is an entry-level system; you might get away with more, but for now use these settings and it’ll get you up and running and you can increase a bit and check again as you go (and this guide really isn’t aimed at big sites anyway, just small ones like mine which were caught on the hop). We’re also going to ensure no apache process takes on too much at once by creating a limit of how many requests any process can take on – we’ll keep it low for now (3), but it can be increased later. So our changed config settings now look like this:

[cc escaped=”true” lang=”apache”]<IfModule mpm_prefork_module>
StartServers          4
MinSpareServers       4
MaxSpareServers       12
ServerLimit           96
MaxClients            96
MaxRequestsPerChild   3

Okay. At this stage, you have two options. The first is to start Apache up again and get back to work. Odds are, this will hold up pretty well – but you want to keep a window open with htop running in the background to keep an eye on things (and mainly you’re watching the swap space usage and the load. The former’s critical, the latter indicative that a problem’s arising – if either go sideways, kill apache and edit apache2.conf setting even lower values for ServerLimit, MaxClients and MaxRequestsPerChild before restarting apache). If that’s your preferred option, skip to the end of this post.

However, if you want to take that extra step, we could install memcached quickly here. It’s a very effective load reducer and under debian, it’s far easier than you’d expect:

[cc lang=”bash”]sudo aptitude install build-essential php5-devel php-pear memcached[/cc]

And let that haul in whatever other libraries it needs, then:

[cc lang=”bash”]pecl install memcached[/cc]

And once that’s done, edit the php.ini file (in Debian, that’ll be /etc/php5/apache2/php.ini ) and insert this (anywhere in the file will do, but the extensions section is the tidiest):

[cc lang=”ini”]extension=memcache.so[/cc]

That should be memcached installed and running in a default configuration (we can finetune later). We now need to drop in the backend that WordPress uses to take advantage of memcached. Download object-cache.php and copy it into the wp-content directory of your website and change the permissions and ownership of the file:

[cc lang=”bash”]cd [insert your www/wp-content directory here]
sudo wget http://plugins.trac.wordpress.org/export/215933/memcached/trunk/object-cache.php
sudo chown www-data.www-data object-cache.php
sudo chmod 644 object-cache.php[/cc]

And that’s it done. Quick, dirty, and everything at default, but that’s a three-minute setup for you (well, maybe five if you do the memcached setup as well, and I am assuming you have a fast net connection for the aptitude step, but still).

Now, restart apache and everything should fire up with memcached caching a lot of requests and keeping the server load to a managable level.

[cc lang=”bash”]sudo /etc/init.d/apache2 force-reload
sudo /etc/init.d/apache2 restart[/cc]

And once that traffic spike is past… take the time to tune it properly!

Oct 09

VisTablet under Debian Squeeze


A new toy arrived today – the VisTablet, which is just a rebadged Aiptek 600U (model number WCK-C121). They sell as Aiptek or Waltop or Medion or any one of a few other names for about €65 on ebay.ie at the moment, so I thought what the heck, I do a fair bit of stuff in Gimp so maybe it’s worth a look. A few clicks and a week or so later and it arrives. Cue tearing open the box and unwrapping!

Plugging it into the laptop (Thinkpad R61 running Debian Squeeze amd64) got an instant result in that it found the device and allowed cursor movement… but no actual buttons or pressure sensor stuff, so it couldn’t work as a tablet. So off to google we go and after trying some frustrating messing about with udev rules (to create /dev/apitektablet – which isn’t necessary it turns out) and hal fdi files (to set up X.org options, which are needed) and I’d advanced in a posterior direction.

By now we were through the phase where plugging in the device crashed X and had reached the phase where plugging the device out crashed X. At this point, I found that the problem lay in the wacom drivers. See, just because it’s aiptek hardware doesn’t mean the aiptek drivers actually work – in this case, the wacom driver works much better.

But, just to be annoying, the new version of the wacom driver doesn’t work because it picks up that it’s not a wacom device and commits seppku to avoid being of any use. Note to the coder who thought that was a good idea – go choke on a hairball please.

Solution? Go to linuxwacom and download the latest driver, then untar the tarball, go into linuxwacom-0.8.4-2/src/xdrv and edit wcmUSB.c. You’re looking for this line (it’s line 530 in version
[cc escaped=”true” lang=”c”]/* vendor is wacom */
if (sID[1] == 0x056A)
And you want to change it to this:
[cc escaped=”true” lang=”c”]/* vendor is wacom or waltop */
if (sID[1] == 0x056A || sID[1] == 0x172f)
Now apt-get install xorg-dev and whatever other packages you need like build-essential and call configure and make and then sudo make install.

You now have a working wacom driver and if you copy the wacom.fdi file into /etc/hal/fdi/policy/ (it’s /usr/share/hal/fdi/policy/20thirdparty/10-linuxwacom.fdi on Debian) and add in the following bit:
[cc escaped=”true” lang=”xml”]<match key=”info.product” contains=”WALTOP”>
<merge key=”input.x11_driver” type=”string”>wacom</merge>
<merge key=”input.x11_options.Type” type=”string”>stylus</merge>
<append key=”info.callouts.add” type=”strlist”>hal-setup-wacom</append>
<append key=”wacom.types” type=”strlist”>eraser</append>
<append key=”wacom.types” type=”strlist”>cursor</append>
<append key=”wacom.types” type=”strlist”>pad</append>
into the <device> section of the xml file (you probably don’t need the eraser and cursor lines, but what the heck, they don’t harm anything and I was too tired of restarting X at this point to check). You should now be almost done – restart hal, restart x.org, and log back in and then plug in the tablet (check dmesg and /var/log/X.0.log so you’re sure it worked) and then fire up Gimp, go to Edit->Preferences->Input Devices->Configure Extended Input Devices and make sure the tablet is listed there and that it’s not set to disabled, then go to Edit->Preferences->Input Controllers and make sure that Linux Input is in the list of active controllers and is set to look at the tablet, and it should work.


Mind you… if this is what it takes to make cheap hardware work on linux, Linux Hater has a very good point.

Stochastic Geometry is Stephen Fry proof thanks to caching by WP Super Cache