Nov 12

Using VimWiki as a distributed, encrypted lab notebook for programming

So what’s the most important coding tool for a software engineer in the long-term? There’s a pretty decent argument that it’s a lab notebook. I’ve kept one for years and it’s one of the more useful tools I have, but I found that my notebook, apart from not quite measuring up to how a lab notebook should be kept, wasn’t quite as useful as it could be because when you have five or six notebooks covering a bit over a decade’s worth of notes and no search function, then how do you use them daily?

I’ve been looking around for a while for a software version of a paper notebook without much success (this isn’t ludditism – we just don’t have very much that can match what paper and pen can do for this kind of task). I’ve used a simple text editor on a cron job for a few years as a way to track what I do during the day – every 30 minutes, up pops a vim window with a ready-inserted line listing the date and time, I scribble in what I’ve done since the last time the window popped up and I save and quit – it takes about 15 seconds and it gives me a file listing everything I’ve done at the end of the day/week/month/year which is handy for things like preparing reports, doing job reviews, that sort of thing. It’s simplicity itself to set up, just call a simple script using a simpler cronjob entry:

[cc escaped=”true” lang=”bash”]


# Cron script for 30-min activity journal

export DISPLAY=:0
echo -n -e “\n[” `date` “] :\n” >> ~/.journal
/usr/bin/gvim -U ~/.journal.gvimrc -geometry 100×40+512+400 + ~/.journal


[cc escaped=”true” lang=”bash”]

# m h dom mon dow command
0,30 8-20 * * mon-fri ~/.journal.sh


Simple and effective… but not a lab notebook. It does help test a few criteria for what I need that notebook program to do to fit in with my workflow though:

  • It has to be *fast* and familiar to use. ’nuff said, really. 
  • It has to be distributed. I work on lots of machines; I need it to be available on them, or else I’d have to have it on something like a netbook that I’d carry everywhere and that’s just not practical for me (besides, what if I forgot it or it got stolen?)
  • It has to be encrypted. Everything I work on is NDA’d at one level or another; and design/debugging notes would definitely be too sensitive to leave lying around in plaintext format. With a physical notebook, I keep it locked away; but this is supposed to be better than that option…

Colleagues have used various products for this over the years – mindmapping software; emacs in one mode or another; and various other software. But none of those really appealed. Mediawiki did seem to be as good a fit as I could find, but something that depends on an entire LAMP stack to run is hardly lightweight; and while I could host it somewhere public, that’s not really very secure (I’d spend more time making sure the full LAMP stack was up to date and mediawiki too than I want to). Besides, I’d rather this be console-accessible if possible (yes, some of us are still happier that way 🙂 ).

I’ve been using vim since around 1993 or so; at this point it’s wired into my fingers. So when I saw vimwiki, it seemed ideal. For those who’ve not encountered vimwiki before, it creates a directory, and then every file in that directory becomes part of a rudimentary text-based wiki (which it can turn into a set of HTML pages so it can handle images and so forth, but you can also navigate it from within vim). It also has a diary function which works in a sub-directory of the wiki directory.

It doesn’t have any support for encryption or distribution. But that’s quite solvable.

The encryption is easy enough – you could use the blowfish encryption in (post-v7.3) vim but that proved a bit awkward as you had to reenter the password every time you navigated down a link (and I don’t always have post-7.3 vim available). This password entering every minute or so broke up my workflow, so no thanks. My netbook and work laptops all have whole-disk-encryption, so I just left the vimwiki directory as normal on those laptops, and on the machines where I don’t have whole-disk-encryption, I use eCryptFS to create an encrypted directory and put the wiki under that. Very simple indeed, but quite effective. Now even theft of the physical hard drive isn’t a major concern.

The distribution was equally simple; you could use any DVCS, but I’m fond of mercurial, so I decided to use that. You have to tweak the vimwiki script ( .vim/ftplugin/vimwiki.vim ) to call it:

[cc escaped=”true” lang=”vim”]

augroup vimwiki
au! BufRead /home/mdennehy/vimwiki/index.wiki  !hg pull;hg update
au! BufWritePost /home/mdennehy/vimwiki/*  !hg add <afile>;hg commit -m ” “;hg push
augroup END


But that’s a simple tweak at best. And you want to have ssh setup with keys for the easiest workflow, but you have that already, right? 😀

Then just modify the crontab script:

[cc escaped=”true” lang=”bash”]

# Cron script for 30-min activity journal

export DISPLAY=:0
/usr/bin/gvim -U ~/.journal.gvimrc -geometry 100×40+512+400 -c “call vimwiki#diary#make_note(v:count1)” + -c “r !date +’\%n= \%H\%Mh =\%n'”[/cc]

And add an Awesome keybinding and menu entry:

[cc escaped=”true” lang=”lua”]

vimwiki_cmd = “/usr/bin/gvim -U /home/mdennehy/vimwiki/.gvimrc -c ‘call vimwiki#base#goto_index(v:count1)'”

mymainmenu = awful.menu({ items = { { “awesome”, myawesomemenu, beautiful.awesome_icon },

{ “VimWiki”,vimwiki_cmd }

awful.key({ modkey, }, “w”, function () awful.util.spawn(vimwiki_cmd) end),


And now whenever I hit <Mod4>-w from within Awesome, it pops up a gVim window open at the root of the wiki; every 30 minutes it pops up a gVim window in today’s diary page with the time inserted automatically for a log entry; and whenever I hit save or switch buffers, it syncs the files up to a central server’s encrypted area.

Distributed, encrypted, fast and useful. I’ve been using it in the job for the last few months now and it does almost everything I need. I do still keep around the paper notebook though – no matter how good the program, we still don’t have anything that can do everything paper can do (doodle, take cornell format notes, sketch diagrams easily for later capture, that sort of thing), but vimwiki’s search function alone is making it the day-to-day workhorse and it’s making my life a lot easier. Notes on development, patent ideas, job review reports, sysadmin notes, notes on papers I’m writing, and a daily log, all in one easy-to-use package. Damn useful tool.

May 10

Can the Nokia N900 replace a netbook?

One of the things I wanted to see with the N900 was whether or not it could replace the laptop or a netbook when travelling. In general when I’m travelling, my computer needs fall into one of three categories: real work (ie. coding); communications (generally video skype calls); and entertainment (generally podcasts and videos). For most weekends away, it’ll be communications and entertainment; on longer trips (like the training courses in kuortane) it’s mostly been those two with either real work or reviewing video footage added in.

So first off, how much kit do you have to carry with the N900? Well, basicly, the phone itself and the power charger, and maybe the tv-out cable. Very little weight, the power charger takes up about a third of the space of the phone (and is about as thick). So in terms of convenience, top marks. I don’t need to bring a speaker because (a) the speakers in the N900 are pretty decent; and (b) the FM transmitter lets me use any nearby radio as a speaker (and it works pretty well). And if there’s a TV around, the tv-out cable turns it into a large monitor, useful for reviewing video or entertainment.

Next, what’s the battery life like? Once at the destination, it’s not so big a deal, but if you’re not actually driving there (and until Ryanair introduce their reduced rates for self-piloted flights, you won’t be for most business trips), you do rather rely on sufficient battery life to get to wherever you’re going so you don’t wind up missing the last half of Foyle’s War because the battery ran flat. Nokia claim 5 hours talk time for the N900; I certainly didn’t manage to drain the battery that way (even I can’t talk for five hours), but a full charge certainly lasted through a three-hour train trip playing mp4 and avi video over headphones, with about another hour left in it according to the BatteryEye app.

So how about email? Well, the email app isn’t as polished as even the e71’s S60 gmail app, but it’s certainly usable. However, the keyboard does take getting used to. After a fortnight, I was still only at half to three-quarters the speed on the N900 as I was at on the e71. But the keyboard hasn’t any really serious flaws (though the ctrl-shift-p sequence to take a screenshot was a bit awkward and the lack of a tab key does tend to crop up a lot). So by the time the reboxing day arrived, I was happily able to tweet on the keyboard. In another week or two, I think I’d have been happy enough to write reasonably-sized emails, if more slowly than on the laptop. But for on-the-go communications, it would certainly have sufficed.

So, can it replace a netbook? Well, basicly, yes. It’s an excellent communications device, though bug 6063 quite soundly kicked my evaluation unit in the head. Email, IM, twitter, phone, skype; it converges all of those communication channels into one portable device and I do love it for that. It is more comfortable to consume content than create it on the N900, to be sure; but I’ve gotten used to mostly operating that way when on the go because of smaller screens and keyboards being a wee bit awkward anyway, and catching up on arrival somewhere. And one bluetooth fold-up keyboard and a hotel TV screen and the N900 tv-out cable and you have a larger screen and more comfortable keyboard to use.

Can it do real work like coding? Well, I’d love to say yes, but frankly, I doubt it. But that’s okay for me – there’s no point in pretending that if you just had a bigger battery or faster CPU that it’d be easier – the problem isn’t the device but where we are using it. On the go on a train or a plane is just not a great work environment for coding.

I think that that points out why the N900 pricetag is actually less insane than it first appears. The problem is that the N900 looks like a phone. It isn’t one. So you’re not paying €500 for a phone. You’re paying €500 for a very small netbook that can do phone calls, has a built-in FM radio and transmitter and GPS (btw, bloody handy when looking for your hotel in a strange town, Ovi maps is quite good for that kind of thing), and so on. Looked at that way, it’s actually not a bad price, it’s in fact right in the mid-range for netbooks in Ireland these days.

So yeah, it’s getting harder to not buy one 😀

Oct 08

New toy!

It finally arrived!

Thinkpad R61

Thinkpad R61

Continue reading →