Thomas Leonard's blog

CueKeeper: Gitting Things Done in the Browser

Git repositories store data with history, supporting replication, merging and revocation. The Irmin library lets applications use Git-style storage for their data. To try it out, I've written a GTD-based action tracker that runs entirely client-side in the browser.

CueKeeper uses Irmin to handle history and merges, with state saved in the browser using the new IndexedDB standard (requires a recent browser; Firefox 37, Chromium 41 and IE 11.0.9600 all work, but Safari apparently has problems if you open the page in multiple tabs).

In the future, I plan to have the browser sync to a master Git repository and use the browser storage only for off-line use, but for now note that:

  • All data is stored only in your browser.
  • There is no server communication.
  • Any changes you make will persist for you, but will not affect other users.
  • Mozilla's IndexedDB docs say that "the general philosophy of the browser vendors is to make the best effort to keep the data when possible", but vaguely notes that your data may be deleted if you run out of space! If someone can clarify things, that would be great. I've been using it for 5 weeks on Firefox, and haven't lost anything, but it would be nice to know the exact conditions for safety.
  • Take backups! On my Linux/Firefox system, the data is stored here: $HOME/.mozilla/firefox/SALT.default/storage/default/http+++roscidus.com/idb
  • This is version 0.1 alpha ;-)

This post contains a brief introduction to using GTD and CueKeeper, followed by a look at some nice features that result from using Irmin. The code is available at https://github.com/talex5/cuekeeper. Alpha testers welcome!

Table of Contents

( this post also appeared on Hacker News and Reddit )

Background

Getting Things Done (GTD)

The core idea behind David Allen's GTD is: the human brain is terrible at remembering things at the right time:

  1. You go into work in the morning thinking about a phone call you need to make this evening.
  2. As you read through your emails, you keep reminding yourself to remember the call.
  3. You're in a meeting and someone is speaking. You're thinking you shouldn't forget the call.
  4. etc

Maybe you end up remembering and maybe you don't, but either way you've distracted yourself all day from the other things you wanted to work on.

The goal of using GTD is to have a system where:

  1. the system reminds you of things when you need to be reminded about them, and
  2. you trust it enough that your brain can stop thinking about them until then.

There is no reason ever to have the same thought twice, unless you like having that thought.

David Allen Getting Things Done

Irmin

Irmin is "a library for persistent stores with built-in snapshot, branching and reverting mechanisms". It has multiple backends (including one that uses a regular Git repository, allowing you to view and modify your application's data using the real git commands).

Git's storage model is useful for many applications because it gives you race-free updates (each worker writes to its own branch and then merges), disconnected operation, history, remote sync and incremental backups.

Using js_of_ocaml I was able to compile Irmin to JavaScript and run it in the browser, adding a new IndexedDB backend.

mGSD

Simon Baird's mGSD is an excellent GTD system, which I've been using for the last few years. It's a set of extensions built on the TiddlyWiki "personal wiki" system. Like CueKeeper, mGSD runs entirely in your browser and doesn't require a server. It's implemented as a piece of self-modifying HTML that writes itself back to your local disk when you save. That's pretty scary, but I've found it surprisingly robust.

However, it's largely unmaintained and there were various areas I wanted to improve:

Browser security
Over the years, browsers have become more locked down, and no longer allow web-pages to write to the disk, requiring a browser plugin to override the check. CueKeeper uses the IndexedDB support that modern browsers provide to store data (mGSD pre-dates IndexedDB).
History
Sometimes you click on a button by mistake and have no idea what changed. Thanks to Irmin, CueKeeper logs all changes and provides the ability to view earlier states. Also, CueKeeper uses brief animations to make it easier to see what changed.
Navigation
Navigation in mGSD can be awkward because overview panels and details are all mixed in together as wiki pages. With CueKeeper, I'm experimenting with a two-column layout to separate overview pages from the details.
Safe multi-tab use
If you accidentally open mGSD in two tabs, changes in one tab will overwrite changes made in the other. CueKeeper uses Irmin to keep multiple tabs in sync, merging changes between them automatically.
Sync between devices
There's no easy way to Sync multiple mGSD instances. CueKeeper doesn't implement sync yet either, but it should be easy to add (it can sync between tabs already, so the core logic is there).
Escaping bugs
mGSD has various bugs related to escaping (e.g. things will go wrong if you use square brackets in a title). CueKeeper uses type-safe TyXML to avoid such problems.
Stale-display bugs
mGSD mostly does a good job of keeping all elements of the display up-to-date, but there are some flaws. For example, if you add a new contact in one panel, then open the contacts menu from another, the new contact doesn't show up. CueKeeper uses Functional reactive programming with the React library to make sure everything is current.
Clean separation of code and data
As a self-modifying .html file, updating mGSD is terrifying! CueKeeper can be recompiled and reloaded like any other program.

Nymote, MirageOS and UCN

The Nymote project describes itself as "Lifelong control of your networked personal data":

By adopting large centralised services we've answered the call of the siren servers and made an implicit trade. That we will share our habits and data with them in exchange for something useful. In doing so we've empowered internet behemoths while simultaneously reducing our ability to influence them. We risk becoming slaves to the current system unless we can create alternatives that compete. It's time to work on those alternatives.

The idea here is to provide services that people can run in their own homes (e.g. on a PC, a low-powered ARM board, or the house router). The three key pieces of infrastructure it needs are Mirage, Irmin and Signpost.

I've talked about MirageOS before (see My first unikernel): it allows you to run extremely small, highly secure services as Xen guests (a few MB in size, written in type-safe OCaml, rather than 100s of MB you would have with a Linux guest). I haven't looked at Signpost yet. Irmin is the subject of this blog post.

UCN (User Centric Networking) is an EC-funded project that is building a "Personal Information Hub" (PIH), responsible for storing users' personal data in their home, and then using that data for content recommendation. If you use Google to manage your ToDo-list then when you add "Book holiday" to it, Google can show you relevant ads. But what if you want good recommendations without sharing personal data with third parties? Tools such as CueKeeper could be configured to sync with a local PIH to provide input for its recommendations without the data leaving your home.

Using CueKeeper

You can either use the example on roscidus.com, or download the standalone release cuekeeper-bin-0.1.zip. To use the release, unzip the directory and open index.html in a browser (no need for a web-server). If you do this, note that the database is tied to the path of the file, so if you move or rename the directory, it will show a different database (which might make it look like your items have disappeared).

Core concepts

There are five kinds of "thing" in CueKeeper:

Action

Something you will do (e.g. "Follow Mirage tutorial").

Beside each action you will see some toggles showing its state: The tick means done, "n" is a next action (something you could start now), "w" means waiting-for (something you can't start now), "f" means future (something you don't want to think about yet). The star is for whatever you want. Repeating actions can't be completed, so for those the tick box will be blank.

Project

Something you want to achieve (e.g. "Make a Mirage unikernel").

A project may require several actions to be taken. The possible states are done (the tick), "a" for active projects, and "sm" for "Someday/Maybe" (a project you don't plan to work on yet).

Area

An "Area of responsibility" is a way of grouping things (e.g. "Personal/Hobbies" or "Job/Accounts").

Unlike projects, areas generally cannot be completed. One thing that confused me when I started with GTD was that what my organisation called "projects" were actually areas. If your boss says "You're working on project X until further notice" then "X" is probably an "area" in GTD terms.

Contact

Someone you work with.

You can associate any area, project or action with a contact, which provides a quick way to find all the things you need to discuss with someone when you meet them. If an action is being performed by someone else, you can also mark it as waiting for them. It will then appear on the Review/Waiting list.

Context

Another way of grouping actions, by what kind of activity it is, or where it will occur.

Assigning a context to an action is an important check that the action isn't too vague. Your eye will tend to glide over vague actions like "Sort out car"; choosing a context "Phone" (garage) or "Shopping" (buy tools) forces you to clarify things.

Notes:

  • GTD also has the concept of a "tickler". In CueKeeper this is just an action waiting until some time.
  • GTD also has "reference material", but I never used this in mGSD, so I didn't implement it. Regular files on your computer seem to work fine for this.
  • mGSD has the concept of "realms" to group areas. CueKeeper uses sub-areas for this instead (e.g. CueKeeper's "Personal/Health" sub-area corresponds to an mGSD "Health" area within a "Personal" realm).

Editing items

Clicking on an item or creating a new one opens a panel showing its details in the right column. There are various things you can edit here:

  • The toggles here work just as elsewhere (see above).
  • Click the title to rename.
  • Click (edit) to edit the notes. These can be whatever you like. They're in Markdown format, so you can add structure, links, etc.
  • Click (add log entry) to start editing with today's date added at the end. This is convenient to add date-stamped notes quickly.
  • The (delete) button at the bottom will remove it (without confirmation; use Show history to revert accidental deletions, as explained later).

For areas, projects and actions:

  • You can convert between these types by clicking on the type (e.g. "An action in ..."). This is useful if e.g. you realise that an action is really a project with multiple steps.
  • Click on the parent to move it to a different parent.
  • You can set the contact field for any of these types too.

For actions, you can also set the context, which is useful for grouping actions on the Work page, and helps to make sure the action is well-defined.

You can also make an action repeat. Setting the repeat for an action will move it to the waiting state until the given date. There are only two differences between repeating actions and regular (one-shot) scheduled actions:

  • You can't mark a repeating action as done (clear the repeat first if you want to).
  • When you click on the "w" on a repeating action, the next repeat date after it was last scheduled is highlighted by default. If that date has already arrived, it keeps moving it forward by the specified interval until it's in the future.

Processing

There are several stages to applying GTD, corresponding to the tabs along the top. The first is processing, which is about going through your various inboxes (email, paper, voicemail, etc) and determining what actions each item requires. After processing, your inbox should be empty and everything you need to do either done (for quick items) or recorded in CueKeeper. Also, see if you can think of any projects or actions that are only in your head and add those too.

  1. Click the + next to an area and enter a name for the new project.
    (the Work tab will go red at this point, indicating an alert: "Active project with no next action")
  2. Click (edit) in the new project panel to add some details, if desired.
  3. Click +action to add the next action to perform towards this project.

Note that it is not necessary to add all the actions needed to complete the project. Just add the next thing that you can do now. When you later mark the action as done, CueKeeper will then prompt you to think about a new next action.

If a project will only require a single action (e.g. "Buy milk"), then instead of adding a project and an action, you can just convert the new project to an action and not bother about having a project at all.

If you don't plan to work on the project soon, click "sm" to convert it to a "Someday/Maybe" project.

Work

This is the default view, showing all the things you could be working on now.

The filters just below the tab allow you to hide top-level areas (e.g. if you don't want to see any personal actions while you're at work).

When an item is done, click on the tick mark.

If it's not possible to start it now, click on the "w" to mark it as waiting:

  • If an action is waiting for someone else, first add them as the contact, then click the w and select "Waiting for name" from the menu.
  • If an action can't be started until some date, click the "w" and choose the date from the popup calendar.
  • Otherwise, you can mark it as "Waiting (reason unspecified)".

If you're not going to do it this week, click on the "f" (future) to defer it until the next review.

Contact

This view lists your contacts and any actions you're waiting for them to do. It's useful if someone phones and you want to see everything you need to discuss with them, for example. The list only shows actions you're actually waiting for, but if you open up a particular contact then you'll also see things they're merely associated with.

Schedule

Lists actions than can't be done until some date.

When due, scheduled actions will appear highlighted on the Work tab (even if their area is filtered out). If you pin the browser tab showing the CueKeeper page, the tab icon will also go red to indicate attention is needed. If you want to test the effect, schedule an action for a date in the past. Click n to acknowledge a due action and convert it to a next action.

The weekly review

GTD only works if you trust yourself to look at the system regularly. There are various reports available under the Review tab to help with this.

The available reports are:

  • Done shows completed actions and projects and provides a button to delete them all. If you're the sort of person who likes to write weekly summaries, this might be useful input to that.
  • Waiting shows actions that are waiting for someone or something (but not scheduled actions). You might want to check up on the status of these, or do something to unblock them.
  • Future shows all actions you marked as "Future" and all projects you marked as "Someday/Maybe".
  • Areas lists all your areas of responsibility.
  • Everything shows every item in the system in one place (you don't need to review this; it's just handy sometimes to see everything).

mGSD has more reports, but these are the ones I use. The default configuration has a repeating action scheduled for next Sunday to review things. This is what I do:

  • Process tab
    Empty inboxes, adding any actions to CueKeeper:
    • email inbox
    • paper inbox
  • Review/Done
    • Admire done list, then delete all.
  • Review/Waiting
    • Any reminders needed?
  • Review/Future
    • Make any of these current?
    • Delete any that will never get done.
  • Review/Areas
    • Any areas that need new projects?
  • Work
    • Make sure each action is obvious (not vague).
    • Could it be started now? Set to Waiting if not.
    • List too long? Mark some actions as Future, or their projects as Someday/Maybe.

It's important to look at all these items during the review. Knowing you're going to look at each waiting or future item soon is what allows you to forget about them during the rest of the week!

The top-right controls

To search, enter some text (or a regular expression) into the box and select from the drop-down menu that appears. Pressing Return opens the first result.

To create a new items, enter a label for it and select one of the "Add" items from the menu. Pressing Return when there are no search results will create a new action.

Export allows you to save the current state (without history) as a tar file. There's no import feature currently, though.

Show history shows some recent entries from the Irmin log (see below).

Interesting Irmin features

So, what benefits do we get from using Irmin?

Sync

The first benefit, of course, is that we can synchronise between multiple instances. You may have already tried opening CueKeeper in two windows (of the same browser) and observed that changes made in one propagate to the other. Here's an easier way to experiment with sync (click the screenshot for the interactive version):

This page has two instances of CueKeeper running, representing two separate devices such as a laptop and mobile phone. You can edit them separately and then click the buttons in the middle to see how the changes are merged.

Clicking Upper to lower pushes all changes from the upper pane to the lower (the lower instance will merge them with its current state). Clicking Lower to upper does the reverse. A full sync would do these two in sequence, but of course it could be interrupted part way through.

The "Criss-cross" button can be used to test the unusual-but-interesting case of merging in both directions simultaneously (i.e. each instance merges with the previous state of the other instance, generating two new merges). CueKeeper tries to merge deterministically, so that both instances should end up in the same state, avoiding unnecessary conflicts on future merges.

Where you make conflicting edits, CueKeeper will pick a suitable resolution and add a conflict note to say what it did. For example, if you edit the title of the "Try OCaml tutorials" action to different strings in each instance and then sync, you'll see something like:

CueKeeper uses a three-way merge - the merge algorithm takes the states of the two branches to be merged and their most recent common ancestor, and generates a new commit from these. The common ancestor is used to determine which branch changed which things (anything that is the same as in the common ancestor wasn't changed on that branch). If there are multiple possible ancestors (which can happen after a criss-cross merge) we just pick one of them.

CueKeeper has a unit test for merging that repeatedly generates three commits at random and ensures the merge code produces a valid (loadable) result. This should ensure that we can merge any pair of states, but it can't check that the result will necessarily seem sensible to a human, so let me know if you spot anything odd!

History

We have the full history, which you can view with the Show history button:

The history view is useful if you clicked on something by accident and you're not sure what you did. Click on an entry to see the state of the system just after that change. A box appears at the top of the page to indicate that you're in "time travel" mode - close the box to return to the present.

If you edit anything while viewing a historical version, CueKeeper will commit against that version and then merge the changes to master and return to the present.

You might like to open each instance's history panel while trying the sync demo above.

Revert

When in time-travel mode, you can click on the Revert this change button there to undo the change.

Reverting was easy to add, as it reuses the existing three-way merge code. The only difference is that the "common ancestor" is the commit being reverted and the parent of that commit is used as the "branch" to be merged.

Because CueKeeper can merge any three commits, it can also revert any commit (with a single parent), although you'll get the most sensible results if you revert the most recent changes first.

For example, if you create an action and then modify it, and then revert the creation then CueKeeper will see that as:

  • One branch that modified the action (the main branch).
  • One branch that deleted the action (the revert of the creation).

When something is modified and deleted, CueKeeper will opt to keep it, so the effect of the "revert" will simply be to add a note that it decided to keep it. Of course, the sensible way to delete something is to use the regular (delete) button.

Check-before-merge

It's important to make sure that the system doesn't get into an inconsistent state, and Irmin can help here. Whenever CueKeeper updates the database, it first generates the new commit, then it loads the new commit to check it works, then it updates the master branch to point at the new commit.

This means that CueKeeper will never put the master branch into a state that it can't itself load.

Out-of-date UI actions

Perhaps the most interesting effect of using Irmin is that it eliminates various edge cases related to out-of-date UI elements. Consider this example:

  1. You open a menu in one tab to set the contact for an action.
  2. You delete one of the contacts in another tab.
  3. You choose the deleted contact from the menu in the first tab.

With a regular database, this would probably result in some kind of error that you'd need to handle. These edge cases don't occur often and are hard to test.

With CueKeeper though, we record which revision each UI element came from and commit against that revision. We then merge the new commit with the master branch, using the existing merge logic to deal with any problems (normally, there is nothing to merge and we do a trivial "fast-forward" merge here). This means we never have to worry about concurrent updates.

A similar system is used with editable fields. When you click on a panel's title to edit it, make some changes and press Return, we commit against the version you started editing, not the current one. This means that CueKeeper won't silently overwrite changes, even if you edit something in two tabs at the same time (you'll get a merge conflict note containing the version it discarded instead).

Next steps

If you'd like to help out, there's still plenty more to do, both coding and testing. For example:

  • Editing doesn't work well on mobile phones. Menus and input boxes should fill the screen in this case.
  • I've had reports that merging between tabs is unreliable on Safari for some reason (AbortError from IndexedDB).
  • It would be good to use pack files for compression. Needs a JavaScript compression library.
  • It should be possible for an action to be marked as waiting for some other action or project to be completed.
  • Remote sync needs to be implemented.
  • The UI needs some work. In particular, could someone find a tasteful way to style the fields in the panels so they look like drop-downs? I keep clicking on the item's name instead of the (show) button by mistake (although this might be because I used to have it the other way around, with a (change) button, but that was worse).
  • CueKeeper's IndexedDB Irmin backend should be split off so other people can use it easily.

If you'd like to help out, the code is available at https://github.com/talex5/cuekeeper and discussion happens on the MirageOS-devel mailing list. If there's interest, I may write a follow-up post documenting my experiences implementing CueKeeper (using Irmin, React, js_of_ocaml and IndexedDB).

Acknowledgements

Some of the research leading to these results has received funding from the European Union's Seventh Framework Programme FP7/2007-2013 under the UCN project, grant agreement no 611001.