Personal information management

post by John_Maxwell_IV · 2012-09-11T11:40:53.747Z · score: 18 (19 votes) · LW · GW · Legacy · 31 comments

Several weeks ago, I began using personal wiki software Zim Wiki (free and cross-platform for Linux & Windows; I recommend nvALT on Mac OS X) to record all of my notes-to-self.  I've found it to be a nice software tool for implementing some of the effectiveness advice I've read on Less Wrong.  This post is a fairly personal overview of my usage.

I looked at a lot of personal information managers before choosing Zim.  Here are the features that caused me to choose it over the other software I looked at:

Using Zim has meant a really low level of friction for writing new stuff and retrieving/reading/adding to stuff I wrote.  I've been using it about a month and I've got ~46K words in it, which seems to be around the length of a short novel. RescueTime says I use it 4-8 hours per week.  Some stuff I'm using it for:

It's hard to measure how much benefit I'm getting out of all this, though it feels pretty useful.  I'm inclined to agree with Paul Graham:

...actually there is something druglike about [the notebook and pen], in the sense that their main purpose is to make me feel better. I hardly ever go back and read stuff I write down in notebooks. It's just that if I can't write things down, worrying about remembering one idea gets in the way of having the next. Pen and paper wick ideas.

31 comments

Comments sorted by top scores.

comment by latanius · 2012-09-12T00:13:21.270Z · score: 3 (3 votes) · LW · GW

Anyone using org-mode here? It's free, cross-platform, and also has links (to arbitary files!), outlines (actually, the whole thing is mostly about hierarchical headings), you can use it using mostly the keyboard only, and there are also some Android / iPhone apps (however, I haven't tried them yet).

It has the added benefit that headings have nice colors (especially with a white on black color theme), so if you put it on full screen everyone gets the impression that you're doing something complicated and useful thing. (Even if you're just churning out notes about how to improve your time management as a quite nice & recursive way of procrastination.)

(And yes, it's an emacs mode, actually this is the reason I ended up using emacs for all kinds of other things...)

Additional note: linking to all kinds of files can be an awesome tool when building maps of big and ugly software systems.

comment by moridinamael · 2012-09-12T03:17:46.012Z · score: 4 (4 votes) · LW · GW

I have probably sunk something like 300 hours in org-mode and ultimately abandoned it in favor of a system incorporating Evernote and Nozbe. Org-mode has been a source of much frustration for me. It seems so great, it seems to have all the features one could ever want, but every time I've tried to implement it (three separate attempts, each time starting from scratch and thinking I knew "what I was doing wrong last time,") the system has grown huge and unweildly, leaky and unreliable, and missing key features that I needed.

On the plus side I learned how to use emacs really well.

edited to add: The iPhone app is pretty bad, for the following reasons: It is ugly and navigation is unintuitive, and the text-wrapping is essentially broken. Furthermore, you have to manually synchronize every little thing you do both pushing and pulling to your central repository or you'll quickly end up with inconsistencies which are a disproportionately huge pain to correct.

I actually still use org-mode if I'm simply going to be outlining a complicated project, but I've given up on using it as a task manager. I really wanted to like org-mode.

comment by jwhendy · 2014-10-20T23:00:16.665Z · score: 1 (1 votes) · LW · GW

Late the party, and actually found this thread googling around for "Org-mode file/organization strategies." I've been using Org exclusively for work notes, and am finding myself in a similar situation re. being unwieldy. I constantly struggle with choosing one file per project, one big file with one headline per project, or files dedicated by type (one for todos, one for daily journal logs of experiments/efforts, references, etc.).

Org seems like it should be great for moving stuff around, but I find it not that easy. Refiling a mess of headlines seems to be cumbersome, and how do I know that my new strategy will last/work?

I'd love to know how Evernote solves the unwieldy issue for you. I've tried Evernote, Wunderlist, TiddlyWiki, todo.sh, Zim, and I'm sure others I'm not remembering.

What I'll never give up is the ability to intersperse prose and code. I love, love, love writing all my work reports with embedded R code for analyses in Org-mode, exporting to really nice looking PDF reports. Super awesome, and soooo easy vs. writing all the code elsewhere to generate plots and then inserting them one by one into a ppt. In that respect, Org is awesome. I just haven't figured out an information hierarchy/taxonomy that makes me happy.

comment by moridinamael · 2014-10-21T01:59:18.780Z · score: 0 (0 votes) · LW · GW

It looks like I wrote the grandparent comment over two years ago and I am still primarily using Evernote and Nozbe. Evernote is invaluable for its ability to capture practically any form of information very quickly and then search it later. I can also intersperse "capture" items like reminders with "work" items like drafts of writing.

Nozbe is a fully functional GTD application and it's the backbone of how I manage my tasks.

Theoretically org-mode is great because it combines capture with workspace, but in practice I always found it impossible to smoothly transfer between those two functions.

comment by latanius · 2012-09-23T20:16:12.518Z · score: 1 (1 votes) · LW · GW

I tried the Android app just after I read your comment (it's a thing I've been putting off for a long time), well... it really doesn't include the "creating nested outlines easily" part I like org-mode for, and the synchronization part also seems to be kind of... strange. Just as you said.

What I really like about it is the minimum effort that it needs to, for example, create a todo item (compared with web-based solutions). Too bad that these todo items usually end up really unorganized. Would be indeed nice to have some interface between, e.g. Nozbe and org-mode, and use each of them for the task it is better suited for.

(I also agree with your point about learning emacs really well... or in my case, at a relatively acceptable level :))

comment by [deleted] · 2012-09-23T15:08:42.314Z · score: 1 (1 votes) · LW · GW

As much as I like org-mode (and I like it so much that I don't see myself changing systems unless someone comes along and refines the hell out of org-mode under a new name), I've wished for more from it. Perhaps I ought to just get to work learning more emacs, but some trivial inconveniences and vague desires I've encountered so far:

  • I've not been able to get the Android app working. Ever. Maybe if I go back and put 40 more minutes into getting it to work.
  • I'd love to work with a high-contrast, white-on-black background. I've still not learned how to do this, even with a fair bit of looking
  • If I could switch to a full-screen (not just maximized, but distraction-free, maximum visibility full-screen) that would be fantastic.
  • I sync my org-file (yeah, I only use one, and it's huge) with Dropbox. Don't have to worry about losing it if anything goes dead. But I really just wish that Evernote did all the same stuff. Because Evernote has a slick look, and a pretty great Android app. And I'd be willing to pay money for such a slick implementation of my huge org-file.
  • I use Astrid as a todo-manager. It's not a bad system (even though I've had some obnoxious database corruption issues every month or so), but what I'd really like is a similar system (and there are so many todo managers out there that have the same features, and they're always adding more) that could just read the TODOs out of my org-file and append the appropriate tags, etc. I could see myself someday knowing enough emacs lisp to figure this out on the org-mode end, but right now I don't plan on learning enough programming to work this out as a supplementary app, and it looks far, far simpler to just offer a bounty on this kind of thing that someone else can program.
comment by latanius · 2012-09-23T20:16:18.094Z · score: 0 (0 votes) · LW · GW

Oh, yes. I do Dropbox syncing, too (this is the other good thing about org-mode: plain text files). And there might be some truth in the statement that while org-mode is excellent for a single file, things start to be less seamless when it comes to more of them... inter-file links don't seem to be that reliable, for example. Is this the reason for your One Big Org File?

For white on black, it's just (setq default-frame-alist '((background-color . "black") (foreground-color . "white"))) in your .emacs.

Actually, it's kind of typical lesswrong that I started off with a comment popularizing org-mode, but ended up changing my mind about it (well... kind of), the newest experiments include Notational Velocity (they seem to be good at the global search stuff org-mode is lacking, but not so nice indented lists locally), and also this system:

http://www.speakeasy.org/~lion/nb/book.pdf

which includes paper notebooks, maps of your thoughts and similar fancy stuff, but I haven't yet finished reading it (it's long and not exactly the most organized stuff I've ever read... but it has good ideas.)

comment by jwhendy · 2014-10-20T23:20:58.788Z · score: 0 (0 votes) · LW · GW

For links, I switched to the org-id module and a unique ID for any new links. It works as long as the file containing the target headline is in the same directory as the file containing the link.

(require 'org-id)

(global-set-key "\C-cl" 'org-store-link)

(setq org-link-to-org-use-id 'create-if-interactive)

comment by jwhendy · 2014-10-20T23:09:07.012Z · score: 0 (0 votes) · LW · GW

Yes, and mostly love it. Just not happy with the structure of my information management strategies, at least for daily work documentation. Put "X" under the specific project I'm doing it for? Or what if the learning seems more general... should i start a new tree for longer-term reference knowledge? Or summarize the specific knowledge more generally and keep a copy of both in separate areas? Or write only one and link to it in the other?

Stuff like that.

Other benefits I've really appreciated:

  • embedded/executable code blocks. This is my favorite, favorite feature, and I will never go back (if I can help it) to running analysis code (I do a lot of data analysis/viz with R/ggplot2), generating plots, and then having to insert them into a .doc or .ppt.
  • on that note, being able to export in general is a fantastic feature of Org. HTML, PDF, markdown, whatever. I love not having to futz with image placement manually. Get something you like with some #+attr_html/latex arguments, and then insert a bunch of images really easily by just linking to their location on disk.
  • org is the only, or one of the only, applications that lets you mix and match notes and todos. I used to use TiddlyWiki for all of my work notes and really liked it. But I had to manage todos in something else. But why? Todos often come about in the context of notes, say in a meeting, or for some home project. I really like being able to keep the todos in Org right with whatever prompted the need for action (then view just the actions you need to complete with agenda views).
  • Clocking/time tracking is another aspect that seems awesome, but I haven't started using. I'd love to get to the point where I record what I work on, and especially an estimate + logged hours. Not sure what I'd do with the data, but having it vs. not at least makes post-analysis a possibility.

Still, my various attempts at org file structure seem to end up cluttered and with things structured really oddly. I'll kick off a project with my estimate of what "categories or knowledge" it will require, and as the months or years go on, I'll be in a rush and just resort to keeping a date tree and stuffing the stuff in there like a journal instead. Now I have project-specific info scattered around through monthly journal trees. Harder to archive/find, and end up unfolding a bunch of headlines to find stuff.

Anyway, neat to find other users, so I thought I'd comment even though I'm really late to do so!

comment by hackerkiba · 2012-09-11T18:19:44.850Z · score: 2 (2 votes) · LW · GW

I don't use zim wiki, but I do have page just like that: notes and thoughts

Currently, it contains ideas like: fear incubation, legoization, serendipitous incentive as well essay like "A Healthcare Ancedote in America", "Self-Quantification", and "Why Choose Prosthesis". They have lot of grammar mistakes and other type of writing bugs as well. I need to get better at learning grammar.

A page like that helps me remember lot of ideas and solidify them into essays, hacks, and other products. So far, it only help me learn information and write essay, but that's because I don't have any solid hardware skill yet. Before, I thought I couldn't write essays, but that's because my ideas and citations are not gathered and remembered.

comment by FiftyTwo · 2012-10-03T20:57:34.092Z · score: 1 (1 votes) · LW · GW

Just started using it, haven't done anything beyond the basic note writing so far. Is there a better source of instructions than the main page? Its somewhat confusing.

comment by John_Maxwell_IV · 2012-10-04T06:16:00.775Z · score: 0 (0 votes) · LW · GW

I mainly figured stuff out by poking around in the menus, I think. If you have specific questions, I ought to be able to answer them.

comment by EvelynM · 2012-09-12T23:06:25.126Z · score: 1 (1 votes) · LW · GW

I use Catch (catch.com). It's through the web, and on android. It has an open API, https://catch.com/developer/ so if you wanted to wrap your own SRS system around it, you could.

comment by EvelynM · 2013-08-17T21:41:48.514Z · score: 1 (1 votes) · LW · GW

Catch.com is shutting down 2013-08-31. You can download a zip file of your notes.

comment by [deleted] · 2012-09-11T19:30:24.404Z · score: 1 (1 votes) · LW · GW

That sounds pretty great. I happened to get a free copy of OneNote 2010 a little over two years ago, and it's played a similar role for me. It has everything you mention above, including the autocomplete search/jump which I agree is killer. I think linking and timestamp don't come with keyboard shortcuts, but you can add shortcuts for any command [edit: see pjeby's reply -- thanks!]. A couple other features I appreciate: constant autosave, and a keyboard shortcut for taking screen clippings when OneNote's running in the background. There's even good math support, although I still use LaTeX for math/physics notes.

The downsides: It's not free, open-source, or cross-platform. There are Android and iOS apps, although I'd need Android 2.3+, so I can't yet comment on that. My least favorite aspect is that it doesn't store things in plaintext, although you can get plaintext out with a little bit of work. (Probably easiest to save a notebook as html and go from there.) Maybe part of the reason is that there's lots you can do that isn't text (more flexible formatting than e.g. Word, recording and inserting video/audio notes, drawing, tags, integration with Outlook tasks) but much of that could at least be formatted sanely, and I don't use it anyway. Zim Wiki sounds like a good choice if I ever get frustrated with the lock-in.

comment by pjeby · 2012-09-12T01:07:10.084Z · score: 5 (5 votes) · LW · GW

I think linking and timestamp don't come with keyboard shortcuts, but you can add shortcuts for any command.

Keyboard shortcuts for OneNote:

  • Alt-Shift-T inserts time
  • Alt-Shift-D inserts date
  • Alt-Shift-F inserts date+time
  • Ctrl-K opens a link-creation dialog (or converts existing text into a link)
  • Surrounding text in [[double brackets]] makes a wiki-style link, creating a new page with the enclosed title (or linking to an existing one) in the same section.

Also, any page or paragraph can be a link target, with its own onenote:// URL that can be launched from other programs that allow links to be embedded.

Yeah, if you happen to be on Windows and have OneNote 2010, it can be pretty sweet for this sort of thing.

comment by Emile · 2012-09-11T12:18:59.884Z · score: 1 (1 votes) · LW · GW

I used to run a wiki (instiki) on my local machine to do something like this, but I eventually gave up reinstalling it on new machines after running into problems with data compatibility (unlike Zip, it isn't stored in plain text files).

Now I just use a plain text file for my todo list, and either google docs or drafts in gmail for my various personal notes; I'm not extremely satisfied with it either - Google docs is convenient for access from anywhere, and it's nice to be able to use spreadsheets, but I miss the ability to easily create many small interlinked pages.

Do you use Zim on several machines, synching the text files from one to another with source control?

comment by John_Maxwell_IV · 2012-09-11T21:04:14.837Z · score: 0 (0 votes) · LW · GW

No, I just use it on one machine.

comment by Omid · 2014-06-27T12:23:13.262Z · score: 0 (0 votes) · LW · GW

Are you still using Zim Wiki?

comment by John_Maxwell_IV · 2014-06-28T05:41:08.882Z · score: 1 (1 votes) · LW · GW

I switched my OS to Mac OS X and started using http://brettterpstra.com/projects/nvalt/ However, there are features I miss from Zim Wiki. I would definitely recommend it on Linux.

(BTW if anyone is ever making the same Zim -> nvALT migration get in touch with me... I wrote some scripts to make it easier.)

comment by Bruno_Coelho · 2012-09-12T21:51:48.234Z · score: 0 (0 votes) · LW · GW

What percentage of the written text do you re-read, or update regularly?

comment by John_Maxwell_IV · 2012-09-12T23:19:29.602Z · score: 2 (1 votes) · LW · GW

I'm guessing that roughly 15% of my notebook is relatively "high-traffic" stuff that I actually review/rewrite on a semiregular basis (once every few weeks at least).

comment by [deleted] · 2012-09-11T13:52:20.538Z · score: 0 (4 votes) · LW · GW

"Brain support"? Sensationalist title.

comment by RomeoStevens · 2012-09-11T20:22:35.498Z · score: 6 (6 votes) · LW · GW

improving the efficiency of the exobrain tools we use seems to be vastly underrated.

comment by Emile · 2012-09-12T09:22:20.462Z · score: 4 (4 votes) · LW · GW

I'm not so sure of that - there's also a risk of spending too much time and attention on tools rather than on actually getting stuff done - what's called "Productivity Porn".

A bit like if an online community spent all it's energy on talking about how to have quality discussions, instead of actually having quality discussions.

comment by RomeoStevens · 2012-09-12T09:48:46.493Z · score: 2 (2 votes) · LW · GW

I agree but it seems like a niche problem. How many people have a good note taking system and understand how to use search engines to full effectiveness?

comment by [deleted] · 2012-09-11T21:05:50.872Z · score: -1 (1 votes) · LW · GW

Zim doesn't represent any improvement in the efficiency of taking computer notes. Evernote beats it on most features.

comment by John_Maxwell_IV · 2012-09-11T21:02:52.679Z · score: 2 (2 votes) · LW · GW

Is "thought support" OK?

comment by [deleted] · 2012-09-11T21:09:56.760Z · score: -2 (4 votes) · LW · GW

You still don't give any arguments to support such a claim (to wit, "Zim Wiki provides thought support"), so no.

I suggest "Zim Wiki made me feel more productive" or "Zim Wiki is pretty cool, give it a try"

comment by Desrtopa · 2012-09-11T14:08:20.861Z · score: 0 (2 votes) · LW · GW

I default to interpreting it as "keeping a brain alive with machinery."

comment by maia · 2012-09-11T13:27:52.640Z · score: 0 (0 votes) · LW · GW

I might use something like this, except I find it extremely useful to be able to note things down on my smartphone wherever I am when I have a sudden thought, and have that synced with my other devices.

So I just use Evernote. Wish it had source-control-ish features, though.