ESC Process Notes: Claim Evaluation vs. Syntheses
post by Elizabeth (pktechgirl) · 2019-12-24T17:00:02.010Z · LW · GW · 4 commentsForgive me if some of this is repetitive, I can’t remember what I’ve written in which draft and what’s actually been published, much less tell what’s actually novel. Eventually there will be a polished master post describing my overall note taking method and leaving out most of how it was developed, but it also feels useful to discuss the journey.
When I started taking notes in Roam (a workflowy/wiki hybrid), I would:
- Create a page for the book (called a Source page), with some information like author and subject (example)
- Record every claim the book made on that Source page
- Tag each claim so it got its own page
- When I investigated a claim, gather evidence from various sources and list it on the claim page, grouped by source
This didn’t make sense though: why did some sources get their own page and some a bullet point on a claims page? Why did some claims get their own page and some not? What happened if a piece of evidence was useful in multiple claims?
Around this time I coincidentally had a call with Roam CEO Conor White-Sullivan to demo a bug I thought I had found. There was no bug, I had misremembered the intended behavior, but this meant that he saw my system and couldn’t hide his flinch. Aside from wrecking performance, there was no need to give each claim its own page: Roam has block references, so you can point to bullet points, not just pages.
When Conor said this, something clicked. I had already identified one of the problems with epistemic spot checks as being too binary, too focused on evaluating a particular claim or book than building knowledge. The original way of note taking was a continuation of that. What I should be doing was gathering multiple sources, taking notes on equal footing, and then combining them into an actual belief using references to the claims’ bullet points. I call that a Synthesis (example). Once I had an actual belief, I could assess the focal claim in context and give it a credence (a slider from 1-10), which could be used to inform my overall assessment of the book.
Sometimes there isn’t enough information to create a Synthesis, so something is left as a Question instead (example).
Once I’d proceduralized this a bit, it felt so natural and informative I assumed everyone else would find it similarly so. Finally you didn’t have to take my word for what was important- you could see all the evidence I’d gathered and then click through to see the context on anything you thought deserved a closer look. Surely everyone will be overjoyed that I am providing this
Feedback was overwhelming that this was much worse, no one wanted to read my Roam DB, and I should keep presenting evidence linearly.
I refuse to accept that my old way is the best way of presenting evidence and conclusions about a book or a claim. It’s too linear and contextless. I do accept that “here’s my Roam have fun” is worse. Part of my current project is to identify a third way that shares the information I want to in a way that is actually readable.
4 comments
Comments sorted by top scores.
comment by Pattern · 2019-12-25T02:59:53.877Z · LW(p) · GW(p)
Feedback was overwhelming that this was much worse, no one wanted to read my Roam DB, and I should keep presenting evidence linearly.
I'm curious what it looks like.
The question linked seemed like a reasonable format. (It helps to know things can be minimized like in reddit.) The claims just seemed to be hanging, and [markdown looking [stuff]] indicates links.
The example page for a book linked was more linear and understandable - though I don't know what the sliders mean, so maybe it's just that they add space.
A while back you reviewed a book which didn't have a thesis - maybe people don't like the disconnected information because there isn't a clear thesis. On the other hand, people use twitter all the time. Is it just the aesthetics/formatting? Or because it's not interactive... If it's lack of familiarity, people who use Roam, might have different opinions. (Another format which presents information as separate is a deck of flashcards.)
Replies from: pktechgirl, pktechgirl↑ comment by Elizabeth (pktechgirl) · 2019-12-26T22:27:08.152Z · LW(p) · GW(p)
Just realized the "it" in "I'm curious what it looks like." probably referred to "my DB", not "the feedback". I'd love to either user test my DB on you (you play with it while I watch) or have you beta test the description I'm writing, if you're interested.
Replies from: Pattern↑ comment by Pattern · 2019-12-31T04:14:16.615Z · LW(p) · GW(p)
either [1] user test my DB on you (you play with it while I watch) or have you [2] beta test the description
Either sounds interesting, though setting up [1] requires time coordination (or specific tools). If you're interested in that, send me a private message.
Some thoughts beforehand (since this could be a general problem, and I'm interested in how my perspective might change as a result of such a process***):
Possible issues with 'communicating in a different medium'*, and how they might be fixed are:
0) They don't understand the medium.
(This seems ruled out by other people who have used Roam not understanding other people's Roam documents (although how widespread this is, isn't clear). Perhaps Roam's tech support (if they have it) will have or eventually obtain the most experience here - helping people with their documents, understanding more of the tools, and possibly how Roam is programmed. The metaphor of "different languages" doesn't seem apt here - it seems to be Roam users who share a language, but use it differently that aren't understanding.)
1) People can't see their way around it - it's like an extension of someone else's mind, and it doesn't connect to other people. Understanding might be built by watching it grow from scratch or be worked with by someone who understands it might help.** (The alternative is multiple people working together on a document, at the time, and then them trying to explain it to other people afterward, or each other if changes are made independently later on.)
2) Understanding it requires asking questions. (Is this a one time cost or continuous one?)
3) Motivation - people enjoy the way words (and thoughts) are "normally" arranged. People are okay with nested comments on sites like Reddit because the structure is that of a conversation, and that's something people engage with.
*or "communicating a different medium"
**This could be done via youtube videos or livestreaming. (With or without audio.) Seeing a document be built, and how, might give insight.
***Though this might be uninteresting to other people.
↑ comment by Elizabeth (pktechgirl) · 2019-12-26T20:31:14.611Z · LW(p) · GW(p)
See here [LW(p) · GW(p)] and here for responses. One of those was in response to a book that did better on the "having a thesis" axis than "having evidence", so I don't think that's the problem.
It seems plausible having a guide will help people, and that's on my list, but I'm aiming for a high level of polish so it's unfinished.