Call For Distillers

What posts of yours do you want distilled?

I want to really signal boost an idea that's come up [? · GW] a bit recently: doing user research on pieces of writing. When writing such distillations, have someone in the target audience read it and sit down next to them (or on a video call) and watch them do so. Try to get them to think out loud. Observe what works and what doesn't. I suspect rather strongly that this is a low hanging fruit.

Looking for "distillers" / happy to pay for this work.

Also distillation seems like a wrong name.  What's often needed seems to be more like dilution & blending  - I can often describe core of an idea by a few sentences, but the inferential steps required from the reader are then too large, or rely on knowledge unknown to many readers.

Bit of a pointless gripe (and not too specific to this post), but I wish we could use the word “pedagogy” instead of “distillation”. Not only is “pedagogy” much more understandable to lots of potential readers, but I don't think “distill” is even capturing a helpful mental image. “Distillation” in chemistry is getting rid of all the excess to make something super-concentrated. But that's often the opposite of good pedagogy! Imagine (1) a dense math proof with almost no English words, (2) an explanation of the same proof with lots of examples and diagrams and stories and intuitions, and explaining the same thing multiple times from multiple perspectives, etc. When I imagine “distillation” in the original (chemistry) sense, to me it invokes a mental image much closer to (1) not (2). But (2) is better pedagogy, and (2) is what we actually want in this context.

(For example, Rob Miles videos are not designed to pack in the maximum possible number of concentrated insights per second of video.)

I think I weakly disagree with the implication that “distillation” should be thought of as a different category of activity from “original research”. It is in a superficial sense, but a lot of the underlying activities and skills and motivations overlap. For example, original researchers also have the experience of reading something, feeling confused about it, and then eventually feeling less confused about it. They just might not choose to spend the time writing up how they came to be less confused. Conversely, someone trying to understand something for the purpose of pedagogy may notice a mistake in the original, or that the original is outright wrong, which is original research.

I guess if I were writing something-like-this-post, I would frame it as:

1. I encourage grant-makers to be impressed by people for creating good pedagogy even if it's technically unoriginal. (I suspect that this is already the case.)
2. I encourage anyone who has the experience of reading something, feeling confused about it, and then eventually feeling less confused about it, to create some piece of pedagogy that would have helped their former selves; for example, this is an excellent type of project for people trying to get into the field.
3. I encourage active researchers doing original research to also consider whether pausing to create better pedagogy would be a good use of time, even at the expense of slowing down their own novel research progress.
4. I encourage anyone who feels very confused about something-in-particular to post calls / bounties / whatever for pedagogy on that topic.

(Maybe other things too.)

For my part I've spent much of the last five months on a #3 project [? · GW], and I think that was the right call for my particular situation—I suspect that I learned more through writing those things than anyone else will by reading them. I also spent the better part of a month on a #2 project [AF · GW], and also found it a good use of time. And the very first thing I did when I decided to learn about the field was spend a few months creating unoriginal pedagogy [LW · GW]. It was a great way to learn. :)

Funny timing seeing Scott Alexander's Yudkowsky Contra Christiano On AI Takeoff Speeds post so soon after this post. My god is he good. I wonder what the implications of that are.

Coming up with good examples strikes me as the sort of thing that is a better fit for crowdsourcing [LW · GW] than for an individual to try to think up themself.

In practice, I understand there is a bit of a catch-22 though: a published post is the time when such crowdsourcing is most practical to occur, but you'd probably want have the good examples be included in that published post. I'm not sure what to do about this.

One idea is to post something like "I'm working on a post for X and am looking for good examples of Y." But that's hard because it very well might be hard to explain X without having really written up X (depending on the X). A second idea that I like more is publishing rough drafts, or even more early stage writeups, and getting help with examples there. That's an idea I am a fan of anyway though, forgetting about the benefit of it helping with the generation of examples.

Since this post was written, I feel like there's been a zeitgeist of "Distillation Projects." I don't know how causal this post was, I think in some sense the ecosystem was ripe for a Distillation Wave) But it seemed useful to think about how that wave played out.

Some of the results have been great. But many of the results have felt kinda meh to me, and I now have a bit of a flinch/ugh reaction when I see a post with "distillation" in it's title. 

Basically, good distillations are a highly skilled effort. It's sort of natural to write a distillation of something as part of your attempt to understand it, and upskill (I think I had previously advocated this sometimes). I think this was basically a reasonable thing to do, but, it did have the cumulative effect of decreasing the signal-noise ratio of distillations, since most people doing this aren't skilled yet.

None of that contradicts the claims of this post, which specify various skills you need, and recommends actually investing in those skills. (The title of this post is "call for Distillers", not "call for Distillations". I think a lot of what happened was things like "Distillation contests", and incorporating distillation into SERI MATS programming, etc, which doesn't automatically produce dedicated distillers)

Curated. I think this is a message that's well worth getting out there, and a write-up of a message I find myself telling people often. As more people are interested in joining the Alignment field, I think we should establish this is a way that people can start contributing. A suggestion here is that people can further flesh out LessWrong wiki-tag pages on AI (see the concepts page), and I'd be interested in building further framework on LessWrong to enable distillation work.

This sounds like something that could be done as an organization creating a job for it, which could help with mentorship/connections/motivation/job security relative to expecting people to apply to EAIF/LTFF

My organization (Rethink Priorities) is currently hiring for research assistants and research fellows (among other roles) and some of their responsibilities will include distillation.

I'd be excited to see more of this happening.

It reminds me of the recent job posting [AF · GW] from Abram, Vanessa and Diffractor, which seems to be a role of adjunct distiller for Infrabaysianism, though they use different terms.

A quadratic funding mechanism (similar to Gitcoin) could make sense for putting up distillation bounties. Quadratic funding (QF) lets a grant-maker put up a pool of matching funds while individual researchers specify how much each individual bounty would be valuable to her/him; then the matching is done via the QF strategy to optimize for aggregate researcher utility. Speaking for myself, I would contribute to a community fund for further distillations, and I would also be more likely to distill.

I find the level of distillation done by Daniel Filan at AXRP to be great. Short and listenable enough for easy access while detailed enough to let you form opinions/directions for further research.

Some IMO valuable targets for distillation: Infra-Bayesianism [LW · GW] and recent work on why deep nets converge e.g. 'Gradient Descent Finds Global Minima of Deep Neural Networks', Learning and Generalization in Overparameterized Neural Networks, Going Beyond Two Layers, A Theoretical Analysis of Deep Q-Learning

I consider myself a fairly good distiller, though fairly bad at math (compared to the average person here). So far I’ve mostly used that skill to write biographies on Wikipedia, but it might be possible for me to work on more math-heavy topics if I either have enough time to learn the math in depth or have a strongly collaborative relationship with someone who already does.

Now that we have GPT-4, is distillation any easier?

Four months in. Is there a list of distillation that one could look at?

I currently receive funding from LTF Fund for translating AI-Safety-related texts into Russian.

My partner (she's also my editor) is interested in something very similar to what this post says. The last year she often says something like "Your rationalist guys are definitely very smart, but they are very bad at speaking human language, so no one understands them". 

We actually thought, that after finishing my grant work (so that she edits everything that I translated, and becomes well versed in the topic) in July, she will send her own grant application with plan "she writes easy-to-understand texts about AI-Safety, hired translator translates them into English (I'm good at English-Russian translations, but not vice-versa), I check that translator doesn't distort the meaning."

She has a lot of experience with writing short texts: she runs a group in social media with ~20000 subscribers 7 years.

Should we really do this, what do you think? 

I'm pretty confused that this is as necessary as it is, particularly with writing that involves a lot of math and math notation. I don't understand how people get the insight and motivation necessary to write that kind of thing without explaining what the point of it is or giving examples of how to apply it as part of their expositions.

Your (johnswentworth's) posts don't seem to suffer nearly as much from this, at least from a quick skim of the ones you said on another thread you'd like distilled, but e.g. Infra-Bayesianism [? · GW] seems maybe important (it's about something I was thinking about anyway), but I've "bounced off" the sequence, as apparently have enough others for there to be a job offering [LW · GW] distilling it. The same thing happens sometimes when I'm reading academic papers.

I have nothing against rigor and I actually enjoy math, but when writing about models or systems one wants to apply to something that isn't purely mathematical, these are the basic parts I generally see and how I react to them:

• Motivation. This is great! Tell me what problem you're trying to solve and sketch your solution. People tend to do this only at the beginning of a piece, if they do it at all, which in my opinion is a mistake. I want to know why you care about proving that theorem, and if you don't say I can't always guess.

• Examples. These are great! If you introduce a construct, show me one. Make it just non-trivial enough that there's a point to using the abstraction at all. Ideally make its components things I can already recognize and understand without the abstraction. Again, there often are some but not enough; any significant point made that isn't illustrated with an example forces me to come up with one on my own or else I just won't understand it.

• Text description of math. This varies a lot. If it's showing the correspondence between the real problem being solved and the abstractions and notation used, that's fine. Other statements relating mathematical objects to one another are okay if I already have a good grasp of what those objects are (less so if some or all of them were only introduced in the current piece and are non-obvious in terms of the underlying constructs). Proofs written in text tend to frustrate me. They require extreme amounts of thought per sentence because they're written very elliptically (stating "X has property Z" while assuming I remember "X has property Y" and "property Y implies property Z", often when neither of these latter things is stated in the piece at all), and when I do figure out what they're saying, 90% of it is extremely obvious and uninteresting, but the 10% that makes the proof work isn't highlighted in any way. I largely end up skipping them, and since it seems like they're probably one of the hardest parts to write, that feels like a big waste.

• Symbolic math. There's no other way to be so precise so concisely, but I find myself having to refer back to definitions constantly to figure out what they're saying, and long blocks of nothing but symbols are even more impenetrable than prose proofs; I end up skipping such blocks if there's any way I can figure out what's being said without decoding them. In programming we all know single letter variable and function names make code hard to understand (with a few common exceptions like naming a loop index "i"); I think symbolic math would be much easier to read if it looked more like a computer program.

Now, I do also see writing that's almost all motivation and examples, and that doesn't seem right either. It doesn't create sufficient technical understanding that I could implement the ideas in a computer program, or adapt them to very different cases than the ones presented. But to summarize, some skills you say are important in distilling are:

• Ability to independently generate intuitive examples when reading mathematical arguments, or having a mathematical discussion
• Ability to extract the core intuitive story from a mathematical argument

And, well, I honestly don't understand how anyone can even think about math, and particularly not how they can come up with useful math, without having examples and an intuitive story in mind already.

Are there examples or best practices you would recommend for this?

We're hoping to alleviate parts of this issue with the use of language models: https://www.lesswrong.com/posts/ebYiodG3MAEqskCDG/a-survey-of-tool-use-and-workflows-in-alignment-research-1 [LW · GW]

Should the role of a distiller include spotting mistakes? I assume that you'd only want distillers to get to work once you have some confidence that the original claims are correct. 

What you're looking for here is someone in a field called technical writing and/or translation analysis. Technical writers and translation analysts can have different specialties. Someone who is mathematics specialized could do the work you describe.

This is a growing and well-paying field for people with excellent writing skills to work in the tech, engineering, medical, etc fields without having much technical training/degrees.

This is extremely articulate and frankly universally applicable. Distillers are needed at any critical point in a business when you’re trying to understand what you have and what to do with it aka decision intelligence. Anyone able to effectively and efficiently “distill” the critical, actionable nuggets which drive outcomes are the most valuable resource in a company!

What is the intended difference between a distiller and the field of science communication? 

I want to point out what could be a serious problem for anybody attempting to do "distillation" in a public setting. Although here, "distillation" is specifically couched as a way of explicating mathematics, I believe the concept generalizes to any repackaging dry, terse and abstract set of ideas into more intuitive language.

Let me start by giving a specific example of an unpublished piece of writing I produced as a form of distillation. The Handbook of the Biology of Aging is a great intellectual resource on the subject of geroscience, but it's written in terse, abstract academese. I rewrote chapter 4 in much livelier language, with expanded examples and  a slightly reworked structure, and credited the original author with both the ideas and the structure, being clear that I'm not claiming any intellectual novelty in my new version. It was always intended for my blog, not for publication in any peer-reviewed journal. I'm pretty confident that most lay audiences would prefer to absorb the original author's ideas via my version than via the original.

The problem is plagiarism. Plagiarism isn't just about copying words - it's also about copying ideas. Although a careful distiller could avoid risk of violating formal policies/laws concerning plagiarism by carefully citing their sources and being clear that their work is not attempting to provide any form of intellectual novelty, it also poses a potential reputational risk to the distiller.

Here, distillation is highlighted in part as a way for students to build into a career as a researcher. However, even if it's unfair, distillation can look like intellectual laziness - the academic version of an artist copying someone else's work and displaying it in a gallery. Even if the artist cites the original source, perhaps by labeling the image with the tag "a copy of Van Gogh's Starry Night" displaying their copy in public is likely to undermine their reputation for being capable of original artistry and build their reputation as a "mere copier." They might be perceived not only as artistically weak, but as a seedy sort of person who may well be on the road to selling art forgeries. A distiller faces the same reputational risk.

Think of it like the difference between an action that violates the law, and an action that could result in being sued. If a lawyer wants to sue you, then even if you ultimately win the case, you might be tied up in court for years, and suffer massive legal expenses. Smart people don't skirt the edge of being potentially sued, at least not as part of their normal business operations. They steer well clear of this whenever possible. I think that distillation is skirting so close to potential perceptions of plagiarism and intellectual laziness that it creates an analogous risk.

I think this is deeply unfortunate, because distillation has all the benefits described in the OP. A good distillation can make important ideas more accessible, and that might in fact be the bottleneck for creating new intellectual contributions based on those ideas. But unfortunately, academia doesn't really have a culture of considering distillation as a valuable form of scientific outreach. It will tend to see distillation as somewhere between plagiarism and intellectual laziness. Even if a persistent argument with one particular person who might accuse the distiller of these failings manages to convince them to see the value in the work of distillation, there will be another person right behind them to lobby the same accusation. And then the distiller looks like the sort of person who doesn't have the judgment to know what's going to rile up academics and create a perception of plagiarism - and who wants to work with somebody like that?

There's a difference between a literature review or piece of journalism, which weaves together properly cited ideas from a variety of sources into a fundamentally new structure in a way that everybody can understand is not plagiaristic, and a distillation which, as I understand it, takes the intellectual architecture of a single source and dresses it up in new language. The latter looks a lot like plagiarism to many people.

The difficulty with producing distillation that doesn't create laziness/plagiarism perceptions is unfortunate. But I think it's akin to the unfortunate effect of patent law in slowing innovation. Right now, we prioritize the need of intellectuals to protect their intellectual contributions over the need for writers to supply audiences with more accessible versions of those ideas.

So if I was going to leave potential distillers with a takeaway message, it would be this:

Be EXTREMELY CAREFUL in how you write distillations. If you must write them, consider not publishing them. It's not enough to properly cite your sources. Every time you publish a distillation, you are taking a reputational risk, and in many situations, there isn't any real personal reward to counterbalance it. Even if you have not plagiarized, and even if you are capable of original thought, you might create a perception that you are an intellectually lazy plagiarizer hiding these failings under the term "distillation." This might permanently damage your career prospects in academia. Unless you're very confident that your specific approach to distillation will avoid that reputational risk to yourself, strongly consider keeping your distillation private.