Mosaic and Palimpsests: Two Shapes of Research

post by adamShimi · 2022-07-12T09:05:28.984Z · LW · GW · 3 comments

Contents

  Introduction
  Issues with Conceptual and Experimental Innovators
  Defining Mosaics and Palimpsests
  Examples from Conceptual Alignment
  My Mistake: Aiming for Mosaics Instead of Palimpsests
  A Harmful Bias for Mosaics
  Open Problems
None
3 comments

(Minor update to change Steve's labelling following this comment [AF(p) · GW(p)], and also because I realized that I never added the footnotes...)

This post is part of the work done at Conjecture.

Introduction

In Old Masters and Young Geniuses, economist-turned-art-data-analyst David Galenson investigates a striking regularity in the careers of painters: art history and markets favors either their early pieces or the complete opposite — their last ones. From this pattern and additional data, Galenson extracts and defends a separation of creatives into two categories, two extremes of a spectrum: conceptual innovators (remembered as young geniuses) and experimental innovators (remembered as old masters).

Conceptual innovators, like painter Pablo Picasso, start with a crystal clear idea of their goal, and spend tremendous amounts of time in preliminary research, preparatory drawing and all-around planning. They then mostly stick to these extensive plans (or have them executed by others) when concretely creating the output. As such, their most impressive innovations generally come from their ground breaking ideas, when they know and have done so little that they can simplify and break all the rules they haven't yet learned — they're young geniuses. Galenson provides additional examples: artist Andy Warhol, novelist Herman Melville, movie director Orson Welles, scientist Albert Einstein...

On the other hand, experimental innovators, like painter Paul Cézanne, only figure out their aim by relentless trial-and-error, making something up and then iterating on it. Their intuitions start vague and their goals cloudy, leading to their perpetual uncertainty and doubt about having accomplished what they wanted. Yet because experimental innovators keep on refining their attempts, and because they build on all that happened before, their best output (measured by metrics like auction prices, bestselling list, mentions in textbooks) emerges towards the end of their lives — they're old masters. Some additional examples given by Galensen are: abstract painter Jackson Pollock, novelist Virginia Woolf, movie director Alfred Hitchcock, and scientist Charles Darwin.

Galenson's distinction jolted my mind and turned it racing with questions: in which box did I fall? And others in alignment? What were the consequences for accelerating science in general and alignment in particular? What are the limits of the distinction?

Unfortunately, the painter-based distinction proved unwieldy to analyze conceptual research, my main application. Yet there was definitely something there...

This post thus follows my process in carving a frame out of the old master/young genius template which can be applied to heady topics like alignment and epistemology, and illuminates both my own mistakes with regard to my work, and the different styles of alignment researchers. My frame applies to research outputs rather than to researchers, and separate them into mosaics (works which fit within a clear, simple, trimmed down structure, like colored tiles in a mosaic), and palimpsests (works which iterate on previous idea by shifting and altering them, rewriting on top like a palimpsest)

Epistemic status: Like most of my work, this post fits into a palimpsest rather than a mosaic (see definitions below). As such, it is but one step of an iterative process. I currently agree with it; I will find it limited or inadequate soon, and will probably build something new on top (or with its ashes).

Issues with Conceptual and Experimental Innovators

Going back to Galenson's conceptual and experimental innovators for a moment — what was the problem? For my purposes, I had to confront three issues.

• (Too anchored in art) Because Galenson discovered his pattern in painting, it embeds part of the art analysis ontology. The main problematic example is the difference between the heady, idea-oriented conceptual innovator and the visual, sensation-oriented experimental innovator. What does that even mean for a mathematician, an epistemologist, or a conceptual alignment researcher?
• (Too focused on the creative process) Counting planning material provides Galenson with his main metric for distinguishing conceptual and experimental innovators. This includes preparatory drawing in paintings, outlines in novels, anything that counts as planning before "making the thing". Yet this metric loses its potency when turning to scholarship and research; when we consider research results rather than papers or books, we lose visibility on the process. In addition, almost everyone has "preparatory work" of one kind or another in research, despite Galenson's distinction tracking an intuitive pattern.
• (Wrong focus on people) Last, Galenson can't resist putting people in boxes, or at least on a continuum. Yet classifications of this kind rarely generalize due to people's quirks and complexity. For example, it's unclear where I fit in the conceptual-experimental spectrum, as I am quite abstract and conceptual yet iterate all the time and cannot get out of the swamp of my doubt.

This lead me to the following corrections to Galenson's distinction:

• Focus on the core of confident structure vs hesitant iteration
• Focus on the output instead of on the creator

Taking the conceptual vs experimental innovators frame through these two transformations gives us "mosaics vs palimpsests".

Defining Mosaics and Palimpsests

A mosaic (from the ancient art form) is a set of research works that all fit within a simple, explicit, preordained structure, like tiles in a mosaic. Whether complete or in progress, mosaics exude coherence and reshape the object of investigation into a form that makes perfect sense. Thus new research within a mosaic builds on top of the previous works, rather than over.

This underlying structure might be quite local (just assumptions and framing about a subproblem) or amount to a whole paradigm.

A palimpsest (from the medieval manuscripts which monks wrote over) is a set of research works which follows a process of iteration without clear initial vision. The sequence of successive ideas reveals the refinement, correction, and wholesale changes of the initial attempt, in order to capture the elusive intuitions that prompted the work.

Then individuals might lean (with different levels of strength) towards one shape of research over another. Which leads to an analogous of the distinction between conceptual innovators/young geniuses vs experimental innovators/old masters. Yet focusing on the shape of research rather than the people lets us accommodate those thinkers who regularly alternate, or consider the possibility of dependencies between the two (maybe mosaics require initial palimpsests).

Now, to refine the broad strokes of these two shapes, let's look at examples from conceptual alignment research.

Examples from Conceptual Alignment

The Sequences (and really everything written by Eliezer) provide the most obvious instance of mosaic in conceptual alignment. All the posts build on each other, often in subtle and implicit ways, and the worldview shared there stays coherent through and through. It even fits perfectly with both more modern works of Eliezer (like Security Mindset and Ordinary Paranoia) and older ones (like the AI Foom Debate).

On the opposite corner, most of Paul's work basks in a definite palimpsest shape. His wealth of blog posts keeps coming back to the same topics (indirect normativity, amplification, universality...), each time explored from a slightly different angle, or correcting an inadequacy in the previous formulation. Even Paul's research methodology [AF · GW] draws the shape of a palimpsest!

(As a side note, this feels like a core reason for the notorious difficulty of Paul's writing (another being the theoretical computer science framing which is not that common here): most of his posts and papers don’t give a self-contained, up-to-date and coherent perspective on Paul's approach. Instead, he has many fascinating iterations that get into seemingly nitpicky details (which matter a lot for the iteration) and require to have internalized most of the previous steps of iteration. Yet when you have, he writes in a clear and direct fashion.)

Back to example-listing. On the mosaic-side, here is a non-exhaustive list^[1]:

• John's work on the natural abstraction hypothesis [AF · GW] and his more recent abstraction work [? · GW]. As well as his epistemology research [? · GW].
• Scott's Cartesian Frames [? · GW] and Finite Factored Sets [? · GW]
• Alex's The ground of optimization [AF · GW]

And here is the palimpsest-shaped one^[2]:

• Evan's work [AF · GW]^[3]
• Stuart's work [AF · GW]
• Steve's work on Brain-like AGI [? · GW]^[4]

In the middle ground, I place for example Vanessa's work, notably InfraBayes [? · GW]. On the one hand, most results (for example InfraBayes Physicalism [AF · GW]) fit cleanly within a structure related to InfraBayes; on the other hand, my model of Vanessa and Diffractor's process includes many unplanned results, with potential alterations of the focus and aims of the research.

Or to say it differently, InfraBayes feels like iterating on the consequences and uses of some mathematical formalization with a vague aim, thus in between mosaics and palimpsests.

My Mistake: Aiming for Mosaics Instead of Palimpsests

Whenever I get some interesting idea like mosaics and palimpsests, I automatically search how to apply it to myself. Although Galenson's original distinction confused me when I tried to apply it to my own creative process (as already mentioned), mosaics and palimpsests let me name one of my main failure modes: I keep wanting to make mosaics, when my intuitions and my personality clearly predispose me to palimpsests.

I can now see and name the urge, the pull towards writing something clean, nice, powerful, final. I can see my admiration (often mixed with jealousy) for the ground of optimization, for John's posts and sequences, for Scott's beautiful and crisp formalisms — for mosaics in general. And how it colored my own constant frustration with my own work, which never was as structured, which always rebelled against the inadequate structure, which never wanted to stop shifting just for once.

And the irony is that the best things I produced, the ideas that people around me leverage the most — epistemic strategies [? · GW], productive mistakes [LW · GW], epistemological vigilance for alignment [LW · GW], unbounded atomic optimization^[5] — all clearly fit into patterns of palimpsests. Some of them are even different steps of the same palimpsest!

Palimpsests just fit my way of thinking: I cannot stop investigating and doubting the assumptions and the simplifications that I make; and my work doesn't cleanly build on itself, but rewrites itself and branches off in sprawling fashion.

Thus aiming for mosaics created the condition of my own unproductivity and misery. Instead of iterating faster and sharing the steps of my palimpsests, I ended up stressing myself out to complete massive projects of superb structure, before realizing that none of the pieces fitted together. And praising structure above all else, I was unable to acknowledge the productivity of these bad fits — the interesting part was the revealed inadequacy.

Somehow I managed to be the advocate of productive mistakes and avoid them in my own work. Do as I say not as I do, right?

A Harmful Bias for Mosaics

One of Galenson's avowed goals is to highlight a cultural trend (in painting, art, entrepreneurship, and science) that favors the dashing masterpiece which springs fully formed (conceptual innovation/mosaic) over the less legible and longer matured result of an iterative process (experimental innovation/palimpsest).

And he has a point. In myself and in others around, I now see a clear bias towards mosaic.^[6] Which makes perfect sense — mosaics look good from the start, as they explicitly hand you a compressed structure with which to see the world. Whereas palimpsests merely gesture at vague intuitions, half-baked arguments, and weird framings.

Or without so many words, mosaics just look cooler.

Out of all the negative effects that come from this bias, the worst are probably:

• Anxiety, stress, feelings of inadequacy for palimpsests-oriented thinkers who feel they must make mosaics. And confirmation effect when they write palimpsests anyway and have little feedback or comments.
• Misguided feedback, when others comment on palimpsests as if they were mosaics, and tear them down for not proposing a complete structure or a clear narrative.^[7]
• Stifling of idea generation, as almost all new approaches and random connections lack the nice and robust structure of a mosaic.
• Loss of much great discussion, iterations, and debates because people are disincentivised to publish anything palimpsests-like.

Palimpsest-oriented researchers are not the only ones to suffer from this bias, though: even someone who focuses on mosaics deals with the effect of the idolization of mosaics. Namely, mosaics often don't get productive pushback and criticism.^[8]

Indeed, reactions to mosaics tend to fit into one of three categories:

• you invest the time to get it, and then it starts shaping your thoughts in ways that make it invisible and obviously correct;
• you don't invest the time but see the general shape, in which case you don't have the understanding necessary to give detailed feedback;
• you don't invest the time and find one or many of the underlying assumptions completely ridiculous and aversive, leading you to deny any value to the mosaic.
  All of these are further compounded by the usual confidence (sometimes crossing into arrogance) with which mosaic-oriented researchers present their works.

In short, mosaics polarize, which is great for attention and status-building, but makes it harder to see them as productive mistakes and thus to separate the brilliant insights from the misguided assumptions.

But maybe the bias gets something right — maybe we really only want mosaics. Just like asking which one is better, this feels like missing the point: it's quite obvious that both contribute to a research field. Mosaics leverage the power of normal science to explore particularly productive assumptions, whereas palimpsests question these assumptions and clear the messy jungle of details and confusion around complex ideas and notions. There might be incredibly rare cases where only one makes sense, but no field of science (certainly not alignment) currently has this property.

With all that said, I don't really know how to correct this bias. I feel like LW and the AF already have a partial community-wide acknowledgment of it, given the repeated encouragement to write more process-giving and iterative posts. That being said, this hasn't translated that well into more karma and comments for people doing this in my experience.

This is one of the open problems around mosaics and palimpsests.

Open Problems

• (Failure Modes) How can mosaics and palimpsests go wrong? What interventions help deal with these problems?
  • Some initial thoughts: mosaics oversimplify and so must be confronted with the relevant lost complexity; palimpsests get lost in details and tangents and so must be regularly redirected towards concrete applications.
• (Good Palimpsest Writing) How to write good palimpsests?
  • Most of our good writing technology and techniques aim at making mosaics: have a clear and simple thesis, have a nice structure, be short and focus on one idea,...^[9] Most of these either don't apply or don't fit with palimpsest-oriented people. Still, there must be better ways of making palimpsests and iteration legible.
• (Accelerating Palimpsests) How to make iteration in palimpsests happen faster while keeping their good properties?
  • Remember that palimpsests come out of Galenson's experimental innovators, who tend to be old masters. Yet if you believe in short timelines, most people don't have the time to become old masters. How can we (palimpsest-oriented thinkers) get stronger as fast as possible in consequence?
• (Ideal Ratio) What proportion of mosaics and palimpsests is best?
  • It will of course depend on the state of the field, what already exists, how hard the problem is.
  • One interesting subquestion asks about the existence of essentially correct mosaics (like John's True Names [AF · GW]) which don't need any questioning from outside and capture everything relevant.
• (Ratio With Limited Numbers) What ratio should we optimize for when we only have a limited number of people?
  • Depends on the state of the field
  • Direct application to Refine
• (Mosaic Pluralism) How to foster more mosaics and make them coexist?
  • Main issue comes from the "jealous lover" quality that mosaics share with paradigms — they don't work well with each other because of their clashing assumptions.
  • Yet at the level of the field, exploring multiple structuring assumptions in parallel has clear benefits.
• (Palimpsest Pluralism) How to foster more palimpsests (on the AF notably) and have them generate new ideas?
  • Whereas mosaics don't play nice with each other, palimpsests have trouble proliferating for more sociological reasons: they're not rewarded, and iterating in new directions is plain hard and often requires extensive knowledge of what has been done.
• (Historical Investigation) Historically, in different fields, are most innovations mosaics or palimpsests? What role is played by both in the history of science and technology?
• (Iteration on the concepts) What is missing from these concepts? How to improve on them?
  • I make palimpsests, and I'm the champion of productive mistakes, so it's painfully obvious to me how far from perfect and inadequate these concepts probably are. Still, I expect them to be useful as first order approximations. What are the second and higher-order approximations?

 

1. ^{^}

   This list reflects my own interpretation, and any pushback — especially from the mentioned researchers — would be interesting.

2. ^{^}

   Note the pattern that while mosaic-shaped research points to specific works, palimpsest-shaped research is harder to isolate from all the work of the author.

3. ^{^}

   This might seem weird given that Evan cowrote the mosaic-looking Risks from Learned Optimization [? · GW]. Yet based on my interactions with him, Evan iterates a lot on his ideas, encourages others to do so, and changed his framing of multiple points of Risks — while still keeping his aim at deception.

4. ^{^}

   I think this one is correct but more subtle. I go into some detail in this comment [AF(p) · GW(p)].

5. ^{^}

   This one still needs to be published, now that I finally realized I shouldn't aim for mosaics.

6. ^{^}

   One clear exception: Paul is palimpsest-oriented yet is widely read and deservedly gets a lot of karma and comments.

7. ^{^}

   John makes a similar distinction in this comment [LW(p) · GW(p)] — palimpsests are rarely ready for "Come at me bro" type feedback.

8. ^{^}

   Once again noting notable exceptions, much of the work on alignment is underlied by productive disagreement with the initial mosaic of Eliezer. That is, Disagreements that acknowledge the core problem while refusing or adapting key assumptions of the original frame. See for example this recent post [LW · GW] by Paul for an explicit exploration of such productive disagreement.

9. ^{^}

   Yet not all mosaics have to be written that way: Eliezer almost always prefers a literary style for example.

3 comments

Comments sorted by top scores.