Novelty Generation - The Art of Good Ideas

post by Shoshannah Tekofsky (DarkSym) · 2022-08-20T00:36:06.479Z · LW · GW · 2 comments

Contents

  A Model of Good Ideas
  Intro- & Outrospection
    1. Sleep/Wake Synthesis
    2. Stillness & Spontaneous Generation
    3. Think Harder & Puzzling
    4. Externalizing Cognition
    5. Social Cognition
    6. Exploration & Recombination
    7. Deep Immersion: Tetrissing  
  Novelty Generation & Pre-Paradigmicity 
  Addendum: Experiments
None
2 comments

Recently I set out to understand how new ideas are formed in the mind. My process consisted of poking at my own mind, poking at other people's minds, and then organizing group sessions of poking at each other's minds. 

The following is a list of insights from introspection and outrospection (a word I made up to refer to probing other people on their introspection), my guesses on how this relates to scientific progress, especially in a pre-paradigmatic [LW · GW] field,  and an addendum on the experimental exercises we tried. Once I told people about my ideas, I discovered my insights overlap heavily with the Babble & Prune [? · GW] sequence as well the Bus Ticket Theory of Genius. I suspect there might be more written about this, but here's the specific, shiny new wheel I reinvented! 

A Model of Good Ideas

But first, what's the target?

My exploration started with attempts to generate the most novel ideas. How are they formed? How can we generate them faster, more efficiently, just ... more-newer? How do we even generate ideas that are farther off the norm?

In response, someone suggested using LSD to maximize novelty generation. That made me pause and reflect. "That's not the thing I mean", I realized. LSD creates a huge perturbation in your experiences and idea formation, but most of it is ... not useful? Which made me realize, most novelty is noise. Random recombinations are very new and (mostly) very useless. There must be a second factor. That second factor is coherence.

If you consider Novelty on the Y-axis and Coherence on the X-axis, you have a 2-dimensional space of idea quality. AI alignment research needs new ideas... but like ... good, new ideas. 

A useful analogy is aliens. People who think of aliens and generate high novelty and low coherence ideas are conspiracy theorists and UFO catchers. People who are high on coherence but low on novelty dismiss the concept of aliens. But people who are high on both coherence and novelty might generate something like the Fermi paradox. Of course, once an idea has been generated, and it's a good idea, it will be adopted by all high coherence people, no matter their skills at novelty generation. But creating the theory in the first place takes both. 

This explains the uniqueness of "geniuses". Einstein or Da Vinci were not the smartest people, but they were extremely smart (coherence) and extremely creative (novelty generating). My guess would be that the two dimensions are independent, and thus it's hard to find someone who is exceptionally high in both skills. Add to that that they need to become interested in high-breakthrough-potential areas and we end up not seeing many geniuses in our lifetime.

Intro- & Outrospection

So the target is to maximize on high novelty, high coherence ideas. How do people do that? Or at least, how do people think they do that? I've heard 7 techniques so far (including my own).

1. Sleep/Wake Synthesis

This all started cause I wrote my first essay on Less Wrong [LW · GW] while lying in bed with a fever. 

The essay was well received. 

I wrote it in 3 hours. 

My internal experience of writing it was ... odd. 

But apparently very efficient!

This is how it felt: I was lying in bed, having slept 1,5 days already, and all I wanted to do was sleep more. However, I kept having these high-speed, high fidelity, intrusive thoughts on how all the AIS reading I'd been doing that month combined and related to each other. It was a fever dream made of ideas - an 'A beautiful mind' scene across coherent thoughts. 

In the end, I got up and grabbed my laptop in the hopes that writing out my thoughts would still them, and let me go back to sleep. Writing was easy as most of my thoughts on what to write came fully formed, including the structure of the essay.

Two weeks later I tried to recreate this process. My hypothesis was that there was something about the state between sleeping and waking that allowed my brain more bandwidth for cognition (sleep state) while also keeping coherence (waking state). So after getting a full night's rest, I woke up 30 minutes earlier than I needed to. I closed my eyes again, and started thinking about a Dungeons & Dragons home brew that I was working on, moving the pieces around in my mind. My visualizations were immediately more high fidelity than trying to imagine things in a full waking state. Next, I noticed the recombination of narrative elements and creation of peripheral items (e.g. what clothes the characters would be wearing) was near effortless and branched easily. Ideas just coalesced, more like dreaming, while they had the coherence I'm used to from thinking. I managed to generate the whole first scene of my home brew in less than 30 minutes, including plot line, puzzles, and dialogue. This would normally take me a couple of hours, and then I wouldn't necessarily be happy with the result. Now I was very happy. What's up with that?

Over my last 3 weeks in Berkeley I've met 3 people who explicitly lie down and close their eyes to solve problems. They report the same experience as I do. Conversely, I haven't actually found anyone yet who has tried this method but couldn't get it to work. However, that was mostly cause I didn't run in to people that tried at all. I only discovered this technique now at 36, so I'm not sure what's up. I can't say what peripheral factors make this method work for someone or not, cause my anecdotal data set is 4 people who love it and 30+ people who never even tried. So all I can say is, try it? (in case this needs saying: make sure you are fully rested before doing this, or you'll simply fall asleep!).

2. Stillness & Spontaneous Generation

People report variations on the theme of letting their thoughts wander while experiencing a sense of calm. Shower thoughts, long walks, and meditation all fall in this category. My best guess on what's happening here comes from the ACX review on Consciousness: The subconscious parts of your brain are processing information in the background all the time. When your consciousness becomes more "still" - less preoccupied with a particular thing - that opens up space to notice what kind of results your subconscious is outputting. The internal experience of this is thoughts and ideas "just coming to you". You can observe them, and recombine them, follow them up, or let them wander past. Some people get amazing results from this, but the caveat is that you don't have access to a consciously directed "search" as you do with the other methods described in this essay. Your subconscious is as likely to throw ideas forward on how to recombine tonight's salsa slightly differently as it is to provide ideas on to how to solve AI alignment. That said, your subconscious seems to approximately prioritize whatever information you have been feeding it most (and maybe also what information elicits the strongest emotions?) so there is some indirect control on the distribution of ideas that will bubble up. Experientially, it's a bit of a random generator of ideas, but the underlying mechanic is more like a draft deck that you've stacked with your experiences and attention so far. I think this process is a common way to generate ideas, and the two key parts are: 1) how to stack your deck in a useful way, and 2) how to clear your mind effectively so you can notice and review the ideas your subconscious is generating.

3. Think Harder & Puzzling

This method seems to be the one taught in traditional schooling systems: if you don't know how to solve a problem, then think harder. This essentially means focusing all your attention on the problem, and trying to feel and probe every corner of it. You'll find your mind tracing the outlines of the problem, recombining the representation of the problem, searching how to fill in gaps in the problem, studying what others have said about the problem, and ... beating your head to pieces against the problem. Experientially, it feels a little like those faux-detective shows where someone is trying to solve a mystery, finds the security tapes, and then "enhances" the video 5 times to find the missing clue in the reflection of the butler's shiny waist coat button. There is that sense of searching, but also that sense of forcing your mind to increase the resolution at which you understand the problem, and that sense of trying to look at the problem at odd angles in the hopes that something clicks. In to place. Like a puzzle piece.

4. Externalizing Cognition

Writing or saying your thoughts out loud pipes them through different cognitive filters and generation processes. Our language models do internally approximate GPT-3 somewhat in that we finish sentences in ways that "feel right" and there can be some drift and correction as we go. Writing and speaking your thoughts also forces you to notice gaps in your reasoning, as the sequential nature of language and putting it out in the world makes it far easier to notice where your mind might have glossed over some bits. Noticing the gaps in your thinking, and generating the language to encompass your thoughts both allow for avenues for thought generation. Both writing and speaking allow for the Babble & Prune [? · GW] method where one let's their cognition flow freely to the outside, piping thoughts through language synthesizers, followed by a review step to prune, experiment, and perfect the model that was created. Writing has the benefit of being highly reviewable and socially more acceptable than mumbling to yourself in a corner. Speaking tends to be higher speed and more ephemeral, which makes me hypothesize it might be more useful for novelty generation and less useful for coherence filtering. 

5. Social Cognition

A fan favorite is sharing ideas with others. The key element here is the synthesis of a collective intelligence process that relies on different people playing the ball back and forth. The input from other people can recombine with your existing frameworks in to new thoughts, which you then output, which then recombine with their frameworks in to new thoughts, which they out, etc. Instead of your brain synthesizing ideas within itself, you're performing the same process with other humans. Bandwidth between humans is lower than within one human, but the inspiration (novelty) and feedback (coherence) you can get from social cognition across ideas essentially constitutes the only known way of increasing your novelty and coherence above their natural limits. The sum can be greater than it's parts, and is pretty much what all human civilization and achievement is based on. So you know, definitely do that thing where you talk/write to your peers. Don't be scared to sound stupid. You stupidity might be the perturbation that pushes someone else out of a local optimum. Or worse, your stupid question is actually pretty smart but no one has dared ask it yet. That's where the cool loot is. And well, your smart ideas are good too. You should share those as well.

 

6. Exploration & Recombination

Do truly novel thoughts exist? My thesis is that all "new" ideas are novel recombinations of existing bits of information in your brain. Some of those thoughts emerge from how your brain happens to be wired and others emerge from all the inputs you have experienced. Therefore, it follows that it is easier to generate new ideas the more experiences you have collected. Novel experiences are like the building blocks of new ideas, and it's always better to have more blocks (barring the costs of acquiring them). Therefore, the exploration method of idea synthesis is more related to the physicist who plays the violin and thus connects concepts between them together to come up with new ideas. 

However, not all exploration is created equally. Some areas provide more functional overlap than others. E.g. Byzantin history and game theory (Von Neumann) will result in more high coherence ideas being generated than tennis and biology. The more two areas overlap in the underlying logic (coherence) while providing a large frame shift (novelty) the more promising the source of exploration and recombination is. Therefore, the trick to applying this method is to explore your pareto frontier [LW · GW] for unusual, high coherence links in your interests and abilities. Don't deny yourself your "relevant" hobbies! You never know when inspiration might strike.
 

7. Deep Immersion: Tetrissing  

Of all the above strategies, this is the only one I've never consciously tried to apply. It was explained to me as a deep immersion process where you dedicate yourself to the relevant problem or activity day in, day out, all day, every day - till the whole universe is tiled with that one experience. Ha! 

But really. 

The experience is probably not foreign to programmers or artists. Or maybe even some students. Say we use the example of tetris. You play every day for hours on end, you read about tetris, talk about tetris, think about tetris, dream about tetris ... till all you see in the world is interpreted as tetris. You see the couch as a tetris block. You stack your fridge efficiently. Cleanly. Perfectly. Cars on the street are moving tetris blocks. Your mind involuntarily rotates them, and scoffs at the inefficient spacing. Your stims and fiddling consist of stacking things precisely in form-fitting configurations. 

All life is tetris.

Now apply this to a problem you are trying to solve. Immerse yourself deeply. Parse all of reality through the lens of the problem you are diving into. Everything is an inspiration source. The feeling of the wind on your skin, the sound of your friend's voice, the way plants grow and intertwine. Everything gets related back and reconfigured. Most of your ideas make no sense, but the search process across novel ideas is high intensity and uses all possible perturbations that it can.

Yeah, ok, I'm kind of excited to try it now.

 

Novelty Generation & Pre-Paradigmicity 

A field being pre-paradigmatic means no one really knows the right way to look at the problem. What's the right frame? In which direction do the answers lie? 

We are basically running a search algorythm across solution space to find the sector that contains the highest coherence ideas (aka, best solutions). Finding new sectors to explore is completely down to novelty generation. Though, to solve the alignment problem, we need to discover new sectors that are also high on coherence - new ideas that are actually good.

So based on the above methods, I'd recommend the following study techniques:

1. Write out your ideas, no matter how stupid.
2. Talk about your new ideas with others. Don't be afraid to ask dumb questions. You need the answers to learn, and maybe you inspire someone on the way.
3. Sleep. And then lie in bed some more. Except if you hate it. But at least try it.
4. Set some time aside to blend tetrissing and thinking harder. They are natural buddies. This is ideally done for at least a month of full-time, all-the-time dedication. 
5. Explore and recombine. Don't neglect your hobbies. Follow your curiosity and incentives through life, but try to loop back your experiences to your studies. This strategy is the opposite of tettrising and thinking harder.
6. Avoid group think and other social dynamics. Write down your first thoughts and questions. Then talk to people or read about them, and update. Create an a priori and an a posteriori model. Fill in the gaps yourself.

There are probably more good techniques, and these might be old news, but either way, thank you for reading about the shiny new wheel I invented.

---

Addendum: Experiments

See comments.

2 comments

Comments sorted by top scores.