Looking for a likely cause of a mental phenomenon

Does this phenomenon happen at all, or is it a product of selective memory? It seems to me that incidents where a solution occurs to us while we are thinking of something else will be more memorable (and make better stories) than incidents where we find a solution while explicitly working on it.

How would we distinguish a world in which stepping away from a problem makes a solution more probable to occur from a world in which stepping away from a problem makes a solution more awesome-feeling and therefore more likely to be remembered and repeated as a story?

One thing that I'm pretty sure is going on and that might be a sufficient explanation is that it takes time to develop fluency in a hard problem. You can solve a simple problem in one go if you can hold enough of it in your mind to see the next steps of a plan that prove useful, and the same happens with the results of those steps, and eventually you reach a solution. For a harder problem, you might fail to see a specific plan, so you develop various observations about the problem and additional representations of its aspects, without having a clear sense of which of them will be useful, and there are too many of these to hold in your mind at the same time, as even though the new observations may be obtained by the methods you know well, they are in themselves new facts that are not yet thoroughly familiar.

To get to the next step, it might be necessary to be able to access a lot of these observations easily, without spending attention on recreating them. It takes time to familiarize yourself with the new observations (and with the way they connect to the original problem and to each other), to commit all these details to long term memory and to train your imagination to easily retrace the connections between them. But once you've done so, you obtain new superpowers with respect to that problem (and perhaps others analogous to it). A proof that would've taken 15 steps in terms of the ideas you had when you started working on the problem (and so wasn't apparent), now takes only 4 steps in terms of the new auxiliary ideas you've developed in the meantime, and you can see it at a glance. (Perhaps if your mind wanders when you're stepping onto a bus, and spends a few seconds on the problem, this proves sufficient to take advantage of the prior training and notice the solution.)

This analysis suggests that if you are stuck on a problem you need to solve, and you have enough time on your hands, (1) you should deliberately and systematically study the observations associated with your problem, even the ones that don't seem immediately useful, and those that are easy to obtain, until these observations and their connections to the rest become obvious without the need to concentrate on reconstructing them, (2) revisit all (relevant) parts of the problem when you expect that new observations have been internalized since the last time you've revisited the problem.

There's been a significant amount of work on this problem, if you are wanting what I think you are wanting. Quick googling got me:

A Generic Article

Jogging, doing the dishes and riding the bus to work are other examples.

The results of neuroimaging studies indicate that these types of situations allow the outermost regions of the prefrontal cortex - those areas of the brain that help exert cognitive control - to loosen the reins and allow thought processes and neural activity not strictly related to the primary task.

In this state of defocused attention, one idea can trigger the next across a relatively unconstrained range of concepts and associations that might otherwise be viewed as completely unrelated. The resulting novel connections may explain how a broad focus can significantly facilitate creative thought and inventive problem-solving.

It may also explain how wandering thoughts about a thistle burr, a hymn-book marker and a falling apple could have inspired developments as useful as Velcro, Post-it notes and a theory of gravity.

And a wikipedia article on "incubation" ( http://en.wikipedia.org/wiki/Incubation_(psychology) )

Incubation is one of the 4 proposed stages of creativity: preparation, incubation, illumination, and verification.[1] Incubation is defined as a process of unconscious recombination of thought elements that were stimulated through conscious work at one point in time, resulting in novel ideas at some later point in time.[2]

The experience of leaving a problem for a period of time, then finding the difficulty evaporates on returning to the problem, or even more striking, that the solution "comes out of the blue", when thinking about something else, is widespread. Many guides to effective thinking and problem solving advise the reader to set problems aside for a time.

...Recent advances in neuroscience provide intriguing evidence of the mechanisms underlying incubation effects, particularly those that occur during sleep.

Unconscious annealing of connections between concepts.

My suspicion about the mechanism is that as you work on a problem, you tend to get stuck in one particular approach. When you step away from the problem, you allow yourself the possibility of approaching it from a new angle. Being so focused on idea X before, could have prevented you from considering alternative ideas Y and Z. So suppose you're thinking about something else, but your brain then briefly flicks back to the problem. This time, ideas Y and Z popup, and it's suddenly obvious that if you combine them, that you'll get the solution. So, it isn't clear at all that you would need any kind of background processing to occur in order to observe this effect. It wouldn't surprise me at all if sleep helped here as we know it rearranges memories, but I've seen this effect without any sleep at all and I don't know enough neuroscience to know if there is any brain optimising going on when you're awake.

For purposes of an informed discussion, vide Daniel Schacter's The Seven Sins of Memory.

Note: The paper is old, but provides good background information for those who lack it. I can't think of an answer yet, but wanted to help others who may not know where to start. The file will deactivate if for thirty days none download the file.

Munchkin is a verb now? A gloss is needed.

I have a couple speculations here, but the're not on the level you're asking for.

The first is that they're getting stuck because the dead end thoughts are becoming too salient and crowding out other thoughts. This certainly happens when getting stuck with emotional issues, and I strongly suspect the same thing is going on with math - especially if you're feeling frustrated and stubborn about it. This suggests the solution of reducing the saliency of those thoughts, and indeed I have had success with that method in physics tutoring (though that was an obviously distressing meta level thought)

The second is a bit more speculative. I think that the problem really is on your mind, just not at the forefront. Sorta like how your dreams reflect stuff you've been thinking about a lot even if you weren't actively thinking about it when you went to sleep. The way I think it works is that you're running into all sorts of patterns and you just happen to come across some with some isomorphism in it. The classic example of this is the benzene ring and the daydream about snakes eating their own tails. This suggests you start looking at things with varied structure that might be similar enough and not trying to make the connections. I haven't experimented with this. Low doses of psychedelics also up your pattern matching and make these connections more likely. I have experiemented with this, and while it has given me insights on problems I was working on and problems I wasn't working on, I don't remember any with a good control where I was totally stuck until trying a low dose psychedelic.

As far as the "looking for answers in neuroanatomical structures" bit, that's how I started my research into hypnosis, but I ended up giving up on that path. The research I could find was strong evidence that it was legit, but it wasn't that helpful for a couple reasons. For one, "structure X is involved in Y" leaves a whole lot unsaid as to how it does Y and gives little more than a general direction to throw ideas and see what sticks. And perhaps more importantly, my interventions aren't on the level of neuroanatomical structures.

I've gotten a whole lot more mileage out of looking at how thoughts and feelings relate to other thoughts and feelings, and not so much where they physically live. Have you actually found otherwise?

I don't have much help on the neuro end, but my go-to explanation of this hasn't been assigning tasks to the homunculus, so here's what inspiration feels like to me, and if you can draw anything useful out of it, mazel tov!

The kinds of problems that feel like they are frequently resolved in this way for me include math proofs (esp linear algebra, topology, fractal geometry, for reasons that may become obvious), theses, and speeches I give or blog posts I write. In college, I was part of a philosophical debating group, and I pretty much never wrote speeches in advance. I'd think about the topic during the week, listen to speeches during the debate, and then have some BLAM moment during the debate, where I suddenly could feel the logic of my speech, and had a sense of the shape of it and where certain quotes/anecdotes/studies would fit in. Then I was ready to take the floor.

Here's what my mental map of this process is: I start thinking about a problem by thinking intensively about particular aspects of it -- these are nodes. I start building up references, linked ideas, objections, difficulties around each of these nodes. So, basically, each node starts becoming a more densely connected subgraph of (what will hopefully be) an eventual solution graph. During my active thinking, I keep taking walks in my mindspace from these nodes, hoping to find a path to link up these subgraphs. Sometimes things resolve here.

When they don't, when the problem isn't actively before me, I think of the subgraphs as still accreting ideas and connections. But now it's more of a random walk than when I'm directing it and looking for a particular connection. I imagine that when I call up some specific idea (a song lyric, a subgraph for a different problem, a passage in a book) the near connections of it are weakly activated, and, if they're close enough to one of the subgraphs I've been thinking about recently, they get aggregated on, and extend the graphs out in serendipitous ways. If the new thing I'm thinking about (or any of the things it's weakly linked to) links up two subgraphs, or puts me near enough to spot the connection, that's what inspiration feels like.

And sometimes, a weakly lit up idea prompts inspiration by being grotesque. (By grotesque, I mean close-but-not-close-enough to the right thing to give you a wiggins, and draw your attention to the particular way it falls short of what you need). The uncanny valley is for more than just robots!

Here are some examples from fiction that feel similar to my subjective experience of inspiration, at least in some particular:

• Lyra's use of the alethiometer
• Miranda thinking in strings that loop around and connect up in Beggars in Spain
• This scene from the Hound of the Baskervilles episode in Sherlock
• Nita and Kit opening a gap in their shield and waiting for something the right shape to fall in in So You Want to Be a Wizard
• The sound design and stage directions for Kid Simple: A Radio Play in the Flesh

Basically, thinking about a problem is building a big graph, but interacting with any of the other graph-thoughts I think about in day to day life might light up a connection to a node of interest. I find the more SNS-kinda mode more helpful here (I'm in the same kind of mood at debate as I am at improv, resting lightly on things, curious about where they'll go).

My own experience, and much of what I've read, suggests a role of dreams and sleep in insight-generation. Perhaps you solve the problem while sleeping, but your brain doesn't store the memory of figuring it out in detail because you're asleep, and the flash of insight you feel in the shower is just remembering what you already figured out.

The little man could just be the promise to yourself to sleep on it, to literally work on the problem while asleep. And you need to walk away from the problem because conscious, deliberative approaches to problem solving will trample the delicate memory of the solution you arrived at while sleeping.

From what I've learned about brains, the left brain is engaged in symbolic thinking about a problem, which engages more logical, methodical problem-solving. For a combination that you won't arrive at through that approach, you have to give your brain, apparently involving activation of the right dorsolateral prefrontal cortex and other right-brain parts, more time to integrate info from stored memory or lower-level processed stimuli or to make novel associations related to the problem. When left prefrontal cortex is engaged in focusing on performing a task, it'll inhibit the processing of info seemingly irrelevant to the task. This is why aha/eureka moments are more likely when you're relaxed, not focused and your mind gets to wander (e.g. getting on bus while on vacation, taking a shower/bath). Studies suggest that more creative or sudden-insight (as opposed to deliberately trying different combinations) problem-solvers have greater right brain activity and lower inhibition of it.

Look up "Aha! moments" in the index of Eric Kandel's book, The Age of Insight, which cites many papers, incl. "Explaining and inducing savant skills: privileged access to lower level, less-processed information". A few of my other references: "Bayes for Schizophrenics: Reasoning in Delusional Disorders", "Creativity tied to mental illness", "Through the Wormhole: Creativity Cap"

I think the answer is hiding in this bit: "In this particular case it also came with a feeling of total confidence that verification would pan out (although Hadamard notes that the validation step after the insight is still essential because sometimes that feeling of total confidence is mistaken)."

I'm going to guess that the class of problems this work on is where the person knows somehow that there IS an answer. Almost as if they see the pattern from the outside yet haven't just worked out the details. This would explain why when they get the answer they 'know' it's correct. It would also explain the hard work AND the frustration of not getting it. After all, if you didn't intuit that there was an answer, if on some level you didn't see it, well then you wouldn't feel frustrated at not getting it!

And so the question then is, if you know the answer, if you can see it, well then why the heck can't you know it?!

For that I think that sometime it takes some time to fully experience what you see and know. For it to open up completely from implicit body knowledge to explicit cognition.

Now if you don't think about it at all, when then it's dormant. It's not on the table. At the same time, if you work on it too hard you very very often block yourself by 'going too fast' with your attempts to 'get it out there already'.

When you work on it, then stop, you have put it on the table and at the same time you aren't getting yourself stuck in a rut by trying too hard.

As for what is actually happening when you aren't thinking about it, well at all times you're implicitly holding in mind many many things. Most are invisible to you (the fact that you live in country X and have family Y and have profession Z). You act on those implicit contexts, you must be 'making contact' with them, yet they are invisible (do the fish know of the water they swim in).

Added to that are the implicit stuck things in your life. Like knowing that the whole thing with the landlord is not resolved... Even when you aren't thinking about it, you feel it. You sense it. Walking around during the day with that issue 'on the table' compared to walking around without it is a completely different experience.

A good way to experience this experientially in less then 5 minutes is to ask yourself "What is not perfect in my life?" When you get the answer, write it down and then say "And X isn't perfect, and without X, what's not perfect in my life?" Keep going till it stops. You'll know when to end... :)

When you do that exercise you FEEL the body sense of layers on implicit stuff that you're holding come off, if even for a moments.

When you have a difficult problem that you KNOW is solvable and you can't find it, well you feel that. All day. And that's the implicit processing going on.

(Let us know how the exercise went).

The literature calls it "incubation". Happy digging.