Words as Mental Paintbrush Handles

post by Eliezer Yudkowsky (Eliezer_Yudkowsky) · 2008-03-01T23:58:21.000Z · LW · GW · Legacy · 75 comments

(We should be done with the mathy posts, I think, at least for now.  But forgive me if, ironically, I end up resorting to Rationality Quotes for a day or two.  I'm currently at the AGI-08 conference, which, as of the first session, is not nearly so bad as I feared.)

Suppose I tell you:  "It's the strangest thing:  The lamps in this hotel have triangular lightbulbs."

You may or may not have visualized it—if you haven't done it yet, do so now—what, in your mind's eye, does a "triangular lightbulb" look like?

In your mind's eye, did the glass have sharp edges, or smooth?

When the phrase "triangular lightbulb" first crossed my mind—no, the hotel doesn't have them—then as best as my introspection could determine, I first saw a pyramidal lightbulb with sharp edges, then (almost immediately) the edges were smoothed, and then my mind generated a loop of flourescent bulb in the shape of a smooth triangle as an alternative.

As far as I can tell, no deliberative/verbal thoughts were involved—just wordless reflex flinch away from the imaginary mental vision of sharp glass, which design problem was solved before I could even think in words.

Believe it or not, for some decades, there was a serious debate about whether people really had mental images in their mind—an actual picture of a chair somewhere—or if people just naively thought they had mental images (having been misled by "introspection", a very bad forbidden activity), while actually just having a little "chair" label, like a LISP token, active in their brain.

I am trying hard not to say anything like "How spectacularly silly," because there is always the hindsight effect to consider, but: how spectacularly silly.

This academic paradigm, I think, was mostly a deranged legacy of behaviorism, which denied the existence of thoughts in humans, and sought to explain all human phenomena as "reflex", including speech.  Behaviorism probably deserves its own post at some point, as it was a perversion of rationalism; but this is not that post.

"You call it 'silly'," you inquire, "but how do you know that your brain represents visual images?  Is it merely that you can close your eyes and see them?"

This question used to be harder to answer, back in the day of the controversy.  If you wanted to prove the existence of mental imagery "scientifically", rather than just by introspection, you had to infer the existence of mental imagery from experiments like, e.g.:  Show subjects two objects and ask them if one can be rotated into correspondence with the other.  The response time is linearly proportional to the angle of rotation required.  This is easy to explain if you are actually visualizing the image and continuously rotating it at a constant speed, but hard to explain if you are just checking propositional features of the image.

Today we can actually neuroimage the little pictures in the visual cortex.  So, yes, your brain really does represent a detailed image of what it sees or imagines.  See Stephen Kosslyn's Image and Brain: The Resolution of the Imagery Debate.

Part of the reason people get in trouble with words, is that they do not realize how much complexity lurks behind words.

Can you visualize a "green dog"?  Can you visualize a "cheese apple"?

"Apple" isn't just a sequence of two syllables or five letters.  That's a shadow.  That's the tip of the tiger's tail.

Words, or rather the concepts behind them, are paintbrushes—you can use them to draw images in your own mind.  Literally draw, if you employ concepts to make a picture in your visual cortex.  And by the use of shared labels, you can reach into someone else's mind, and grasp their paintbrushes to draw pictures in their minds—sketch a little green dog in their visual cortex.

But don't think that, because you send syllables through the air, or letters through the Internet, it is the syllables or the letters that draw pictures in the visual cortex.  That takes some complex instructions that wouldn't fit in the sequence of letters.  "Apple" is 5 bytes, and drawing a picture of an apple from scratch would take more data than that.

"Apple" is merely the tag attached to the true and wordless apple concept, which can paint a picture in your visual cortex, or collide with "cheese", or recognize an apple when you see one, or taste its archetype in apple pie, maybe even send out the motor behavior for eating an apple...

And it's not as simple as just calling up a picture from memory.  Or how would you be able to visualize combinations like a "triangular lightbulb"—imposing triangleness on lightbulbs, keeping the essence of both, even if you've never seen such a thing in your life?

Don't make the mistake the behaviorists made.  There's far more to speech than sound in air.  The labels are just pointers—"look in memory area 1387540".  Sooner or later, when you're handed a pointer, it comes time to dereference it, and actually look in memory area 1387540.

What does a word point to?

75 comments

Comments sorted by oldest first, as this post is from before comment nesting was available (around 2009-02-27).

comment by Tom_McCabe2 · 2008-03-02T00:47:28.000Z · LW(p) · GW(p)

"What does a word point to?"

I'm not a neurologist, but I believe the classical answer is that words point to patterns in clusters of neurons. Everything filed under "apple" slightly rearranges neurons in the "apple" category, including pictures of apples, the taste of an apple, stories about apples, etc. Features which all apples have in common are amplified by the pattern-matching, while dissimilar features cancel each other out. Eventually, we get a mental pattern of the "archetypal" apple, describing the category in general, which can be drawn or described from scratch, without identifiable references to any specific apple. (If you haven't personally experienced this with apples, try ideas. Relying on one or two authors for your information will cause you to include some of their identifiable little details in your own writings.)

comment by Douglas_Knight3 · 2008-03-02T02:19:06.000Z · LW(p) · GW(p)

It's not the fault of the behaviorists. It's mentioned in: Galton (1880), Statistics of mental imagery, Mind 5 p 301-18. (jstor, oup)

comment by FungibleFun · 2008-03-02T03:47:02.000Z · LW(p) · GW(p)

You've been reading too much Hume.

comment by Yvain2 · 2008-03-02T04:11:30.000Z · LW(p) · GW(p)

I had a professor, David Berman, who believed some people could image well and other people couldn't. He cited studies by Galton and James in which some people completely denied they had imaginative ability, and other people were near-perfect "eidetic" imagers. Then he suggested psychological theories denying imagination were mostly developed by those who could not themselves imagine. The only online work of his I can find on the subject is http://books.google.co.jp/books?id=fZXoM80K9qgC&pg=PA13&lpg=PA13&ots=Zs03EkNZ-B&sig=2eVzzMmK7WBQnblNx2KMVpUWBnk&hl=en#PPA4,M1 pages 4-14.

My favorite thought experiment of his: Imagine a tiger. Imagine it clearly and distinctly. Got it? Now, how many black stripes does it have? (Some people thought the question was ridiculous. One person responded "Seven. Now what?")

He never formally tested his theory because he was in philosophy instead of the sciences, which is a shame. Does anyone know of any modern psychology experiment that tests variations in imaging ability?

Replies from: diegocaleiro, DilGreen
comment by diegocaleiro · 2010-09-08T21:50:58.906Z · LW(p) · GW(p)

No large N experiments. but Feynman in one of his autobiographies tests this with a friend. One of them hears numbers, the other sees them. They are unable to multitask within the domain they use to process numbers. I for one hear numbers. I can count while performing visual tasks. My father sees them. He cannot. He can speak and count, which I find amusing.

Replies from: Lomky
comment by Lomky · 2012-03-26T14:11:01.456Z · LW(p) · GW(p)

I can read and count, but not speak and count. While trying this out, I realized I can not count silently, but am always pronouncing the numbers in my head. I wonder if that is related?

comment by DilGreen · 2016-07-12T11:50:22.623Z · LW(p) · GW(p)

It's been a few years, but the answer is now - yes. Here's a link to a New Scientist article from earlier this year. I'm afraid there's a pay barrier: https://www.newscientist.com/article/2083706-my-minds-eye-is-blind-so-whats-going-on-in-my-brain/ The article documents recent experiments and thinking about people who are poor or incapable (about 2 to 3% report this) of forming mental pictures (as opposed to manipulating concepts). Key quote:

To find out how MX’s brain worked, Zeman put him into an MRI scanner and showed him pictures of people he was likely to recognise, including former UK prime minister Tony Blair. The visual areas towards the back of his brain lit up in distinctive patterns as expected. However, when MX was asked to picture Blair’s face in his mind’s eye, those areas were silent. In other words, the visual circuits worked when they had a signal from the outside world, but MX couldn’t switch them on at will (Neuropsychologia, vol 48, p 145).

Test yourself here: http://socrates.berkeley.edu/~kihlstrm/MarksVVIQ.htm

comment by Caledonian2 · 2008-03-02T04:12:36.000Z · LW(p) · GW(p)

Words are just pointers. Concepts are the referents to which words make reference, and they are wordless things.

comment by Lee_Corbin · 2008-03-02T05:11:30.000Z · LW(p) · GW(p)

I knew a man who was chess champion of California who claimed that neither he nor his father could visualize anything. His father was a taxi driver in London, and long knew that the other drivers had mental maps, whereas he had to memorize lists of street names in just the right order. My friend, by the way, could also play blindfold chess just fine. He'd merely remember that "e7 is attacked on the diagonal from g5", presumably from long practice with the sequence e-f-g, 7-6-5. He lamented that his inability to visualize put an upper limit on his chess ability, and given how hard he worked, I believe it.

Galton was astonished that many scientists of his acquaintance did not visualize. Judging from Galton and other things I've read, the ability to visualize is more common today. TV, maybe. Please try to picture this without thinking of the old joke: "A man walked down the street, and turned into a drugstore". (The joke, of course, is that people do not become drugstores.)

Now did you happen to notice which side of the street he was walking on (left or right), and whether he was walking towards or away from you? The problem for those of us who use visualization for almost all our thinking, is that we must add irrelevant and often distracting information, which can be costly in math and science.

Replies from: Kenny, Bluestorm_321
comment by Kenny · 2013-02-12T14:56:09.854Z · LW(p) · GW(p)

Sometimes it's useful to add "irrelevant" information to better visualize the full implications of something, i.e. fill in the gaps.

Replies from: amaury-lorin
comment by momom2 (amaury-lorin) · 2023-01-27T19:44:25.912Z · LW(p) · GW(p)

Sometimes it is useful, but sometimes it is detrimentary (when you don't realise that the "full implications" are not actually implied, or believe that they are enough to paint the full picture).
In general, it's better to be aware that you add extraneous information, and be able to remove it at will, and discern whether or not you want to do it.

comment by ship_shlap (Bluestorm_321) · 2022-02-24T17:58:48.548Z · LW(p) · GW(p)

I don't have anything important to add but the inability to visualize is called Aphantasia and some people still have it today. 

comment by Douglas_Knight3 · 2008-03-02T06:26:58.000Z · LW(p) · GW(p)

Yvain and Lee Corbin, I tend to agree with Eric Schwitzgebel (which is how I found the Galton paper) that the difference in claims of visualizing ability is due to changing norms, not changing abilities. He's too quick to discount the possibility of real change, but professed inability of scientists in Galton's day is striking. I don't think that demographic has been adequately surveyed today, but I don't think they're as different from the general public as they were 50 or 100 years ago.

In particular, I don't think there was an early psychologist who couldn't visualize, as I wouldn't expect that to be so early as to have such a widespread affect on the scientists Galton knew. That is, I doubt a psychologist would have an impact on non-psychologists' self-assessment.

Also, I think people overrate the importance of visualization to navigation, blindfold chess, and other things where it seems intuitively important.

paper linked above: "How Well Do We Know Our Own Conscious Experience? The Case of Visual Imagery" (2002), Journal of Consciousness Studies, 9, no. 5-6, 35-53.

comment by Ron_Hardin · 2008-03-02T11:43:21.000Z · LW(p) · GW(p)

What does a word point to? See an essay on words as labels in Stanley Cavell The Claim of Reason p.175

In the background is always : what is this fantasy about? Meaning in this context the AI fantasy.

Actual robot fantasies begin around p.403

comment by Hendrik_Boom · 2008-03-02T14:24:18.000Z · LW(p) · GW(p)

other people were near-perfect "eidetic" imagers

I read a report in the Scientific American a few years ago in which they were doing experiments with rend0m-dot stereograms -- the kind of thing where if you just look at one image or the other you just see random dots, but if you look at one with one eye and the other with the other eye you see a square full of random dots floating above a background with random dots.

Some people could be shown one image one week, and the other the next week, each by itself, and suddenly get it. Evidently they had remembered the entire first-week image in memory to scompare with the second a week later.

I was impressed. This seemed eidetic enough for me.

Replies from: Ender
comment by Ender · 2011-09-05T18:02:49.583Z · LW(p) · GW(p)

I heard about this study in the book Moonwalking with Einstien: the Art and Science of Remembering Everything, by Joshua Foer. Apparently there was only one test subject who seemed to have eidetic memory, and instead of doing more tests after the one that you described, the experimenter married the subject.

When John Merritt put a similar test in newspapers, nobody who wrote in with the correct answer could do the test "with scientists looking over their shoulders."

Foer, Joshua. Moonwalking with Einstein: the Art and Science of Remembering Everything. New York: Penguin, 2011. Print.

comment by Caledonian2 · 2008-03-02T14:26:22.000Z · LW(p) · GW(p)
Now did you happen to notice which side of the street he was walking on (left or right), and whether he was walking towards or away from you?

Your visualizations include such details? As the description didn't include such details, they're necessarily undefined - so why did you define them out of their uncertainty?

How many of you dream in concepts rather than images?

Replies from: AndyCossyleon, Snowyowl, bigjeff5, Dojan, None, AspiringRationalist, notriddle
comment by AndyCossyleon · 2010-08-05T18:41:43.507Z · LW(p) · GW(p)

My visualization was thus: he was walking away from me, on the right side of the street, while I was on the left side. The street was also at incline, such that he was walking uphill, the sidewalks were rather wide, and the buildings were contiguous. it was afternoon, because the sun was even more to the right of the man (I guess that means he was also walking south), though I couldn't see it. The drugstore door swung out (oddly) around the left hinge, and it was on the left side of the drugstore. There were also cars parked on the street.

Basically, I visualized a random scene from a familiar inner city locale.

comment by Snowyowl · 2010-12-09T14:29:19.212Z · LW(p) · GW(p)

My visualisation was that he was walking away from me, on the right-hand side of the street. I was slightly above the right-hand side of the street, viewing the scene at a 30° angle or thereabouts. My visualisation didn't include any other details.

comment by bigjeff5 · 2011-02-11T04:09:26.953Z · LW(p) · GW(p)

He was walking away from me, left side of the street (sidewalk, obviously), and turned left into the store.

He was also dressed in a cowboy hat, blue jeans and a brown jacket.

The drugstore was a single story downtown style store - big glass windows, and a big sign that said "DRUG STORE" above the store.

My POV would be in the street about the middle of the left lane.

All that from "a guy walked down the street and turned into a drugstore". I could get a whole lot more if I kept thinking about it, but that would kind of spoil the whole "first thing that pops into your head" bit.

[EDIT] I've been reading the Dark Tower series lately, which is probably where the cowboyish image came from.

comment by Dojan · 2011-11-24T16:44:12.614Z · LW(p) · GW(p)

I saw him walking downhill, away from me, wearing a long, brown treanchcoat, and then suddenly flinging his arms wide, holding out the sides of the coat, on the inside of which there were lots of little pockets filled with syringes and bottles of pills, then the police suddenly comes into view and turns the sirens on, then I go, whait a minute! This can't be right...

comment by [deleted] · 2012-01-10T22:31:55.881Z · LW(p) · GW(p)

My own visualisation has now been irredeemably biased by reading all the above descriptions...

But mine had districtly less detail; or at least the unneccesary detail quickly vanished once I saw him turn to his left and enter a pharmacy.

comment by NoSignalNoNoise (AspiringRationalist) · 2013-05-06T03:37:35.478Z · LW(p) · GW(p)

Your visualizations include such details? As the description didn't include such details, they're necessarily undefined - so why did you define them out of their uncertainty?

Because a visual image cannot lack these details, in the same way that if you are given the prompt "visualize a rectangle" and you do, the image must have some aspect ratio, even though the prompt didn't specify what it should be. You can certainly imagine a man walking into a drugstore in a non-visual way and not include such details (that's what I did, for example).

How many of you dream in concepts rather than images?

For the dreams I remember, it's probably ~1/2 visual, ~1/3 concepts and ~1/6 sound only.

comment by notriddle · 2013-05-13T04:05:24.640Z · LW(p) · GW(p)

In my visualization, a man (don't remember what he looked like) was walking along a line with a 90 degree turn away from me (think of how GPS navigators represent roads, only the only thing I visualized was the road that was actually used). When he turned, the camera panned along with him, keeping his back to me, and we were facing a double-door entryway.

My visualizations seem to lack every detail they can possibly lack...

Replies from: VAuroch
comment by VAuroch · 2013-11-13T02:35:58.022Z · LW(p) · GW(p)

Mine lacked one detail yours had; the actual walking. In classic dream-logic, the man was briefly in a walking pose on a streetlike object, then instantly at the door of the drugstore, opening it and turning in.

On the other hand, in my image he had white hair and was nearly bald. So clearly I'm not 'winning' in a 'how much detail can you omit' contest.

comment by Meta_and_Meta · 2008-03-02T15:37:01.000Z · LW(p) · GW(p)

On behaviorism, I always liked Morgenbesser's alleged remark to Skinner: "Let me see if I understand your thesis. You think we shouldn’t anthropomorphize people?"

Replies from: anonym
comment by anonym · 2011-01-24T17:16:11.409Z · LW(p) · GW(p)

In the same vein:

One behaviorist says to the other, just after they've had sex: "it was good for you, how was it for me?"

comment by Eliezer Yudkowsky (Eliezer_Yudkowsky) · 2008-03-02T17:04:26.000Z · LW(p) · GW(p)

Some people could be shown one image one week, and the other the next week, each by itself, and suddenly get it. Evidently they had remembered the entire first-week image in memory to compare with the second a week later.

I defy the data. That doesn't sound like it should be possible.

comment by Jack2 · 2008-03-02T17:12:02.000Z · LW(p) · GW(p)

How many of you dream in concepts rather than images?

I've noticed this from time to time. It often seems that a dream will have a sense of urgency, or of being a child again, or of anything else, without any details from which this sense could be inferred. But it's not a flat 'all dreams' sort of thing; some dreams will be movie-like, others will be built out of pure feelings-that-something-is-happening.

Replies from: Amanojack
comment by Amanojack · 2012-04-13T18:51:56.718Z · LW(p) · GW(p)

I've had the same experience. To me this suggests that although not all conscious thoughts are visual, they may all be sensual. That is, of the five senses.

Actually, this is tautological if "conscious thought" means a thought we are completely aware of, unless we can be 'aware' of something other than five-sense experience.

Replies from: AspiringRationalist
comment by NoSignalNoNoise (AspiringRationalist) · 2013-05-06T03:40:16.232Z · LW(p) · GW(p)

We can be aware of emotions without them having any reference to the five senses.

comment by Tom_McCabe2 · 2008-03-02T17:23:01.000Z · LW(p) · GW(p)

"I defy the data. That doesn't sound like it should be possible."

I am also skeptical of this particular feat, but we know it's possible for the brain to record huge amounts of data in reasonably good detail in realtime. My theory is that everyone without a neurological impairment stores this data, but only a few rare people (savants and such) can access the data on request. To name a simple example, in one of my middle school classes, we had to read Dumas's The Count Of Monte Cristo. Very few people could read the book once or twice and recite the whole thing back in any kind of detail. Yet, when given a quotation from the book, most people were able to remember who said it, where they were at the time, who they were talking to, the surrounding context, etc.

comment by Caledonian2 · 2008-03-02T19:47:56.000Z · LW(p) · GW(p)

I can produce vivid memories of images, sounds, textures, smells, etc. But my dreams are in streams of words - not the sounds of speech or the appearance of text, but the meanings alone. If I had a dream of the man walking down the street and turning into a drugstore, there would be no direction he was walking in, no perspective relative to me, and he wouldn't be turning left or right. He'd just be.

So you'd have a movie-like experience? Mine is more like reading a novel - only an invisible novel. While I have no body. Everywhere.

Replies from: bigjeff5
comment by bigjeff5 · 2011-02-11T04:13:28.373Z · LW(p) · GW(p)

Novels for me are like movies that last for many hours instead of just a couple.

That's why I like to read them.

Replies from: wedrifid
comment by wedrifid · 2011-02-11T04:23:59.886Z · LW(p) · GW(p)

If you were reading a book and the characters were introduced as "this one looks like Summer Glau" and "this one looks like Amy Acker" would that set the scene for how the entire movie looked? That'd be awesome. :P

Replies from: bigjeff5
comment by bigjeff5 · 2011-02-11T10:17:52.363Z · LW(p) · GW(p)

Yeah, pretty much, though that's a pretty cheesy thing to do, heh.

Unless the author goes into great detail or I already have a particular person in mind for a character type, faces tend to be indistinct. For example, in Stephen King's Dark Tower series (which I'm finishing up now) I picture Roland, the main character, to be very similar to the classic "spaghetti western" Clint Eastwood, but everyone else is much less detailed.

Replies from: wedrifid
comment by wedrifid · 2011-02-11T10:40:58.151Z · LW(p) · GW(p)

but everyone else is much less detailed.

What a waste! All that imagination ability going to 'indistinct' when you could be having day long Summer Glau movies! It's almost criminal.

comment by Roko · 2008-03-02T20:20:59.000Z · LW(p) · GW(p)

"What does a word point to?"

  • unfortunately, I don't think you can say in words what the data structures of thought are. These structures and representations (and algorithms) need to be written down in maths.

and I don't think that the appropriate maths is probability theory or first order logic.

[that's a short explanation of why neat AI hasn't really worked in the last 50 years]

comment by Cyan2 · 2008-03-02T20:31:16.000Z · LW(p) · GW(p)

My visualization of the man walking down the street included a number of irrelevant details: my view was from above eye level; the man was wearing a hat; he was walking towards me, but moving towards the right of me, not directly towards me; he was walking on the right side of the street (my right, his left), on a sidewalk; and he turned right (my right, his left) to go into the drugstore.

I believe my view from above was triggered by the word "down" in the description of the scenario, and the hat came from the word "old" -- the generic man-in-hat is also what I usually imagine when I read Nero Wolfe novels, which are set in the '30s through '70s.

I'm not claiming that these details a strongly fixed -- as soon as I consider alternatives to the specific details I related above, I get visualizations of those alternatives. I'm also not claiming that these visualizations are close to photographic in immediacy and sharpness -- for me, they have about the same clarity as visual memories, which is to say, not much.

comment by Brandon_Reinhart · 2008-03-02T20:42:08.000Z · LW(p) · GW(p)

Roko - "I don't think that" is not explanation.

comment by Alan_Gunn · 2008-03-02T21:42:29.000Z · LW(p) · GW(p)

I was astonished to learn years ago that some people read without "hearing" the words on the page; even today, though I know that this happens, it strikes me as odd. I even dislike reading the word "quay" because my first reaction is that it should rhyme with "way," and I know that it doesn't. Ditto with names that don't correspond to their spelling (Menzies, for instance--pronounced "mingiss" by Scots). And, perhaps relatedly, I have great difficulty visualizing anything, and never visualize anything clearly. I'm sure that there are genuine differences in visualizing ability--there are people who easily spell words backward by visualizing them and reading them off from right to left; I could no more do that than levitate. Richard Feynman somewhere described people at MIT learning to estimate the passage of time in different ways: some by counting (in their heads), others by picturing a moving tape with numbers on it.

Replies from: DilGreen
comment by DilGreen · 2010-09-30T20:43:30.683Z · LW(p) · GW(p)

I am the same - when I read, I hear my own voice speaking the words, and am also a poor visualiser of sense data in its sensory form.

The odd thing is that I am highly discriminating in terms of music, art, food (you might call it fussy, I would say I have an eye/ear for true quality...).

Even stranger is that I am an architect, and for the last twenty years I have been developing and practicing (as best I can) imagining non-existent three dimensional forms and relationships, and being very particular about them.

I do have fuzzy mental pictures, but what I really experience is more of a 'gestalt' of the character of the 'right' solution- which I then sketch and attempt to integrate into the technical/practical aspects of the design often testing back against the 'feeling' of the imagined result. I happen to think that I am quite good at making things which are considered beautiful using this method.

My dreams (of which I remember very few) are experienced as narrative - again more experientially than as pictures or words.

The interesting thing about this thread is that it would seem that there is a wide range of ways in which individuals experience the mental 'currency' brought into consciousness by any given concept or word - yet there is wide acceptance of the quality of certain types of writing. Thus individuals reading Dickens, say, will happily converse together, but are likely to be experiencing the narrative in radically different ways.

comment by Roko · 2008-03-02T22:54:31.000Z · LW(p) · GW(p)

@Brandon Reinhart:

True, true. I wish I had something more rigorous, but if I did then I would be writing a paper on it right now! All I have are some vauge intuitive ideas.

My intuition is that the key here is good knowledge representation systems. First Order Predicate Logic (FOPL) is good for something rather different than representing knowledge about the real world; it's good at representing statements about clean, abstract entities, namely about the truths of a particular formal system.

I think that my first intuition about the difference between statements of some formal language, and representations which would be useful to describe the real world is that FOPL statements don't come with a natural topology, and I think that one ought to have something like that. I think that a fairly key idea in the mathematics which underlies physics (calculus, etc) is the idea that we can approximate something. This is ultimately formalized by the metric on the real numbers, or on R^n. I seem to remember that those who invented calculus started by thinking about cutting things into very small slices, or about approximating a curve by small line segments.

I think that one needs some way of expressing the fact that two statements are conceptually close, and I don't think that something like mutual entropy or correlation of random variables does the trick. Why not? Well, think about the way a Taylor series approximates a curve (for example, the way it approximates the solution to a differential equation). You build up the representation of the (quite complex) real world thingy from a series of rather simple elements; the monomials (1, x, x^2, etc ) by taking a limit. Do we see probability theory allowing us to do this? Can I take a series of simple FOPL statements and approximate a messy, real world thing like a "fuzzy category" with them? I don't think so. Well, at least I've read some AI books and they all seem to limit themselves to encoding a real world concept as a particular FOPL statement, e.g.

Apple(X) <==> [ Green(X) or Red(X) ] and Edible(X) and Size(X, medium), etc.

This, I think, is the equivalent of thinking that the motion of a particle is ALWAYS some actual polynomial. In general, the motion of a particle is given by the solution to some system of PDEs or ODEs, which (by various theorems of analysis) (under nice circumstances) have a solution (on some open subset of R or R^n) which you can ultimately approximate by a polynomial.

I think that we're missing a whole chunk of theory about how you can iteratively build complex representations from simple ones, and how you can take limits of conceptual representations, and then how you can manipulate those limits.

comment by Latanius2 · 2008-03-02T23:41:43.000Z · LW(p) · GW(p)

Are words really just pointers? If you want to refer to objects which you've visualized, they indeed are. But people even do some peculiar "arithmetic" with words, forming sentences, which has nothing to do with meanings.

For example, when I'm sleepy (half sleeping state), sometimes I notice that whole sentence structures are running through my head, without the words filled in, but I know where the sentences begin and end, and how they are connected. Even specific words show up time to time, but the whole stream has no sense at all. But if you don't visualize and use concepts... it just sounds right.

So I think words as stand-alone things (with their sound and syntactical role) also have an important role for connecting those things in a more abstract way, whose connections can't be inferred only by visualization. (Think of a linked list of pointers: the position of the pointers in the list can be as important as the referenced object itself.)

Roko: FOPL is similar to the taxi driver who never visualizes anything. (It never dereferences the pointers.) I don't think the solution would be a much better symbolic system (although FOPL is not really designed for dereferencig), but to connect a visual cortex to the symbol manipulation system. So the similarity of two symbols could be checked by simply visualizing them.

comment by Ron_Hardin · 2008-03-03T00:01:17.000Z · LW(p) · GW(p)

Apple(X) <==> [ Green(X) or Red(X) ] and Edible(X) and Size(X, medium), etc.

The criteria for ordinary language making something count, or fit the case, are ordinary language criteria, not mathematical criteria, of counting or fitting.

That is, ordinary language rules the operation of ordinary language, using the ordinary meanings of count and fit, not the mathematical ones.

Ordinary concepts (nice red apple) are not less precise than mathematical concepts ; but they give precision a certain shape.

The philosopher (not the mathematician!) wants to say that ordinary langauge lacks something that mathematics has. The philosopher however is not curious about why he thinks this.

comment by michael_vassar3 · 2008-03-03T03:06:25.000Z · LW(p) · GW(p)

Field's Medalist insists that people don't visualize. http://gowers.wordpress.com/2007/09/13/how-should-logarithms-be-taught/#more-5

The debate apparently goes on in educated circles to this day, and is compatible with high mathematical ability.

comment by michael_vassar3 · 2008-03-03T03:19:28.000Z · LW(p) · GW(p)

By the way Alan, it is possible to learn to read without hearing the words, and once you learn to do so you will probably find that you can read much faster that way. This won't help you to digest complex material faster, but will be useful when reading an article on a subject that you already know in hopes of finding some new and interesting information.

comment by Yelsgib · 2008-03-03T10:11:05.000Z · LW(p) · GW(p)

Word as "pointer" implies the requirement for infinite storage - unless you say that "the word duck" is not a symbol referring to the symbol "duck" - or unless you claim that we can generate static memory spontaneously - or unless you believe that there is some special class or symbolized objects (like, you're -really- storing "duck" (content) somewhere but you are not storing "duck" (symbol) or "symbl duck" (symbol) somewhere).

Words, content, pointers, blah - it's all just computation until you can prove otherwise.

Do not presume structure.

comment by Alan_Gunn · 2008-03-03T13:34:49.000Z · LW(p) · GW(p)

Just a quick response to Michael Vassar: I am a very fast reader--just about the fastest I know. And I very much doubt that I could, at my advanced age, learn to read without hearing. Anyway, why would I want to? Among other things, I suspect that those who don't hear the words they read don't enjoy poetry as much as I do. What interests me about all this is that it seems to me to show that people's mental processes differ a lot more than we usually think--a topic that psychology doesn't seem to have paid a lot of attention to, and if the psychologists don't look into it, who will? (I don't know much about psychology, though; maybe my last point is wrong--hope so.)

Replies from: PetjaY
comment by PetjaY · 2015-05-17T14:42:48.228Z · LW(p) · GW(p)

Until poetry everything you wrote could´ve come from me, but then i´ve never seen any beauty in poetry. So using same brain structures for hearing and reading, and enjoying poetry don´t always correlate

comment by Caledonian2 · 2008-03-03T13:55:58.000Z · LW(p) · GW(p)

I have a sneaking suspicion that most human talents come not from having mental resources that others do not, but by taking neural systems and tying them into different kinds of relationships with other modules.

I half suspect that the sections of my brain that are supposed to handle mathematics are devoted to processing language instead, for example.

Remarkable how many people confuse "what my mind does" with "what minds do". Perhaps this is related to the ubiquitous fallacy that mathematics has a Platonic existence beyond other phenomena and our conclusions are therefore universally binding.

comment by Roko · 2008-03-03T14:17:19.000Z · LW(p) · GW(p)

@ Latanius: "FOPL is similar to the taxi driver who never visualizes anything. (It never dereferences the pointers.) I don't think the solution would be a much better symbolic system (although FOPL is not really designed for dereferencig), but to connect a visual cortex to the symbol manipulation system. So the similarity of two symbols could be checked by simply visualizing them."

Ok, and how do you visualize the concept "technology advances exponentially because technology feeds back positively on itself" or the concept "you can't define a word any way you like because the definitions you choose will bias your thinking in subtle ways", or even the concept of "status quo bias"

Not all concepts are amenable to useful analysis by the visual system. In fact, most of the concepts that are most useful (the general, abstract ones) are not amenable to this sort of analysis.

Futhermore, anyone who criticizes "symbolic" thinking would do well to read Marvin Minsky's book, "The Emotion Machine", in which he carefully explains why symbols are more useful then, say, connection strengths in a neural net, or (as you seem to be arguing for) pixel values in a bitmap image.

comment by Roko · 2008-03-03T14:21:01.000Z · LW(p) · GW(p)

@Ron: Can you program your notion of "ordinary language criteria" into a machine? If so, you may well be using "mathematical" criteria to do so; this would be the case if you were using a programming language like c++ or prolog to write your program.

If you can't program your notion of "ordinary language criteria" into a machine, then I'm not really interested.

comment by Zubon · 2008-03-03T14:24:41.000Z · LW(p) · GW(p)

I even dislike reading the word "quay" because my first reaction is that it should rhyme with "way," and I know that it doesn't.

Easily the most traumatic thing I have learned all day. I have been mentally pronouncing it "kway" for my entire life. I cannot recall having ever heard the word aloud. I reject your pronunciation and substitute my own! (The life of a reader is filled with many such traumas. Rendezvous? Epitome?)

comment by Ben_Jones · 2008-03-03T15:17:09.000Z · LW(p) · GW(p)

So, What does a word point to?

The same as anything else that enters our heads.

I look at a tiger. The actual tiger is not now in my brain. An image is represented and decoded in my brain. The relevant sensory cortex, cross-referencing with my long-term memory, flags up the concept 'tiger' and I'm done.

I look at the printed word 'tiger'. The resulting image is now represented and decoded in my brain. The relevant sensory cortex, cross-referencing with my long-term memory, flags up the concept 'tiger' and I'm done.

Same process when I hear "Tiger!", just via a different cortex. None of these things-that-enter-my-head is a tiger.

There is no hard, fast boundary between, say, a photorealistic image and a printed word. Neither of them is the referent. They're both representations. Hieroglyphics are the missing link, if you like. Granted, a word may have nothing objectively to do with what it refers to, but a smiley :) looks nothing like a face, and we have no trouble decoding that. Why? Because we share common knowledge of that meme/word before we look. Nothing magical or different about words. Just a slightly different mental process of decoding before the cross-referencing and concept-flagging.

Language may well be hard-coded into the human brain, but to some extent this is because 'language' is how we experince the world: via signifiers and representations (yes, even when looking). The 'language of abstract symbols' (i.e. words) isn't a separate magisterium!

Try for yourself. Taboo 'word' back to 'information that enters my mind through medium x'. Then see if changing x fundamentally changes what happens in your brain.

Replies from: Armok_GoB, AspiringRationalist
comment by Armok_GoB · 2011-02-01T17:14:27.865Z · LW(p) · GW(p)

If the actual tiger were to enter your head, you'd probably not survive. It is also unlikely the entire tiger would fit.

comment by NoSignalNoNoise (AspiringRationalist) · 2013-05-06T03:49:32.000Z · LW(p) · GW(p)

Language may well be hard-coded into the human brain, but to some extent this is because 'language' is how we experince the world

Beware the typical mind fallacy. Other people may well experience it very differently.

comment by Acheman2 · 2008-03-03T16:49:40.000Z · LW(p) · GW(p)

Much of this discussion seems to be taking place within a somewhat naive conception of language and representation; in particular, it seems to me to be neglecting the insights that structuralism provided, overapplied though they may at some points have been. Contemporary linguistics, and much though sadly not all contemporary philosophy, recognise that language and thought operate both through the binding of words to images and then to 'things', and through distinguishing things and locating them within articulated systems. You can't have pointing, to give the example of a seemingly basic form of 'reference', without a system that distinguishes pointing-gestures from other motions of the hand, that defines the range of motions that count as pointing-gestures etc. I'm not going to get into a comment war about it, because there simply isn't space here, nor do I have the time, to have the sort of sustained argument that would be necessary to 'prove' this to someone used to thinking about language in another way - I can only recommmend more reading. Wittgenstein of course discusses this in the 'Philosophical Investigations', but - and I know I'm going to lose 90% of you when I say this, yet here you are anyway: one of the most lucid articulations of this double aspect of cognition occurs within psychoanalytic theory. I'm reading a very interesting book, 'Freud as Philosopher' by Richard Boothby, at the moment which I'm finding to be a particularly good treatment of this. To Eliezer in particular, I can't recommend this strongly enough - he provides a reading of Freud that's very different to the 'folk' one, and so far I particularly like his discussion of the relation of gestalt perception to the experience of mental imagery. Since we're discussing our personal experiences, it came as a real surprise to me when I realised that some people couldn't hear the rhythm of words in their heads - having to count out the syllables of an iambic pentameter, for example. Another dimension I think it's interesting to consider here is imagining abstract concepts kinaesthetically. When I visualise the concept 'status quo bias', to use Roko's example, I imagine being subject to a force - a little like gravity, but stronger, maybe something like the feeling of holding a strong magnet away from something it's attracted to. I'm sure this is a heuristic that helps me to think more fluidly and intuitively about the concept.

comment by Alan_Gunn · 2008-03-03T17:41:13.000Z · LW(p) · GW(p)

Zubon: "(The life of a reader is filled with many such traumas. Rendezvous? Epitome?)"

I call your "epitome" and raise you a Yosemite (first encountered in Bugs Bunny comics; I thought for years it was "YOSE-mite"). Furrin words like rendezvous are OK, though.

comment by Joseph_Knecht · 2008-03-04T00:23:39.000Z · LW(p) · GW(p)

I personally saw the man walking away from me on the left side of the street, and my persective was just to the left of the curb on that side of the street and slightly higher than the man, who was a short distance from me. I saw him turn left into a drugstore for a split second, and then when I realized the joke briefly saw him morph into a drugstore on the sidewalk.

To the people who say that they visualized the scene but, for example, didn't see the person walking towards or away from you, or didn't see the man on one side of the street or the other: how can you visualize a man walking down a street and not at the same time have him be on one side of the street or the other (or in it), and was he not either facing you or facing away from you in the image? If not, was it that the man was vague and so didn't have features like a face that would indicate direction? I'm still puzzled how he could not have a position relative to the street in an image that includes both a person and a street.

comment by Caledonian2 · 2008-03-04T00:45:09.000Z · LW(p) · GW(p)

Easy: it wasn't an image, and we didn't visualize it.

Thus: no position, no perspective, no direction, no facing. No features, either. Just a man walking down a street and turning into a drugstore.

comment by Joseph_Knecht · 2008-03-04T01:01:08.000Z · LW(p) · GW(p)

Caledonian: you said "Your visualizations include such details? As the description didn't include such details, they're necessarily undefined - so why did you define them out of their uncertainty?"

I understood from your statement that you expressed surprise that the reported visualization contained such details as "which side of the street the person is walking down". This implied to me that you believe it is possible to visualize a man walking down a street, but not be either walking down the left or right side or in the street itself, etc.

comment by Caledonian2 · 2008-03-04T02:20:13.000Z · LW(p) · GW(p)

No, not really. I was just surprised that people attempted visual-style representations, complete with details, without having sufficient data for even an approximation.

If I'm specifically asked to visualize a triangle, I can do so - and then tell you how it's oriented, what color it is, and what color the background is. None of those things are really implied, and they're not necessary either. But if I just hear a triangle mentioned, I perceive none of those things. The thought that someone would is strange to me.

comment by Cyan2 · 2008-03-04T04:07:16.000Z · LW(p) · GW(p)

It's not like I have a choice, dude.

comment by Latanius2 · 2008-03-04T11:24:54.000Z · LW(p) · GW(p)

@Roko: The visual cortex isn't the only one thing we use. Other parts of the brain probably "cache" some of the insights gained by visualizing things, or trying / imagining movements etc., also common sentences, so we can use these areas for other things we've never seen before. These cached things are our concepts, I think.

You're right, I won't visualize every part of the thought "technology advances exponentially because technology feeds back positively on itself". But I've seen a lot of exponential functions in math classes, plotted them on screen, and noticed that they can grow very big. Now I use this concept for understanding this sentence. It would be hard to explain this to a five year old, or to somebody who has never seen exponential functions: you can't visualize so many things at once, without using any cache mechanisms. (That's why inferential distances are so long in reality, I think.)

With only language and the cached thoughts (grammar / logic and rules in a symbolic system) we can get surprisingly far, but not far enough. (For us, even logic is a cached thought from the visual cortex, for it describes the connections of distinct things. This is a special feature of vision: try to imagine two songs at the same time...)

comment by Roko · 2008-03-04T13:35:33.000Z · LW(p) · GW(p)

@latanius:

sure. But what I'm saying is that you need some mathematical machinery to manage all of these interactions between pictures, sentences, sounds, etc. There's no shortage of narrow AIs which make use of one aspect of what you've said, e.g. face recognition "AIs", text-matching AIs (google), etc. But none of them can pass the Turing test.

Furthermore, I think that there are abstract concepts (like "bias") which are not well represented by anything we have at the moment. I think that we need to forge new abstract representations, and that those representations will be symbolic, they will look more like FOPL than like a neural net or a bitmap.

comment by AndyCossyleon · 2010-08-05T18:57:47.426Z · LW(p) · GW(p)

I wonder how the topic of audio dramas could inform this discussion. I was raised listening to Focus on the Family's Adventures in Odyssey, a radio drama that began in the 80s and continues to the present, I believe. Our family had tons of episodes on cassettes, and we would listen to them as we played with legos and such.

While listening to it, I would always visualize the location, where streets are, how big a room is, where the door is, sometimes even the cardinal directions. But to this day I have no idea what the characters looked like. I could never imagine their faces, though I could do a good job of imagining their height, hair color, glasses, etc. I have no visualization of their faces, whereas I do for most other things. But I'm rambling.

I wonder how this and also being read to aloud would be experienced by others, especially by Caledonian2.

comment by Elizabeth · 2010-10-17T19:11:31.010Z · LW(p) · GW(p)

Much of this discussion seems to be people expressing differences in their own internal processes when it comes to visualization, reading, etc. This seems to me to be directly connected with a concept it took me many years to learn and which still feels unnatural to me: not all people think the same way. Even if one assumes that people all have identical brains at birth (which is not true, but useful for the sake of this argument) our brains start with a vast number of connections, and then as we age and gain experience those connections are pruned, so that we retain only the connections we use. The parts of our brain we use to process specific types of data would then be reinforced to process that data in the future, and our thinking would diverge.

This is a topic particularly prone to argument because whether or not other people think the same way we do is a difficult concept to think about, but the comments on this article are strong evidence that people at least perceive their mental processes differently. I cannot think of a good experiment to validate our different internal perceptions, but I think it would be overhasty to try to denounce other people's perceptions of how they process certain data such as visualizations.

I personally cannot picture my mother's face. I cannot picture my childhood bedroom. I can describe them in my head with words, but that is all. I can, however, rearrange the furniture in a room in my head and be sure that it will fit, or perceive the three dimensional shape described by a calculus problem. It feels to me like proprioception. I cannot say that with certainty, as I don't have a test for it, but the evidence of my own perceptions, and the evidence of, for example, the visualize-a-tiger experiment (in which some people had visualized it down to the stripes and others were taken aback by the question), both contribute to my belief that the parts of my brain I use to perceive certain data are not necessarily the same as yours, and that different mental abilities, such as visualization, are found at different strengths in different people.

Replies from: David_Gerard
comment by David_Gerard · 2011-01-13T16:53:57.904Z · LW(p) · GW(p)

Yep - a.k.a. the typical mind fallacy.

comment by Benevolence · 2012-07-12T06:41:59.696Z · LW(p) · GW(p)

This is probably one the most thought-provoking comments section i've read on this site so far.

I never really thought how different people's visualizing (or lack thereof) could be. Specifically, I never thought some people couldn't visualize at all. I always kind of assumed that people visualized fairly similarly to me. Looking back, this was a naive and selfish view, but still, so much difference...

For example, I saw the man walking on the left side of the street. I was standing in the middle of the sidewalk, at roughly my real-life height. the man was shorter than me (im 190cm, this is usually a safe bet). The man was elderly and heavily hunched, using a cane as he walked away from me (he was moving slowly). he was wearing a faded brown suit and hat you might have seen a senior wear in the 60's or 70's (funny considering im 22, and was nowhere near alive in that time-frame. I blame movies for this classic small-town scene). The sidewalk had small trees with short metal fence-cages around them (whatever those things are called) by the road spaced evenly every couple buildings apart. there were cars parked (no details on the cars other then that they were all sedan-sized). The street was deserted other than the man. it was approximately 5-6pm in summer by the sunlight and warmth (i don't know how temperature sneaked in there). the man turns left into a drugstore. he did not open the door, it swung open for him by itself somehow (though there was a faint metallic bell-sound when the door opened, as you often hear when walking into a small store). I didn't see him go in. the scene ended before he passed out of vision behind the door (it opened on the near side from him).

I have no idea how someone could possible imagine a scene like this without actually knowing the mans orientation, or personal position. I literally cannot conceive this scene without being a part of it, and having that level of detail. maybe this is why I enjoy novels and gaming so much. Specifically fantasy novels and games. I feel like I'm literally a part of another world and living the adventure.

Granted, thinking further on my "vision" some very interesting things come to mind. The man walks slowly and in real life his walking should have taken 30-40 seconds. However in my scene it happened in a split second. I have no idea how my brain managed to compress time. and it doesn't feel like the scene jumped. it felt normal. Also there was no one else on the street that i visualized, but i somehow knew it was a semi-busy small town street, and there were some people walking around, even though i could not see them.

Its actually mildly unnerving how i can visualize the scene with no other people there, and yet be absolutely certain that the scene i visualized had many other people too. And even more unnerving how I managed to watch 30-40 secs of footage in normal speed and have it happen in 1-2 secs. Anyone know how these things happen in the brain? I'd love to know more about these phenomena.

comment by A1987dM (army1987) · 2013-09-10T16:20:35.427Z · LW(p) · GW(p)

When the phrase "triangular lightbulb" first crossed my mind—no, the hotel doesn't have them—then as best as my introspection could determine, I first saw a pyramidal lightbulb with sharp edges, then (almost immediately) the edges were smoothed, and then my mind generated a loop of flourescent bulb in the shape of a smooth triangle as an alternative.

Same happened to me, except it was a prism rather than a pyramid.

comment by Mister_Tulip · 2014-02-27T04:06:50.731Z · LW(p) · GW(p)

Hmm. That was interesting. When I tried to visualize a triangular lightbulb, my initial response was a pyramidal one, but after a second or so, it shifted to a null response, where it stayed for a while. Once I started analyzing why I couldn't seem to visualize one, though, it started mapping to my video-game-granted ability to visualize 2D leaves, and after a few minutes, I actually became able to visualize it again.

comment by Дмитрий Зеленский (dmitrii-zelenskii) · 2019-08-20T18:49:15.368Z · LW(p) · GW(p)

"Believe it or not, for some decades, there was a serious debate about whether people really had mental images in their mind—an actual picture of a chair somewhere—or if people just naively thought they had mental images (having been misled by "introspection", a very bad forbidden activity), while actually just having a little "chair" label, like a LISP token, active in their brain" - AFAIK, you misrepresent the debate. It was rather about what is primary and what is secondary. Sure, your brain paints a chair - but does it first paint the chair and then search for its properties, or is the mental image merely one of the properties of a pre-found concept (the correspondence to the options you represent is _in the order given_)? Not _that_ silly, is it? (It still has a right answer, but it happens to be the second one not the first one.)

comment by [deleted] · 2019-12-25T10:56:58.381Z · LW(p) · GW(p)

As an aphantasiac myself, this article picks the wrong example. And it did not age well given that aphantasia has been experimentally confirmed. Hard to take seriously advice which would recommend rejecting true propositions.

Replies from: jeronimo196
comment by jeronimo196 · 2020-05-06T09:32:18.196Z · LW(p) · GW(p)

The article is asserting that there are people who can construct mental images, not that all people can construct mental images.