Many methods of causal inference try to identify a "safe" subset of variation 2021-03-30T14:36:10.496Z
On the Boxing of AIs 2015-03-31T21:58:08.749Z
The Hardcore AI Box Experiment 2015-03-30T18:35:19.385Z
Boxing an AI? 2015-03-27T14:06:19.281Z


Comment by tailcalled on Pathways: Google's AGI · 2021-09-25T07:22:23.569Z · LW · GW

As I understand it, Google's proposed model is a MoE model, and I've heard MoE models achieve poorer understanding for equivalent parameter count than classical transformer models do.

Comment by tailcalled on What is the evidence on the Church-Turing Thesis? · 2021-09-19T13:26:47.293Z · LW · GW

Turing machines are a finite state machine that have access to a memory tape. This was intended to be sort of analogous to humans being able to take notes on unbounded amounts of paper when thinking.

Comment by tailcalled on Jitters No Evidence of Stupidity in RL · 2021-09-17T20:34:57.867Z · LW · GW

I feel like I once saw RL agents trained with and without energy costs, where the agents trained with energy costs acted a lot less jittery. But I can't remember where I saw it. 

Comment by tailcalled on Reflection in Probabilistic Logic · 2021-09-10T20:42:05.015Z · LW · GW

Did anything happen with the unrestricted comprehension?

Comment by tailcalled on LessWrong is providing feedback and proofreading on drafts as a service · 2021-09-07T18:55:53.442Z · LW · GW

I'm trying to get started on making a rationalist blog (here), and I've posted this to LessWrong before. I have several half-finished drafts for the blog; if I ever finish up those drafts, would it be bad form to ask about feedback from LessWrong, since it's not strictly speaking part of LessWrong? Or would it be fine, since I'm planning on posting it to LessWrong and partly (though not wholly) have a rationalist audience in mind?

Comment by tailcalled on Causal without correlation, how? · 2021-08-30T17:36:50.585Z · LW · GW

Why, when is the latter? Is there one reason or more and if so how can they be structured and by what? If one of the observables does not change, because there is a controlling observer (prediction+feedback), there is no way to establish correlation.

Typically you get causality without correlation when there is some controller that manipulates the causal variable in order to control the variable that it has an effect on.

I am displeased by bayesian probability combined with graphs (DAG), it so obviously lacks the nonlinear activation function.

DAGs only encode the structural relations, they make no inherent claims that things have to be linear. A common model is to allow each node to be an arbitrary function of its parents. The reason this isn't used much in practice, even though this is what the math is based on, is that it is usually very hard to fit.

Comment by tailcalled on Generator Systems: Coincident Constraints · 2021-08-24T16:59:22.478Z · LW · GW

I mean something like, modernity has led to improvements to a whole bunch of different things (and also worsening in some small number of other things). It doesn't seem all it would be all that surprising to me that improvements would on average have some sort of directional effect, even if a priori predicting that effect (easier to do crime vs easier to prevent crime) is hard.

Comment by tailcalled on Generator Systems: Coincident Constraints · 2021-08-24T09:01:37.272Z · LW · GW

Generally, multi-causal models are subject to a significant complexity penalty: if you think 50 different things contributed to falling crime rates in the 90s, you must also explain why all 50 different things happened at the same time. This is not true when the generator system is evolution.

What if there are just a lot of things that happen in general? In particular in modernity, where a lot of stuff has obviously changed due to technology.

Comment by tailcalled on Core Pathways of Aging · 2021-08-23T16:59:24.668Z · LW · GW

Does this blood finding change anything here?

Comment by tailcalled on Factors of mental and physical abilities - a statistical analysis · 2021-08-20T10:58:09.696Z · LW · GW

g is the thing that factor analysis gives you, and is your position along the cigar.

g is generally defined as being the underlying cause that leads to the different cognitive tests correlating. This is literally what is meant by the equation:

x = wg + e

 Factor analysis can estimate g, but it can't get the exact value of g, due to the noise factors. People sometimes say "g" when they really mean the estimate of g, but strictly speaking that is incorrect.

Factor analysis is the method of finding the best fitting ellipsoid to the data.

This is incorrect. Principal component analysis is the method for finding the best fitting ellipsoid to the data. Factor analysis and principal component analysis tend to yield very similar results, and so are usually not distinguished, but when it comes to the issue we are discussing here, it is necessary to distinguish.

There is nothing in factor analysis about underlying “real” variables that the observations are noisy measurements of, nor about values along the principal axes being noisy observations of “real” factors.

Factor analysis (not principal component analysis) makes the assumption that the data is generated based on an underlying factor + noise/specificities. This assumption may be right or wrong; factor analysis does not tell you that. But under the assumption that it is right, there is an underlying g factor that makes the variables independent (not negatively correlated).

Comment by tailcalled on Factors of mental and physical abilities - a statistical analysis · 2021-08-20T08:37:41.942Z · LW · GW

Sort of. The problem with the post is that it doesn't distinguish between g (the underlying variable that generates the cigar shape) and IQ (your position along the cigar, which will be influenced by not just g but also noise/test specificities). If you condition on IQ, everything becomes anticorrelated, but if you condition on g, they become independent.

Comment by tailcalled on Factors of mental and physical abilities - a statistical analysis · 2021-08-19T07:30:56.553Z · LW · GW

This seems to be an argument for including more variables than just g (which most psychometric models IME already do btw), but it doesn't seem to support your original claim that g doesn't exist at all.

(Also, g isn't a model of the brain.)

Comment by tailcalled on Factors of mental and physical abilities - a statistical analysis · 2021-08-18T18:11:25.552Z · LW · GW

I agree with your conclusion, though, g is not a real thing that exists.

What would your response be to my defense of g here?

(As far as I can tell, there are only three problems with the study I linked: 1. due to population structure, the true causal effects of the genes in question will be misestimated (this can be fixed with within-family studies, as was done with a similar study on Externalizing tendencies, 2. the study might lack the power to detect subtle differences between the genes in their specific degrees of influences on abilities, which if detected might 'break apart' g into multiple distinct factors, 3. the population variance in g may be overestimated when fit based on phenotypic rather than causally identified models. Of these, I think issue 2 is unlikely to be of practical importance even if it is real, while issue 1 is probably real but will gradually get fixed, and issue 3 is concerning and lacks a clear solution. But your "g is not a real thing that exists" sounds like you are more pessimistic about this than I am.)

Comment by tailcalled on Why must plausibilities be represented using real numbers? · 2021-08-18T14:12:56.929Z · LW · GW

I think so. Complex numbers and infinitesimals are IIRC the only possible alternatives to the reals, but complex numbers only apply to certain limited quantum contexts (roughly speaking, complex numbers apply when information is perfectly preserved, while real numbers apply in contexts where there's information leakage into the environment), while infinitesimals can be approximated perfectly by real numbers. So in everyday contexts (which is presumably where "common sense" applies), plausibility is captured by real numbers.

Comment by tailcalled on Factors of mental and physical abilities - a statistical analysis · 2021-08-18T14:06:32.261Z · LW · GW

I agree that a simple factor analysis does not provide anything even close to proof of 3 or 4, but I think it's worth noting that the evidence on g goes beyond the factor-analytic, e.g. with the studies I linked.

Comment by tailcalled on Factors of mental and physical abilities - a statistical analysis · 2021-08-18T13:36:33.742Z · LW · GW

Well, there's sort of a spectrum of different positions one could take with regards to the realism of g:

  1. One could argue that g is pure artifact of the method, and not relevant at all. For instance, some people argue that IQ tests just measure "how good you are at tests", argue that things like test-taking anxiety or whatever are major influences on the test scores, etc..
  2. One could argue that g is not a common underlying cause of performance on tests, but instead a convenient summary statistic; e.g. maybe one believes that different abilities are connected in a "network", such that learned skill at one ability transfers to other "nearby" abilities. In that case, the g loadings would be a measure of how central the tests are in the network.
  3. One could argue that there are indeed common causes that have widespread influence on cognitive ability, and that summing these common causes together gives you g, without necessarily committing to the notion that there is some clean biological bottleneck for those common causes.
  4. One could argue that there is a simple biological parameter which acts as a causal bottleneck representing g.

Of these, the closest position that your post came to was option 2, though unlike e.g. mutualists, you didn't commit to any one explanation for the positive manifold. That is, in your post, you wrote "It does not mean that number causes test performance to be correlated.", which I'd take to be distancing oneself from positions 3+. Meanwhile, out of these, my comment defended something inbetween options 3 and 4.

You seem to be asking me about option 4. I agree that strong versions of option 4 seem implausible, for probably similar reasons to you; it seems like there is a functional coordination of distinct factors that produce intelligence, and so you wouldn't expect strong versions of option 4 to hold.

However, it seems reasonable to me to define g as being the sum of whichever factors have an positive effect on all cognitive abilities. That is, if you have some genetic variant which makes people better at recognizing patterns, discriminating senses, more knowledgeable, etc., then one could just consider this variant to be part of g. This would lead to g being composed of a larger number of heterogeneous factors, some of which might possibly not be directly observable anymore (e.g. if they are environmental factors that can no longer be tracked); but I don't see anything wrong with that? It would still satisfy the relevant causal properties.

(Of course then there's the question of whether all of the different causes have identical proportions of effect on the abilities, or if some influence one ability more and others influence another ability more. The study I linked tested for this and only included genetic variants that had effects in proportion to the g factor loadings. But I'm not sure their tests were well-powered enough to test it exactly. If there is too much heterogeneity between the different causes, then it might make sense to split them into more homogeneous clusters of causes. But that's for future research to figure out.)

Comment by tailcalled on Factors of mental and physical abilities - a statistical analysis · 2021-08-18T11:39:13.862Z · LW · GW

This is a point that often comes up in "refutations" of the existence of g. People argue, essentially, that even though tests are correlated, they might be produced by many independent causes. I'd go further---we know there are many causes. While intelligence is strongly heritable, it's highly polygenic. Dozens of genes are already known to be linked to it, and more are likely to be discovered. It's harder to quantify environmental influences, but there are surely many that matter there, too.

So, no, there's no magical number g hidden in our brains, just like there's no single number in our bodies that says how good we are at running, balancing, or throwing stuff. But that doesn't change the fact that a single number provides a good description of how good we are at various mental tasks.

I disagree here. g can totally exist while being a wildly heterogenous mixture of different causes. As a point of comparison, consider temperature; there are many different things that can influence the temperature of an object, such as absorbing energy from or emitting energy via light/other EM waves, exothermic or endothermic chemical reactions, contact friction with a moving object, and similar. The key point is that all of these different causes of temperature variation follow the same causal rules with regards to the resulting temperature.

When it comes to the polygenic influence on g, the same pattern arises as it does for temperature; while there are many different genetic factors that influence performance in cognitive tasks, many of them do so in a uniform way, improving performance across all tasks. We could think of g as resulting from the sum of such cross-cutting influences, similar to how we might think of temperature variation as resulting from the sum of various heating and cooling influences. (Well, mathematically it's more complicated than that, but the basic point holds.)

Importantly, this notion of g is distinct from the average performance across tests (which we might call IQ). For instance, you can increase your performance on a test (IQ score) by getting practice or instruction for the test, but this doesn't "transfer" to other tasks. The lack of cross-task transfer distinguishes g from these other things, and also it is what makes g so useful (since something that makes you better at everything will... well, make you better at everything).

Comment by tailcalled on Why must plausibilities be represented using real numbers? · 2021-08-16T17:17:24.074Z · LW · GW

Complex numbers don't have an ordering, which seems counterintuitive as usually you can talk about things being more or less plausible. (Though they do get used in quantum mechanics for something that could be said to resemble plausibility, but that introduces a whole bunch of complicating factors that are not relevant here.)

When it comes to number systems with infinitesimals, you could often in principle use them, but in practice they aren't relevant because infinitesimal values will almost always be outweighted by non-infinitesimal values.

Comment by tailcalled on Social media: designed to be bad for you & for society · 2021-08-06T09:00:47.112Z · LW · GW

That second phenomenon seems to be a thing, though I wouldn't use the word "echo chamber" to refer to it. More like "polarization" or "radicalization".

Comment by tailcalled on Social media: designed to be bad for you & for society · 2021-07-26T10:19:51.956Z · LW · GW

The study you linked smells fishy to me. They found that the overwhelming majority of users were deeply ingrained in either a science community or a conspiracy community; but that doesn't match my experience on facebook, where most people just seem to share stuff from their life. Is it possible that they specifically sampled from science communities/conspiracy communities? (Which would obviously overstate the degree of polarization and echo chambers by a huge amount.) They don't seem to describe how they sampled their users, unless I'm missing something, but given the context I would guess that they specifically sampled users who were active in the communities they looked at.

Regarding the studies that said it was overstated, as I said I haven't looked into it in detail, I just follow a bunch of social science people on twitter and they've discussed this, with the people who seem more trustworthy converging towards a view that the echo chamber issue is overrated and based on misinformation. But based on your comment I decided to look at the studies more closely, and they seemed a lot less convincing than I had expected, sometimes updating me in the opposite direction. Probably the twitter user that has the most comprehensive set of links is Rolf Degen, but I've also seen it from other sources, e.g. Michael Bang Petersen.

Comment by tailcalled on Social media: designed to be bad for you & for society · 2021-07-24T21:11:34.957Z · LW · GW

Do you have a source on the claims about echo chambers? I feel like most studies I encounter on it say that echo chambers are an overrated issue, with people tending to interact with contradictory views, but I haven't looked into it in detail.

Comment by tailcalled on Chess and cheap ways to check day to day variance in cognition · 2021-07-08T07:36:16.783Z · LW · GW

It seems like it would be really interesting to study this quantitatively, recording data collected over time. If you (or anyone else) would be interested, I know a bunch of psychological research methods that could help.

Comment by tailcalled on Is there a "coherent decisions imply consistent utilities"-style argument for non-lexicographic preferences? · 2021-06-30T17:26:14.311Z · LW · GW

We were comparing epsilon to no-epsilon (what I had in mind with my post).

Anyway, the point is that strict equality would require astronomical consequences, and so only be measure 0. So outside of toy examples it would be a waste to consider lexicographic preferences or probabilities.

Comment by tailcalled on Is there a "coherent decisions imply consistent utilities"-style argument for non-lexicographic preferences? · 2021-06-30T14:10:38.816Z · LW · GW

ah, I think I am starting to follow. It is a bit ambigious whether it is supposed to be two instances of one arbitraliy small finite or two (perhaps different) arbitrarily small finites. If it is only one then the tails are again relevant.

I don't see what you mean. It doesn't make a different whether you have only one or two small-but-finite quantities (though usually you do have on for each quantity you are dealing with), as long as they are in general position. For instance, in my modification to the example you gave, I only used one: While it's true that the tail becomes relevant in (0, 1)(1, 0) vs (1, 0)(0, 1) because 0*1=1*0, it is not true that the tail becomes relevant in the slightly modified (eps, 1)(1, 0) vs (1, 0)(0, 1) for any real eps != 0, as eps*1 != 1*0.

So the only case where infinitesimal preferences are relevant are in astronomically unlikely situations where the head of the comparison exactly cancels out.

Comment by tailcalled on Is there a "coherent decisions imply consistent utilities"-style argument for non-lexicographic preferences? · 2021-06-30T13:35:49.363Z · LW · GW

I am tripping over the notation a little bit. I was representing eps times 1 as (0,1)(1,0) so (eps, 1)(1, 0) and (1, 0)(eps, 1) both evaluate to 2eps in my mind which would make the tail relevant.

Surely this would be your representation for infinitesimals, not for eps? By eps, I mean epsilon in the sense of standard real numbers, i.e. a small but positive real number. The issue is that in the real world, there will always be complexity, noise and uncertainty that will introduce small-but-positive real gaps, and these gaps will outweigh any infinitesimal preference or probability that you might have.

Comment by tailcalled on Is there a "coherent decisions imply consistent utilities"-style argument for non-lexicographic preferences? · 2021-06-30T12:13:37.276Z · LW · GW

In your example, r1m1*p1m1 is exactly equal to r2m1*p2m1; 0*1=0=1*0. The point is that if in the former case, you instead have (eps, 1)(1, 0), or in the latter case, you instead have (1, 0)(eps, 1), then immediately the tail of the sequence becomes irrelevant, because the heads differ. So only when the products of the heads are exactly equal do the tails become relevant.

However if the differences are 0.000000000000002 then you migth be more paranoid and start to actually look whether all the "assumed 0" assumptions should actually be made.

This requires you to not use infinitesimal probabilities. 0.000000000000002 is still infinitely bigger than infinitesimals.

Comment by tailcalled on Is there a "coherent decisions imply consistent utilities"-style argument for non-lexicographic preferences? · 2021-06-30T10:28:59.267Z · LW · GW

Oh, you meant literal infinitesimal (non-real) probabilities. I thought you meant very small probabilities.

The argument against lexicographic ordering is also an argument against infinitesimal probabilities. Which is to say, suppose you have some lotteries l1, l2 that you are considering choosing between (the argument can be iterated to apply to a larger number of lotteries), where each lottery ln can have payouts rn1, rn2, rn3, ... with probabilities pn1, pn2, np3, ....

Suppose that rather than being real numbers, the probabilities and payouts are ordered lexicographically, with payout m being of shape (rnm1, rnm2, rnm3, ...) and probability m being of shape (pnm1, pnm2, pnm3, ...).

In order for the infinitesimal tail of the sequence to matter, you need the sum over r1m1\*p1m1 to be exactly precisely 100% equal to the sum over r2m1\*p2m1 (that is, after all, the definition of lexicographic orderings). If they differ by even the slightest margin (even by 10^-(3^^^^^^3)), this difference outweights anything in the infinitesimal tail. While you could obviously set up such a scenario mathematically, it seems like it would never happen in reality.

So while you could in principle imagine lexicographic preferences, I don't think they would matter in reality, and a proper implementation of them would act isomorphic to or want to self-modify into something that only considers the leading term.

Comment by tailcalled on Do incoherent entities have stronger reason to become more coherent than less? · 2021-06-30T08:21:16.745Z · LW · GW

"Incoherent entities have stronger reason to become more coherent than less" is very abstract, and if you pass through the abstraction it becomes obvious that it is wrong.

This is about the agent abstraction. The idea is that we can view the behavior of a system as a choice, by considering the situation it is in, the things it "could" do (according to some cartesian frame I guess), and what it then actually does. And then we might want to know if we could say something more compact about the behavior of the system in this way.

In particular, the agent abstraction is interested in whether the systen tries to achieve some goal in the environment. It turns out that if it follows certain intuitively goal-directed rules like always taking a decision and not oscillating around, it must have some well-defined goal.

There's then the question of what it means to have a reason. Obviously for an agent, it would make sense to treat the goal and things that derive from it as being a reason. For more general systems, I guess you could consider whatever mechanism it acts by to be a reason.

So will every system have a reason to become an agent? That is, will every system regardless of mechanism of action spontaneously change itself to have a goal? And I was about to say that the answer is no, because a rock doesn't do this. But then there's the standard point about how any system can be seen as an optimizer by putting a utility of 1 on whatever it does and a utility of 0 on everything else. That's a bit trivial though and presumably not what you meant.

So the answer in practice is no. Except, then in your post you changed the question from the general "systems" to the specific "creatures":

you should predict that reasonable creatures will stop doing that if they notice that they are doing it

Creatures a produced by evolution, and those that oscillate endlessly will tend to just go extinct, perhaps by some other creature evolving to exploit them, and perhaps just by wasting energy and getting outcompeted by other creatures that don't do this.

Comment by tailcalled on Is there a "coherent decisions imply consistent utilities"-style argument for non-lexicographic preferences? · 2021-06-30T06:55:44.004Z · LW · GW

No, when there are infinitesimal chances, you take the ones which favor your leading term.

Comment by tailcalled on Is there a "coherent decisions imply consistent utilities"-style argument for non-lexicographic preferences? · 2021-06-29T19:30:49.859Z · LW · GW

I imagine you would be able to solve this by replacing real-valued utilities with utilities in a numbering system that contains infinitesimals. However, it seems to me that it would not matter much in practice, since if you had even a tiny chance of affecting the leading term, then that chance would outweigh all of the other terms in the calculation, and so in practice only the leading term would matter for the decisions.

Or to give an argument from reflection, there's a cost to making decisions, so an agent with a lexicographic ordering would probably want to self-modify into an agent that only considers the leading term in the preference ordering, since then it doesn't have to spend resources on the other terms.

Comment by tailcalled on How much interest would there be in a fringe theories wiki? · 2021-06-29T18:15:56.050Z · LW · GW

Also, if you (or someone else) arranges some site collecting adversarial collaborations, then please contact me, as there's some topics I can contribute with. Most notably, I've studied transsexuality a lot and am very familiar with the strengths and weaknesses of some controversial theories on the topic.

Comment by tailcalled on How much interest would there be in a fringe theories wiki? · 2021-06-29T17:19:36.249Z · LW · GW

I would be more interested in people arranging adversarial collaborations on it. A big problem with raw presentations of fringe theories is that then you have to go through all of the arguments in detail. Having people on both sides of the issue who are well-acquainted with the evidence point out the most important core disagreements would be better.

Comment by tailcalled on Reply to Nate Soares on Dolphins · 2021-06-10T09:03:57.033Z · LW · GW

Another tree rotation is the human vs animal distinction. Humans are animals, but sometimes one uses "animal" to refer to nonhuman animals. I wonder if there's some general things one could say about tree rotations. The human/animal distinction seems to have a different flavor to me than the fish/tetrapod distinction, though.

Something that also seems related is asking for an eggplant and expecting a fully-developed, non-rotting eggplant.

Comment by tailcalled on Reply to Nate Soares on Dolphins · 2021-06-10T08:42:09.112Z · LW · GW

I wonder to what degree the perspective comes from us generally not thinking about nonvertebrate animals. So the things that distinguish "phylogenetic fish" from them (like having craniums and vertebrae) are just considered "animal things". And so instead, when defining fish we end up focusing on what distinguishes tetrapods and taking the negation of that. Sort of a categorical "tree rotation" if you will.

Comment by tailcalled on Reply to Nate Soares on Dolphins · 2021-06-10T08:25:19.837Z · LW · GW

It's easier to keep track of the underlying relatedness as if it were an "essence" (even though patterns of physical DNA aren't metaphysical essences), rather than the all of the messy high-dimensional similarities and differences of everything you might notice about an organism.

Hmm, isn't DNA metaphysical essences?

IIRC, the metaphysical notion of essence came from noticing similarities between different creatures, that they seemed to cluster together in species as if constructed according to some blueprint. The reason for these similarities is the DNA - if "essences" had to correspond to anything in reality, then that would seem to be DNA.

Comment by tailcalled on Reply to Nate Soares on Dolphins · 2021-06-10T07:31:14.341Z · LW · GW

Am I misunderstanding something? You seem to be defending a phylogenetic definition of "fish" as a reason why dolphins aren't fish, but if you used a phylogenetic definition of "fish", you'd still have dolphins be fish - that's the first part of his argument.

Comment by tailcalled on Often, enemies really are innately evil. · 2021-06-07T14:45:15.327Z · LW · GW

(Sorry Scott Alexander, you totally fell down as a rationalist when you saw one of the biggest effect sizes ever, in a study that controlled for so many things, and still thought "hmm, i doubt being tortured regularly for a decade has a long term bad effect.")

Controlling for things isn't a good way to go about researching the effects of this. Instead, you should ask, what factors lead to variation in who gets bullied?

Comment by tailcalled on Search-in-Territory vs Search-in-Map · 2021-06-06T08:23:42.444Z · LW · GW

Recently I've also been thinking about something that seems vaguely related, which could perhaps be called inference in the map vs inference in the territory.

Suppose you want to know how some composite system works. This might be a rigid body object made up of molecules, a medicine made out of chemicals to treat a disease that is ultimately built out of chemicals, a social organisation method designed for systems made out of people, or anything like that.

In that case there are two ways you can proceed: either think about the individual components of the system and deduce from their behavior how the system will behave, or just build the system in reality and observe how the aggregate behaves.

If you do the former, you can apply already-known theory about the components to deduce it's behavior without needing to test it in reality. Though often in practice this theory won't be known, or will be too expensive to use, or similar. So in practice one generally has to investigate it holistically. But this requires using the territory as a map to figure it out.

(When investigating it holistically there is also the possibility of just using holistic rather than reductionistic theories. Often this holistic theory will originate from one of the previous methods though, e.g. our math for rigid body dynamics comes from actual experience with rigid bodies. Though also sometimes it might come from other places, e.g. evolutionary reasoning. So my dichotomy isn't quite as clean as yours, probably.)

Comment by tailcalled on The Homunculus Problem · 2021-06-03T13:20:45.601Z · LW · GW

I might not have explained the credence/propositional assertion distinction well enough. Imagine some sort of language model in AI, like GPT-3 or CLIP or whatever. For a language model, credences are its internal neuron activations and weights, while propositional assertions are the sequences of text tokens. The neuron activations and weights seem like they should definitely have a Bayesian interpretation as being beliefs, since they are optimized for accurate predictions, but this does not mean one can take the semantic meaning of the text strings at face value; the model isn't optimized to emit true text strings, but instead optimized to emit text strings that match what humans say (or if it was an RL agent, maybe text strings that make humans do what it wants, or whatever).

My proposal is, what if humans have a similar split going on? This might be obscured a bit in this context, since we're on LessWrong, which to a large degree has a goal of making propositional assertions act more like proper beliefs.  

In your model, do you think there's some sort of confused query-substitution going on, where we (at some level) confuse "is the color patch darker" with "is the square of the checkerboard darker"?

Yes, assuming I understand you correctly. It seems to me that there's at least three queries at play:

  1. Is the square on the checkerboard of a darker color?
  2. Is there a shadow that darkens these squares?
  3. Is the light emitted from this flat screen of a lower luminosity?

If I understand your question, "is the color patch darker?" maps to query 3?

The reason the illusion works is that for most people, query 3 isn't part of their model (in the sense of credences). They can deal with the list of symbols as a propositional assertion, but it doesn't map all the way into their senses. (Unless they have sufficient experience with it? I imagine artists would end up also having credences on it, due to experience with selecting colors. I've also heard that learning to see the actual visual shapes of what you're drawing, rather than the abstracted representation, is an important step in becoming an artist.)

Do the credences simply lack that distinction or something?

The existence of the illusion would seem to imply that most people's credences lack the distinction (or rather, lacks query 3, and thus finds it necessary to translate query 3 into query 2 or query 1). However, it's not fundamental to the notion of credence vs propositional assertion that it lacks this. Rather, the homunculus problem seems to involve some sort of duality, either real or confused. I'm proposing that the duality is real, but in a different way than the homunculus fallacy does, where credences act like beliefs and propositional assertions can act in many ways.

This model doesn't really make strong claims about the structure of the distinctions credences make, similar to how Bayesianism doesn't make strong claims about the structure of the prior. But that said, there must obviously be some innate element, and there also seems to be some learned element, where they make the distinctions that you have experience with.

We've seen objects move in and out of light sources a ton, so we are very experienced in the distinction between "this object has a dark color" vs "there is a shadow on this object". Meanwhile...

Wait actually, you've done some illustrations, right? I'm not sure how experienced you are with art (the illustrations you've posted to LessWrong have been sketches without photorealistic shading, if I recall correctly, but you might very well have done other stuff that I'm not aware of), so this might disprove some of my thoughts on how this works, if you have experience with shading things.

(Though in a way this is kinda peripheral to my idea... there's lots of ways that credences could work that don't match this.)

More generally, my correction to your credences/assertions model would be to point out that (in very specific ways) the assertions can end up "smarter". Specifically, I think assertions are better at making crisp distinctions and better at logical reasoning. This puts assertions in a weird position.

Yes, and propositional assertions seem more "open-ended" and separable from the people thinking of them, while credences are more embedded in the person and their viewpoint. There's a tradeoff, I'm just proposing seeing the tradeoff more as "credence-oriented individuals use propositional assertions as tools".

Comment by tailcalled on The Homunculus Problem · 2021-05-27T21:35:35.016Z · LW · GW

One model I've played around with is distinguishing two different sorts of beliefs, which for historical reasons I call "credences" and "propositional assertions". My model doesn't entirely hold water, I think, but it might be a useful starting point for inspiration for this topic.

Roughly speaking I define a "credence" to be a Bayesian belief in the naive sense. It updates according to what you perceive, and "from the inside" it just feels like the way the world is. I consider basic senses as well as aliefs to be under the "credence" label.

More specifically, in this model, your credence when looking at the picture is that there is a checkerboard with consistently colored squares, a cylinder standing on the checkboard, and casting a shadow on it, which obviously doesn't change the shade of the squares, but does make them look darker.

In contrast, in this model, I assert that abstract conscious high-level verbal beliefs aren't proper beliefs (in the Bayesian sense) at all; rather, they're "propositional assertions". They're more like a sort of verbal game or something. People learn different ways of communicating verbally with each other, and these ways to a degree constrain their learned "rules of the game" to act like proper beliefs - but in some cases they can end up acting very very different from beliefs (e.g. signalling and such).

When doing theory of mind, we learn to mostly just accept the homunculus fallacy, because socially this leads to useful tools for talking theory of mind, even if they are not very accurate. You also learn to endorse the notion that you know your credences are wrong and irrational, even though your credences are what you "really" believe; e.g. you learn to endorse a proposition that "B" has the same color as "A".

This model could probably be said to imply overly much separation of your rational mind away from the rest of your mind, in a way that is unrealistic. But it might be a useful inversion on the standard account of the situation, which engages in the homunculus fallacy?

Comment by tailcalled on Finite Factored Sets · 2021-05-23T22:55:02.522Z · LW · GW

Ah of course! So many symbols to keep track of 😅

Comment by tailcalled on Finite Factored Sets · 2021-05-23T22:41:27.825Z · LW · GW

I think one thing that confuses me is, wouldn't Y also be before X then?

Comment by tailcalled on Finite Factored Sets · 2021-05-23T22:30:28.427Z · LW · GW

Or wait, I'm dumb, that can definitely happen if X and Y are coin flips. But I feel like this doesn't add up with the other stuff, will need to read more carefully.

Comment by tailcalled on Finite Factored Sets · 2021-05-23T22:23:49.654Z · LW · GW

I'm a bit confused about how X can be independent of both Y and of (X xor Y). What would a probability distribution where this holds look like?

Comment by tailcalled on SGD's Bias · 2021-05-19T19:14:59.670Z · LW · GW

This depends on whether it can achieve perfect predictive power or not, no? What I had in mind was something like autoregressive text prediction, where there will always be some prediction errors. I would've assumed those prediction errors constantly introduce some noise into the gradients?

Comment by tailcalled on SGD's Bias · 2021-05-19T10:07:20.319Z · LW · GW

Hmm, and the -E[u(X, theta)] term would shrink during training, right? So eventually it would become more about the drift term? This makes me think of the "grokking" concept.

Comment by tailcalled on Meditations on Momentum · 2021-05-14T08:41:41.274Z · LW · GW

Psychologists have discovered the same effect in education. The longer it takes kids to learn how to read, the slower the development of their other cognitive skills and performance:

This sounds confounded by g.

Comment by tailcalled on Agency in Conway’s Game of Life · 2021-05-13T09:44:26.070Z · LW · GW

Unlike our universe, game of life is not reversible, so I don't think entropy is the key.

Comment by tailcalled on Sexual Dimorphism in Yudkowsky's Sequences, in Relation to My Gender Problems · 2021-05-06T10:03:28.675Z · LW · GW

See e.g. this.

Also in the context of health benefit associated with transition this is irrelevant because transitioning will not change your body size...

Yeah I know, I could've pointed out the body size effect to nim too.

Comment by tailcalled on Sexual Dimorphism in Yudkowsky's Sequences, in Relation to My Gender Problems · 2021-05-05T11:50:23.521Z · LW · GW

Also, bigger bodies = more cells that can end up turning into tumors and such.