Posts

Why I stopped being into basin broadness 2024-04-25T20:47:17.288Z
Blessed information, garbage information, cursed information 2024-04-18T16:56:17.370Z
Ackshually, many worlds is wrong 2024-04-11T20:23:59.416Z
[GPT-4] On the Gradual Emergence of Mechanized Intellect: A Treatise from the Year 1924 2024-04-01T19:14:02.363Z
Opinions survey 2 (with rationalism score at the end) 2024-02-17T12:03:02.410Z
Opinions survey (with rationalism score at the end) 2024-02-17T00:41:20.188Z
What are the known difficulties with this alignment approach? 2024-02-11T22:52:18.900Z
Against Nonlinear (Thing Of Things) 2024-01-18T21:40:00.369Z
Which investments for aligned-AI outcomes? 2024-01-04T13:28:57.198Z
Practically A Book Review: Appendix to "Nonlinear's Evidence: Debunking False and Misleading Claims" (ThingOfThings) 2024-01-03T17:07:13.990Z
Could there be "natural impact regularization" or "impact regularization by default"? 2023-12-01T22:01:46.062Z
Utility is not the selection target 2023-11-04T22:48:20.713Z
Contra Nora Belrose on Orthogonality Thesis Being Trivial 2023-10-07T11:47:02.401Z
What are some good language models to experiment with? 2023-09-10T18:31:50.272Z
Aumann-agreement is common 2023-08-26T20:22:03.738Z
A content analysis of the SQ-R questionnaire and a proposal for testing EQ-SQ theory 2023-08-09T13:51:02.036Z
If I showed the EQ-SQ theory's findings to be due to measurement bias, would anyone change their minds about it? 2023-07-29T19:38:13.285Z
Autogynephilia discourse is so absurdly bad on all sides 2023-07-23T13:12:07.982Z
Boundary Placement Rebellion 2023-07-20T17:40:00.190Z
Prospera-dump 2023-07-18T21:36:13.822Z
Are there any good, easy-to-understand examples of cases where statistical causal network discovery worked well in practice? 2023-07-12T22:08:59.916Z
I think Michael Bailey's dismissal of my autogynephilia questions for Scott Alexander and Aella makes very little sense 2023-07-10T17:39:26.325Z
What in your opinion is the biggest open problem in AI alignment? 2023-07-03T16:34:09.698Z
Which personality traits are real? Stress-testing the lexical hypothesis 2023-06-21T19:46:03.164Z
Book Review: Autoheterosexuality 2023-06-12T20:11:38.215Z
How accurate is data about past earth temperatures? 2023-06-09T21:29:11.852Z
[Market] Will AI xrisk seem to be handled seriously by the end of 2026? 2023-05-25T18:51:49.184Z
Horizontal vs vertical generality 2023-04-29T19:14:35.632Z
Core of AI projections from first principles: Attempt 1 2023-04-11T17:24:27.686Z
Is this true? @tyler_m_john: [If we had started using CFCs earlier, we would have ended most life on the planet] 2023-04-10T14:22:07.230Z
Is this true? paulg: [One special thing about AI risk is that people who understand AI well are more worried than people who understand it poorly] 2023-04-01T11:59:45.038Z
What does the economy do? 2023-03-24T10:49:33.251Z
Are robotics bottlenecked on hardware or software? 2023-03-21T07:26:52.896Z
What problems do African-Americans face? An initial investigation using Standpoint Epistemology and Surveys 2023-03-12T11:42:32.614Z
What do you think is wrong with rationalist culture? 2023-03-10T13:17:28.279Z
What are MIRI's big achievements in AI alignment? 2023-03-07T21:30:58.935Z
🤔 Coordination explosion before intelligence explosion...? 2023-03-05T20:48:55.995Z
Prediction market: Will John Wentworth's Gears of Aging series hold up in 2033? 2023-02-25T20:15:11.535Z
Somewhat against "just update all the way" 2023-02-19T10:49:20.604Z
Latent variables for prediction markets: motivation, technical guide, and design considerations 2023-02-12T17:54:33.045Z
How many of these jobs will have a 15% or more drop in employment plausibly attributable to AI by 2031? 2023-02-12T15:40:02.999Z
Do IQ tests measure intelligence? - A prediction market on my future beliefs about the topic 2023-02-04T11:19:29.163Z
What is a disagreement you have around AI safety? 2023-01-12T16:58:10.479Z
Latent variable prediction markets mockup + designer request 2023-01-08T22:18:36.050Z
Where do you get your capabilities from? 2022-12-29T11:39:05.449Z
Price's equation for neural networks 2022-12-21T13:09:16.527Z
Will Manifold Markets/Metaculus have built-in support for reflective latent variables by 2025? 2022-12-10T13:55:18.604Z
How difficult is it for countries to change their school curriculum? 2022-12-03T21:44:56.830Z
Is school good or bad? 2022-12-03T13:14:22.737Z
Is there some reason LLMs haven't seen broader use? 2022-11-16T20:04:48.473Z

Comments

Comment by tailcalled on The first future and the best future · 2024-04-26T07:44:25.246Z · LW · GW

If you go slower, you have more time to find desirable mechanisms. That's pretty much it I guess.

Comment by tailcalled on Losing Faith In Contrarianism · 2024-04-26T06:11:40.558Z · LW · GW

I'm convinced by the mainstream view on COVID origins and medicine.

I'm ambivalent on education - I guess if done well, it'd consistently have good effects, and that currently, it on average has good effects, but also the effect varies a lot from person to person, so simplistic quantitative reviews don't tell you much. When I did an epistemic spot check on Caplan's book, it failed terribly (it cited a supposedly-ingenious experiment that university didn't improve critical thinking, but IMO the experiment had terrible psychometrics).

I don't know enough about sleep research to disagree with Guzey on the basis of anything but priors. In general, I wouldn't update much on someone writing a big review, because often reviews include a lot of crap information.

I might have to read Jayman's rebuttal of B-W genetic IQ differences in more detail, but at first glance I'm not really convinced by it because it seems to focus on small sample sizes in unusual groups, so it's unclear how much study noise, publication bias and and sampling bias effects things. At this point I think indirect studies are getting obsolete and it's becoming more and more feasible to just directly measure the racial genetic differences in IQ.

However I also think HBDers have a fractal of bad takes surrounding this, because they deny the phenotypic null hypothesis and center non-existent abstract personality traits like "impulsivity" or "conformity" in their models.

Comment by tailcalled on Examples of Highly Counterfactual Discoveries? · 2024-04-25T19:12:27.023Z · LW · GW

Yes.

Comment by tailcalled on Examples of Highly Counterfactual Discoveries? · 2024-04-25T19:11:28.969Z · LW · GW
  • The RLCT = first-order term for in-distribution generalization error and also Bayesian learning (technically the 'Bayesian free energy').  This justifies the name of 'learning coefficient' for lambda. I emphasize that these are mathematically precise statements that have complete proofs, not conjectures or intuitions. 

Link(s) to your favorite proof(s)?

Also, do these match up with empirical results?

  • Knowing a little SLT will inoculate you against many wrong theories of deep learning that abound in the literature. I won't be going in to it but suffice to say that any paper assuming that the Fischer information metric is regular for deep neural networks or any kind of hierarchichal structure is fundamentally flawed. And you can be sure this assumption is sneaked in all over the place. For instance, this is almost always the case when people talk about Laplace approximation.

I have a cached belief that the Laplace approximation is also disproven by ensemble studies, so I don't really need SLT to inoculate me against that. I'd mainly be interested if SLT shows something beyond that.

it can be estimated at scale.

As I read the empirical formulas in this paper, they're roughly saying that a network has a high empirical learning coefficient if an ensemble of models that are slightly less trained on average have a worse loss than the network.

But then so they don't have to retrain the models from scratch, they basically take a trained model, and wiggle it around using Gaussian noise while retraining it.

This seems like a reasonable way to estimate how locally flat the loss landscape is. I guess there's a question of how much the devil is in the details; like whether you need SLT to derive an exact formula that works.


I guess I'm still not super sold on it, but on reflection that's probably partly because I don't have any immediate need for computing basin broadness. Like I find the basin broadness theory nice to have as a model, but now that I know about it, I'm not sure why I'd want/need to study it further.

There was a period where I spent a lot of time thinking about basin broadness. I guess I eventually abandoned it because I realized the basin was built out of a bunch of sigmoid functions layered on top of each other, but the generalization was really driven by the neural tangent kernel, which in turn is mostly driven by the Jacobian of the network outputs for the dataset as a function of the weights, which in turn is mostly driven by the network activations. I guess it's plausible that SLT has the best quantities if you stay within the basin broadness paradigm. 🤔

Comment by tailcalled on Examples of Highly Counterfactual Discoveries? · 2024-04-25T13:39:55.474Z · LW · GW

Newton's Universal Law of Gravitation was the first highly accurate model of things falling down that generalized beyond the earth, and it is also the second-most computationally applicable model of things falling down that we have today.

Are you saying that singular learning theory was the first highly accurate model of breadth of optima, and that it's one of the most computationally applicable ones we have?

Comment by tailcalled on Examples of Highly Counterfactual Discoveries? · 2024-04-25T06:32:35.528Z · LW · GW

I would say "the thing that contains the inheritance particles" rather than "the inheritance particle". "Particulate inheritance" is a technical term within genetics and it refers to how children don't end up precisely with the mean of their parents' traits (blending inheritance), but rather with some noise around that mean, which particulate inheritance asserts is due to the genetic influence being separated into discrete particles with the children receiving random subsets of their parent's genes. The significance of this is that under blending inheritance, the genetic variation between organisms within a species would be averaged away in a small number of generations, which would make evolution by natural selection ~impossible (as natural selection doesn't work without genetic variation).

Comment by tailcalled on Examples of Highly Counterfactual Discoveries? · 2024-04-25T06:20:40.133Z · LW · GW

Isn't singular learning theory basically just another way of talking about the breadth of optima?

Comment by tailcalled on David Udell's Shortform · 2024-04-24T17:38:34.596Z · LW · GW

The tricky part is, on the margin I would probably use various shortcuts, and it's not clear where those shortcuts end short of just getting knowledge beamed into my head.

I already use LLMs to tell me facts, explain things I'm unfamiliar with, handle tedious calculations/coding, generate simulated data/brainstorming and summarize things. Not much, because LLMs are pretty bad, but I do use them for this and I would use them more on the margin.

Comment by tailcalled on Subjective Questions Require Subjective information · 2024-04-24T12:18:18.858Z · LW · GW

isn't a reference frame; rather, if is a world then aka are the reference frames for .

Essentially when dealing with generalized reference frames that contain answers to questions such as "who are you?", the possible reference frames are going to depend on the world (because you can only be a real person, and which real people there are depends on what the world is). As such, "reference frames" don't make sense in isolation, rather one needs a (world, reference frame) pair, which is what I call an "interpretation".

Comment by tailcalled on Subjective Questions Require Subjective information · 2024-04-23T13:35:40.021Z · LW · GW

An idea I've been playing with recently:

Suppose you have some "objective world" space . Then in order to talk about subjective questions, you need a reference frame, which we could think of as the members of a fiber of some function , for some "interpretation space" .

The interpretations themselves might abstract to some "latent space" according to a function . Functions of would then be "subjective" (depending on the interpretation they arise from), yet still potentially meaningfully constrained, based on . In particular if some structure in lifts homomorphically up through and down through , you get exactly the same structure in . (And these obviously compose nicely since they're just spans, so far.)

The key question is what kind of space/algebra to preserve. I can find lots of structures that work well for particular abstractions, but it seems like the theory would have to be developed separately for each type of structure, as I don't see any overarching one.

Comment by tailcalled on Blessed information, garbage information, cursed information · 2024-04-22T18:18:10.275Z · LW · GW

Maybe "barren information"?

Comment by tailcalled on Blessed information, garbage information, cursed information · 2024-04-22T18:07:57.643Z · LW · GW

On the one hand, I do see your point that in some cases it's important not to make people think I'm referring to malfunctioning sensors. On the other hand, malfunctioning sensors would be an instance of the kind of thing I'm talking about, in the sense that information from a malfunctioning sensor is ~useless for real-world tasks (unless you don't realize it's malfunctioning, in which case it might be cursed).

I'll think about alternative terms that clarify this.

Comment by tailcalled on tailcalled's Shortform · 2024-04-22T11:15:34.426Z · LW · GW

But the way to resolve definitional questions is to come up with definitions that make it easier to find general rules about what happens. This illustrates one way one can do that, by picking edge-cases so they scale nicely with rules that occur in normal cases. (Another example would be 1 as not a prime number.)

Comment by tailcalled on Why Would Belief-States Have A Fractal Structure, And Why Would That Matter For Interpretability? An Explainer · 2024-04-22T09:52:00.380Z · LW · GW

I guess to expand:

If you use a Markov chain to transduce another Markov chain, the belief state geometry should kind of resemble a tensor of the two Markov chains, but taking some dependencies into account.

However, let's return to the case of tensoring two independent variables. If the neural network is asked to learn that, it will presumably shortcut by representing them as a direct sum.

Due to the dependencies, the direct sum representation doesn't work if you are transducing it, and arguably ideally we'd like something like a tensor. But in practice, there may be a shortcut between the two, where the neural network learns some compressed representation that mixes the transducer and the base together.

(A useful mental picture for understanding why I care about this: take the base to be "the real world" and the transducer to be some person recording data from the real world into text. Understanding how the base and the transducer relate to the learned representation of the transduction tells you something about how much the neural network is learning the actual world.)

Comment by tailcalled on "Justice, Cherryl." · 2024-04-21T14:47:05.115Z · LW · GW

You're wrong about the dynamic portrayed in The Fountainhead. I suspect you might also be wrong about the dynamic portrayed in the other of Ayn Rand's books, though I don't know for sure as I haven't read them.

Intuitively, you'd measure altruism by the sum of one's contributions over all the people one is contributing to. In practice, you could measure this by wealth (which'd seem sensible because people pay for what they want), unique regard for the poor and weak (also would seem sensible because the poor and weak have less resources to communicate their needs) and reputation (also would seem sensible because of Aumann's agreement theorem). But then Ayn Rand shows conditions where these seemingly-sensible measurements fail catastrophically.

Consider, for instance, the case of Gail Wynand; he sought wealth, thinking it would grant him power. But because economic inequality was relatively low, the only way to earn wealth is to appeal to a lot of people, i.e. he had to be high in the preference ranking obtained by summing many different people's preference rankings together.

If you sum together a bunch of variables, the resulting scores will become an extremely accurate measure of the general factor(s) which contribute positively to all of these variables. Because it is the general factor shared across common people, you could abstract this as "the common man". In seeking wealth, one becomes forced to follow this general factor, as Gail Wynand did.

Now, what is this general "the common man" factor? It's people's opinions, based on whatever they care about, from sex to justice to whatever. But notably, nobody has time to get properly informed about everything, and a lot of people don't have the skills to make sound judgments on things, so the the general factor underlying people's opinions consists of the most superficial, ignorant positions people could take.

Gail Wynand sought wealth because he thought it would grant him power to escape the control of the common man. However, the only way to gain and keep wealth was to submit absolutely to the common man's judgement, as he found out the hard way.

Now let's go back to Peter Keating. A major theme of the book is that there were two things he was torn between seeking; his own desires (especially Catherine Halsey but also to some extent his own passions in painting and in architecture) versus his reputation (among strangers, though the book shows how his mother kept particular tabs on it).

The issue Peter Keating faced wasn't that he liked honestly-earned wealth and disliked dishonestly-earned wealth. It was partly the same as that of Gail Wynand - that reputation is based on people's most superficial, ignorant judgments. But it was deeper than that, because Gail Wynand had an awareness that people's judgments were trash and mostly let them judge him as being trash, whereas Peter Keating respected their judgments and tried to change himself to fit their judgments.

I think this may have lead him to practice dissociating from his own desires and instead go with other's judgement. Though I don't remember reading much of this process in the story, perhaps because it was already far along in the beginning of the story (as shown with his relationship with his mother). Either way, we see how despite liking Catherine Halsey, he ends up getting married to Dominique Francon (who he's also attracted to, but differently), under the assumption that this is better for his reputation.

But the problem was, there was an endless line of men aiming to impress Dominique Francon by deferring to her desires and by building a strong reputation. If Dominique really wanted that, then Peter Keating would have had much tougher competition than he really had. But instead what Dominique wanted was someone who had his own strength in judgment, and while Peter Keating showed some potential in that (at least in recognizing the social game and standing up to her jabs), he eventually degenerated fully into just deferring to social judgments, and Dominique lost interest in him.

And ultimately this was also the way Peter Keating lost everything else. In competing for bending to other's judgment, there were younger people with less historical baggage who could do so better. He wasn't particularly good at anything, and so he ended up supporting and building his life around an ideology which said that being good is evil, such that his lack of goodness became a virtue.

That's the distinction between Rand's heroes and villains. The heroes want to get rich by means of doing genuinely good work that other people will have a genuine self-interest in paying for. The villains want to wield power by means of psychological manipulation, guilt-tripping and blackmailing the people who can do good work into serving their own parasites and destroyers.

One important thing to notice is that Howard Roark used some means that would be considered quite immoral by ordinary standards. He blew up the Cortlandt, he raped Dominique Francon (sort of, it seems to me that Ayn Rand has a strange understanding of rape, but that's beside the point), he poached Austen Heller from John Erik Snyte, and he often socially put people in situations he knew they could not handle.

The difference between Howard Roark and Peter Keating isn't that Howard Roark wants to use good means and Peter Keating wants to use scummy means. The difference is that Howard Roark has a form of contribution that he cares about making and which he himself is able to judge the quality of, whereas Peter Keating doesn't chase much object-level preference himself, but instead tries to be good as per other's judgment.

Something people have occasionally noticed about my intellectual style is that I like to win arguments. I take pride and pleasure in pointing out flaws in other people's work in the anticipation of the audience appreciating how clever I am for finding the hole in someone's reasoning.

...

Overall, when I look at the world of discourse I see, the moral I draw is not that that collaborative truth-seeking is bad.

It's that collaborative truth-seeking doesn't exist. The people claiming to be collaborative truth-seekers are lying. Given that everyone wants to be seen as right, the question is: are you going to try to be seen as right by means of providing valid evidence and reasoning, or by—other means?

Or to put it another way: the commenter who admits they care for status is all right. They're usually worth the status they earn. They won't lie for it. They don't have to. But watch out for the commenter who yells too loudly how much they scorn status. Watch out particularly for the one who yells that others must scorn it. They're after something much worse than status.

This seems more similar to Peter Keating than to Howard Roark. Perhaps most similar to Gail Wynand, except Gail Wynand at least had a sort of contempt for their judgment. You'd still be dancing for the judgment of people who are not really paying attention.

Comment by tailcalled on shortplav · 2024-04-21T09:28:12.162Z · LW · GW

What do you want to use the logical correlation measure for?

Comment by tailcalled on Blessed information, garbage information, cursed information · 2024-04-19T18:25:26.211Z · LW · GW

I'd say "in many contexts" in practice refers to when you are already working with relatively blessed information. It's just that while most domains are overwhelmingly filled with garbage information (e.g. if you put up a camera at a random position on the earth, what it records will be ~useless), the fact that they are so filled with garbage means that we don't naturally think of them as being "real domains".

Basically, I don't mean that blessed information is some obscure thing that you wouldn't expect to encounter, I mean that people try to work with as much blessed information as possible. Logs were sort of a special case of being unusually-garbage.

You can't distill information in advance of a bug (or anomaly, or attack) because a bug by definition is going to be breaking all of the past behavior & invariants governing normal behavior that any distillation was based on.

Depends. If the system is very buggy, there's gonna be lots of bugs to distill from. Which bring us to the second part...

The logs are for the exceptions - which are precisely what any non-end-to-end lossy compression (factor analysis or otherwise) will correctly throw out information about to compress as residuals to ignore in favor of the 'signal'.

Even if lossy compression threw out the exceptions we were interested in as being noise, that would actually still be useful as a form of outlier detection. One could just zoom in on the biggest residuals and check what was going on there.

Issue is, the logs end up containing ~all the exceptions, including exceptional user behavior and exceptional user setups and exceptionally error-throwing non-buggy code, but the logs are only useful for bugs/attacks/etc. because the former behaviors are fine and should be supported.

Comment by tailcalled on Blessed information, garbage information, cursed information · 2024-04-19T09:18:53.431Z · LW · GW

It's also putting the attribute on the wrong thing - it's not garbage data, it's data that's useful for other purposes than the one at hand.

Mostly it's not useful for anything. Like the logs contains lots of different types of information, and all the different types of information are almost always useless for all purposes, but each type of information has a small number of purpose for which a very small fraction of that information is useful.

Blessed and cursed are much worse as descriptors. In most cases there's nobody doing the blessing or cursing, and it focuses the mind on the perception/sanctity of the data, not the use of it.

This is somewhat intentional. One thing one can do with information is give it to others who would not have seen it. Here one sometimes needs to be careful to preserve and highlight the blessed information and eliminate the cursed information.

Comment by tailcalled on Why Would Belief-States Have A Fractal Structure, And Why Would That Matter For Interpretability? An Explainer · 2024-04-18T15:00:02.878Z · LW · GW

Actually I guess that's kind of trivial (the belief state geometry should be tensored together). Maybe a more interesting question is what happens if you use a Markov chain to transduce it.

Comment by tailcalled on Why Would Belief-States Have A Fractal Structure, And Why Would That Matter For Interpretability? An Explainer · 2024-04-18T13:26:01.018Z · LW · GW

One thing I'm concerned about is that this seems most likely to work for rigid structures like CNNs and RNNs, rather than dynamic structures like Transformers. Obviously the original proof of concept was done in a transformer, but it was done in a transformer that was modelling a Markov model, whereas in the general case, transformers can model non-Markov processes

Well, sort of - obviously they ultimately still have a fixed context window, but the difficulty in solving the quadratic bottleneck suggests that this context window is an important distorting factor in how Transformers work - though maybe Mamba will save us, idk.

Comment by tailcalled on Why Would Belief-States Have A Fractal Structure, And Why Would That Matter For Interpretability? An Explainer · 2024-04-18T13:08:01.388Z · LW · GW

What do the fractals look like if you tensor two independent variables together?

Comment by tailcalled on Why Would Belief-States Have A Fractal Structure, And Why Would That Matter For Interpretability? An Explainer · 2024-04-18T07:06:47.947Z · LW · GW

Market:

Reminder that making multiple similar markets is encouraged so we get different angles on it.

Comment by tailcalled on Ackshually, many worlds is wrong · 2024-04-16T16:19:06.094Z · LW · GW

with "one outcome is real" axiom

How would you formulate this axiom?

The point is that I can understand the argument for why the true stochasticity may be coherent, but I don't get why it would be better.

I find your post hard to respond to because it asks me to give my opinion on "the" mathematical model of true stochasticity, yet I argued that classical math is deterministic and the usual way you'd model true stochasticity in it is as many-worlds, which I don't think is what you mean (?).

Comment by tailcalled on Ackshually, many worlds is wrong · 2024-04-16T07:55:37.086Z · LW · GW

Like, are you saying the mathematical model of true stochasticity (with some "one thing is real" formalization) is somehow incomplete or imprecise or wrong, because mathematics is deterministic?

Which model are you talking about here?

Comment by tailcalled on Ackshually, many worlds is wrong · 2024-04-15T18:40:03.397Z · LW · GW

Another issue with "ontologising" the Wigner function is that you need some kind of idea of what those negatives "really mean". I spent some time thinking about "If the many worlds interpretation comes from ontologising the wavefunction, what comes from doing that to the Wigner function?" a few years ago. I never got anywhere.

Wouldn't it also be many worlds, just with a richer set of worlds? Because with wavefunctions, your basis has to pick between conjugate pairs of variables, so your "worlds" can't e.g. have both positions and momentums, whereas Wigner functions tensor the conjugate pairs together, so their worlds contain both positions and momentums in one.

Comment by tailcalled on Ackshually, many worlds is wrong · 2024-04-15T12:56:02.043Z · LW · GW

Neat!

I'd expect Wigner functions to be less ontologically fundamental than wavefunctions because a wavefunction into a real function in this way introduces a ton of redundant parameters, since now it's a function of phase space instead of configuration space. But they're still pretty cool.

Comment by tailcalled on Speedrun ruiner research idea · 2024-04-15T06:46:12.997Z · LW · GW

Oh wait, maybe I misunderstood what you meant by "any game". I thought you meant a single program that could detect it across all games, but it sounds much more feasible with a program that can detect it in one specific game.

Comment by tailcalled on Ackshually, many worlds is wrong · 2024-04-15T06:34:05.480Z · LW · GW

But it's a generic type; A could be anything. I had the functional programming mindset where it was to be expected that the Distribution type would be composed into more complex distributions.

Comment by tailcalled on Ackshually, many worlds is wrong · 2024-04-14T09:45:33.079Z · LW · GW

In some but not all imaginable Truly Stochastic worlds, perhaps it's like the probability distribution of the whole state of the universe, but OP's intuition-pumping example seems to be imagining a case where A is some small bit of the universe.

Oops, I guess I missed this part when reading your comment. No, I meant for A to refer to the whole configuration of the universe.

Comment by tailcalled on Speedrun ruiner research idea · 2024-04-14T06:26:12.784Z · LW · GW

The issue with this idea is that it seems pretty much impossible

Comment by tailcalled on Consequentialism is a compass, not a judge · 2024-04-13T14:46:15.317Z · LW · GW

I think your position here is approximately-optimal within the framework of consequentialism.

It's just that I worry that consequentialism itself is the reason we have problems like AI x-risk, in the sense that the thing that drives x-risk scenarios may be the theory of agency that is shared with consequentialism.

I've been working on a post - actually I'm going to temporarily add you as a co-author so you can see the draft and add comments if you're interested - where I discuss the flaws and how I think one should approach it differently. One of the major inspirations is Against responsibility, but I've sort of taken inspiration from multiple places, including critics of EA and critics of economics.

The ideal of consequentialism is essentially flawless; it's when you hand it to sex-obsessed murder monkeys as an excuse to do things that shit hits the fan.

I've come to think that isn't actually the case. E.g. while I disagree with Being nicer than clippy, it quite precisely nails how consequentialism isn't essentially flawless:

Now, of course, utilitarianism-in-theory was never, erm, actually very tolerant. Utilitarianism is actually kinda pissed about all these hobbies. For example: did you notice the way they aren't hedonium? Seriously tragic. And even setting aside the not-hedonium problem (it applies to all-the-things), I checked Jim's pleasure levels for the trashy-TV, and they're way lower than if he got into Mozart; Mary's stamp-collecting is actually a bit obsessive and out-of-balance; and Mormonism seems too confident about optimal amount of coffee. Oh noes! Can we optimize these backyards somehow? And Yudkowsky's paradigm misaligned AIs are thinking along the same lines – and they've got the nano-bots to make it happen.

Unbounded utility maximization aspires to optimize the entire world. This is pretty funky for just about any optimization criterion people can come up with, even if people are perfectly flawless in how well they follow it. There's a bunch of attempts to patch this, but none have really worked so far, and it doesn't seem like any will ever work.

Comment by tailcalled on Consequentialism is a compass, not a judge · 2024-04-13T11:51:31.716Z · LW · GW

Upvoted, but I think I disagree out of a tangent.

The consequences of someone's actions are nonetheless partial evidence of their morality. If you discover that embezzled funds have been building up on Bob's bank account, that's evidence Bob is an unethical guy—most people who embezzle funds are unethical. But then you might discover that, before he was caught and the money confiscated, Bob was embezzling funds to build an orphanage. The consequences haven't changed, but Bob's final (unresolved) intentions are attenuating circumstances. If I had to hang around with either your typical fund-embezzler or Bob, I would pick Bob.

An orphanage is sort of a funky example, because I don't intuitively associate it with cost-effectiveness, but I don't know much about it. If it's not cost-effective to build an orphanage, then what logic does Bob see in it? Under ordinary circumstances, I associate non-cost-effective charity with just doing what you've cached as good without thinking too much about it, but embezzlement doesn't sound like something you'd cache as good, so that doesn't sound likely. Maybe he's trying to do charity to build reputation that he can leverage into other stuff?

Anyway, if I don't fight the hypothetical, and assume Bob's embezzling for an orphanage was cost-effective, then that's evidence that he's engaging in fully unbounded consequentialism, aspiring to do the globally utility-maximizing action regardless of his personal responsibilities, his attention levels and his comparative advantages.

This allows you to predict that in the future, he might do similar things, e.g. secretly charge ahead with creating AI that takes over the world 0.1% more quickly and 0.1% more safely than its competitors even if there's 99.8% chance everyone dies, in order to capture the extra utility in that extra sliver he gains. Or that he might suppress allegations of rape within his circles if he fears the drama will push his group off track from saving the world.

If, on the other hand, someone was embezzling funds to spend on parties for himself and his friends, then while that's still criminal, it's a much more limited form of criminality, where he still wouldn't want to be part of the team that destroys the world, and wouldn't want to protect rapists. (I mean, he might still want to protect rapists if he's closer friends with the person who is raping than with the victims, but the point is he's trying to help at least some of the people around himself.)

Honestly the one who embezzles funds for unbounded consequentialist purposes sounds much more intellectually interesting, and so I would probably still prefer to hang around him, but the one who embezzles funds for parties seems much safer, and so I think a moral principle along the lines of "unbounded consequentialists are especially evil and must be suppressed" makes sense. You know, the whole thing where we understand that "the ends justify the means" is a villainous thing to say.

I think this is actually pretty cruxy for consequentialism. Of course, you can try to patch consequentialism in various ways, but these problems show up all over the place and are subject to a lot of optimization pressure because resources are useful for many things, so one needs a really robust solution in order for it to be viable. I think the solution lies in recognizing that healthy systems follow a different kind of agency that doesn't aspire to have unbounded impact, and consequentialists need to develop a proper model of that to have a chance.

Comment by tailcalled on Ackshually, many worlds is wrong · 2024-04-12T20:33:47.650Z · LW · GW

Measure theory and probability theory was developed to describe stochasticity and uncertainty, but they formalize it in many-worlds terms, closely analogous to how the wavefunction is formalized in quantum mechanics. If one takes the wavefunction formalism literally to the point of believing that quantum mechanics must have many worlds, it seems natural to take the probability distribution formalism equally literally to the point of believing that probability must have many worlds too. Or well, you can have a hidden variables theory of probability too, but the point is it seems like you would have to abandon True Stochasticity.

True Stochasticity vs probability distributions provides a non-quantum example of the non-native embedding, so if you accept the existence of True Stochasticity as distinct from many worlds of simultaneous possibility or ignorance of hidden variables, then that provides a way to understand my objection. Otherwise, I don't yet know a way to explain it, and am not sure one exists.

As for the case of how a new branch of math could describe wavefunctions more natively, there's a tradeoff where you can put in a ton of work and philosophy to make a field of math that describes an object completely natively, but it doesn't actually help the day-to-day work of a mathematician, and it often restricts the tools you can work with (e.g. no excluded middle and no axiom of choice), so people usually don't. Instead they develop their branch of math within classical math with some informal shortcuts.

Comment by tailcalled on Ackshually, many worlds is wrong · 2024-04-12T19:59:34.051Z · LW · GW

Okay, so by "wavefunction as a classical mathematical object" you mean a vector in Hilbert space?

Yes.

In that case, what do you mean by the adjective "classical"?

There's a lot of variants of math; e.g. homotopy type theory, abstract stone duality, nonstandard analysis, etc.. Maybe one could make up a variant of math that could embed wavefunctions more natively.

Comment by tailcalled on Any evidence or reason to expect a multiverse / Everett branches? · 2024-04-12T18:13:37.051Z · LW · GW

Hi? Edit: the parent comment originally just had a single word saying "Test"

Comment by tailcalled on Ackshually, many worlds is wrong · 2024-04-12T13:43:22.363Z · LW · GW

Do you actually need any other reason to not believe in True Randomness?

I think I used to accept this argument, but then came to believe that simplicity of formalisms usually originates from renormalization more than from the simplicity being Literally True?

Comment by tailcalled on Ackshually, many worlds is wrong · 2024-04-12T10:16:02.985Z · LW · GW

As a matter of fact, it is modeled this way. To define probability function you need a sample space, from which exactly one outcome is "sampled" in every iteration of probability experiment.

No, that's for random variables, but in order to have random variables you first need a probability distribution over the outcome space.

And this is why, I have troubles with the idea of "true randomness" being philosophically coherent. If there is no mathematical way to describe it, in which way can we say that it's coherent?

You could use a mathematical formalism that contains True Randomness, but 1. such formalisms are unwieldy, 2. that's just passing the buck to the one who interprets the formalism.

Comment by tailcalled on Ackshually, many worlds is wrong · 2024-04-12T10:12:18.793Z · LW · GW

The wavefunction in quantum mechanics is not like the probability distribution of (say) where a dart lands when you throw it at a dartboard. (In some but not all imaginable Truly Stochastic worlds, perhaps it's like the probability distribution of the whole state of the universe, but OP's intuition-pumping example seems to be imagining a case where A is some small bit of the universe.)

The reason why it's not like that is that the laws describing the evolution of the system explicitly refer to what's in the wavefunction. We don't have any way to understand and describe what a quantum universe does other than in terms of the evolution of the wavefunction or something basically equivalent thereto.

In my view, the big similarity is in principle of superposition. The evolution of the system in a sense may depend on the wavefunction, but it is an extremely rigid sense which requires it to be invariant to chopping up a superposition to a bunch of independent pieces, or chopping up a simple state into an extremely pathological superposition.

I have the impression -- which may well be very unfair -- that at some early stage OP imbibed the idea that what "quantum" fundamentally means is something very like "random", so that a system that's deterministic is ipso facto less "quantum" than a system that's stochastic. But that seems wrong to me. We don't presently have any way to distinguish random from deterministic versions of quantum physics; randomness or something very like it shows up in our experience of quantum phenomena, but the fact that a many-worlds interpretation is workable at all means that that doesn't tell us much about whether randomness is essential to quantum-ness.

It's worth emphasizing that the OP isn't really how I originally thought of QM. One of my earliest memories was of my dad explaining quantum collapse to me, and me reinventing decoherence by asking why it couldn't just be that you got entangled with the thing you were observing. It's only now, years later, that I've come to take issue with QM.

In my mind, there's four things that strongly distinguish QM systems from ordinary stochastic systems:

  • Destructive interference
  • Principle of least action (you could in principle have this and the next in deterministic/stochastic systems, but it doesn't fall out of the structure the ontology as easily, without additional laws)
  • Preservation of information (though of course since the universe is actually quantum, this means the universe doesn't resemble a deterministic or stochastic system at the large scale, because we have thermodynamics and neither deterministic nor stochastic systems need thermodynamics)
  • Pauli exclusion principle (technically you could have this in a stochastic system too, but it feels quantum-mechanical because it can be derived from fermion products being anti-symmetric, and anti-symmetry only makes sense in quantum systems)

Almost certainly this isn't complete, since I'm mostly autodidact (got taught a bit by my dad, read standard rationalist intros to quantum, like The Sequences and Scott Aaronson, took a mathematical physics course, and coded a few qubit simulations, binged some Wikipedia and Youtube). Of these, only destructive interference really seems like an obstacle, and only a mild one.

(And, incidentally, if we had a model of Truly Stochastic physics in which the evolution of the system is driven by what's inside those probability distributions -- why, then, I would rather like the idea of claiming that the probability distributions are what's real, rather than just their outcomes.)

I would say this is cruxy for me, in the sense that if I didn't believe Truly Stochastic systems were ontologically fine, then I would take similar issue with Truly Quantum systems.

Comment by tailcalled on Ackshually, many worlds is wrong · 2024-04-12T07:46:57.273Z · LW · GW

In the absence of a measurement/collapse postulate, quantum mechanics is a deterministic theory

You can make a deterministic theory of stochasticity using many-worlds too.

In the absence of a postulate that the wavefunction is Literally The Underlying State, rather than just a way we describe the system deterministically, quantum dynamics doesn't fit under a deterministic ontology.

Also, what do you mean by "the wavefunction as a classical mathematical object"?

If you have some basis , you can represent quantum systems using functions  (or perhaps more naturally, as  where  denotes the free vector space, but then we get into category theory, and that's a giant nerdsnipe).

Comment by tailcalled on Ackshually, many worlds is wrong · 2024-04-12T07:28:22.272Z · LW · GW

For any well-controlled isolated system, if it starts in a state |Ψ⟩, then at a later time it will be in state U|Ψ⟩ where U is a certain deterministic unitary operator. So far this is indisputable—you can do quantum state tomography, you can measure the interference effects, etc. Right?

It will certainly be mathematically well-described by an expression like that. But when you flip a coin without looking at it, it will also be well-described by a probability distribution 0.5 H + 0.5 T, and this doesn't mean that we insist that after the flip, the coin is Really In That Distribution.

Now it's true that in quantum systems, you can measure a bunch of additional properties that allow you to rule out alternative models. But my OP is more claiming that the wavefunction is a model of the universe, and the actual universe is presumably the disquotation of this, so by construction the wavefunction acts identically to how I'm claiming the universe acts, and therefore these measurements wouldn't be ruling out that the universe works that way.

Or as a thought experiment: say you're considering a simple quantum system with a handful of qubits. It can be described with a wavefunction that assigns each combination of qubit values a complex number. Now say you code up a classical computer to run a quantum simulator, which you do by using a hash map to connect the qubit combos to their amplitudes. The quantum simulator runs in our quantum universe.

Now here's the question: what happens if you have a superposition in the original quantum system? It turns into a tensor product in the universe the quantum computer runs in, because the quantum simulator represents each branch of the wavefunction separately.

This phenomenon, where a superposition within the system gets represented by a product outside of the system, is basically a consequence of modelling the system using wavefunctions. Contrast this to if you were just running a quantum computer with a bunch of qubits, so the superposition in the internal system would map to a superposition in the external system.

I claim that this extra product comes from modelling the system as a wavefunction, and that much of the "many worlds" aspect of the many-worlds interpretation arises from this (since products represent things that both occur, whereas things in superposition are represented with just sums).

OK, so then you say: “Well, a very big well-controlled isolated system could be a box with my friend Harry and his cat in it, and if the same principle holds, then there will be deterministic unitary evolution from |Ψ⟩ into U|Ψ⟩, and hey, I just did the math and it turns out that U|Ψ⟩ will have a 50/50 mix of ‘Harry sees his cat alive’ and ‘Harry sees his cat dead and is sad’.” This is beyond what’s possible to directly experimentally verify, but I think it should be a very strong presumption by extrapolating from the first paragraph. (As you say, “quantum computers prove larger and larger superpositions to be stable”.)

Yes, if you assume the wavefunction is the actual state of the system, rather than a deterministic model of the system, then it automatically follows that something-like-many-worlds must be true.

…And then there’s an indexicality issue, and you need another axiom to resolve it. For example: “as quantum amplitude of a piece of the wavefunction goes to zero, the probability that I will ‘find myself’ in that piece also goes to zero” is one such axiom, and equivalent (it turns out) to the Born rule. It’s another axiom for sure; I just like that particular formulation because it “feels more natural” or something.

Huh, I didn't know this was equivalent to the born rule. It does feel pretty natural, do you have a reference to the proof?

I’m really unsympathetic to the second bullet-point attitude, but I don’t think I’ve ever successfully talked somebody out of it, so evidently it’s a pretty deep gap, or at any rate I for one am apparently unable to communicate past it.

I agree with the former bullet point rather than the latter.

FWIW last I heard, nobody has constructed a pilot-wave theory that agrees with quantum field theory (QFT) in general and the standard model of particle physics in particular. The tricky part is that in QFT there’s observable interference between states that have different numbers of particles in them, e.g. a virtual electron can appear then disappear in one branch but not appear at all in another, and those branches have easily-observable interference in collision cross-sections etc. That messes with the pilot-wave formalism, I think.

Someone in the comments of the last thread claimed maybe some people found out how to generalize pilot-wave to QFT. But I'm not overly attached to that claim; pilot-wave theory is obviously directionally incorrect with respect to the ontology of the universe, and even if it can be forced to work with QFT, I can definitely see how it is in tension with it.

Comment by tailcalled on Ackshually, many worlds is wrong · 2024-04-11T21:18:34.668Z · LW · GW

I guess it's hard to answer because it depends on three degrees of freedom:

  • Whether you agree with my assessment that it's mostly arbitrary to demand the fundamental ontology to be deterministic rather than stochastic or quantum,
  • Whether you count "many worlds" as literally asserting that the wavefunction as a classical mathematical object is real or as simply distancing oneself from collapse/hidden variables,
  • Whether you even aim to describe what is ontologically fundamental in the first place.

I'm personally inclined to say the many-worlds interpretation is technically wrong, hence the title. But I have basically suggested people could give different answers to these sorts of degrees of freedom, and so I could see other people having different takeaways.

Comment by tailcalled on Any evidence or reason to expect a multiverse / Everett branches? · 2024-04-11T16:22:38.381Z · LW · GW

The observer is highly sensitive to differences along a specific basis, and therefore changes a lot in response to that basis. Due to chaos, this then leads to everything else on earth getting entangled with the observer in that same basis, implying earth-wide decoherence.

Comment by tailcalled on Any evidence or reason to expect a multiverse / Everett branches? · 2024-04-11T05:54:15.840Z · LW · GW

This is just chaos theory, isn't it? If one person sees that Schrodinger's cat is dead, then they're going to change their future behavior, which changes the behavior of everyone they interact with, and this then butterflies up to entangle the entire earth in the same superposition.

Comment by tailcalled on Any evidence or reason to expect a multiverse / Everett branches? · 2024-04-10T18:26:15.899Z · LW · GW

Uncharitable punchline is "if you take pilot wave but keep track of every possible position that any particle could have been (and ignore where they actually were in the actual experiment) then you get many worlds." Seems like a dumb thing to do to me.

How would you formalize pilot wave theory without keeping "track of every possible position that any particle could have been" (which I assume refers to, not throwing away the wavefunction)?

Comment by tailcalled on Any evidence or reason to expect a multiverse / Everett branches? · 2024-04-10T18:22:54.396Z · LW · GW

We'd still expect strongly interacting systems e.g. the earth (and really, the solar system?) to have an objective splitting. But it seems correct to say that I basically don't know how far that extends.

Comment by tailcalled on Any evidence or reason to expect a multiverse / Everett branches? · 2024-04-10T13:48:42.056Z · LW · GW

Let's say you have some unitary transformation . If you were to apply this to a coherent superposition , it seems like it would pretty much always make you end up with a decoherent superposition. So it doesn't seem like there's anything left to explain.

Comment by tailcalled on Any evidence or reason to expect a multiverse / Everett branches? · 2024-04-10T08:17:52.685Z · LW · GW

Kind of, because "multiple future outcomes are possible, rather than one inevitable outcome" could sort of be said to apply to both true stochasticity and true quantum mechanics. With true stochasticity, it has to evolve by a diffusion-like process with no destructive interference, whereas for true quantum mechanics, it has to evolve by a unitary-like process with no information loss.

So to a mind that can comprehend probability distributions, but intuitively thinks they always describe hidden variables or frequencies or whatever, how does one express true stochasticity, the notion where a probability distribution of future outcomes are possible (even if one knew all the information that currently exists), but only one of them happens?

Comment by tailcalled on Any evidence or reason to expect a multiverse / Everett branches? · 2024-04-10T08:05:51.344Z · LW · GW

Before I answer that question: do you know what I mean by a truly stochastic universe? If so, how would you explain the concept of true ontologically fundamental stochasticity to a mind that does not know what it means?

Comment by tailcalled on Any evidence or reason to expect a multiverse / Everett branches? · 2024-04-10T07:11:27.164Z · LW · GW

But |cat alive> + |cat dead> is a natural basis because that's the basis in which the interaction occurs. No mystery there; you can't perceive something without interacting with it, and an interaction is likely to have some sort of privileged basis.

Comment by tailcalled on Any evidence or reason to expect a multiverse / Everett branches? · 2024-04-10T07:02:36.812Z · LW · GW

Gonna post a top-level post about it once it's made it through editing, but basically the wavefunction is a way to embed a quantum system in a deterministic system, very closely analogous to how a probability function allows you to embed a stochastic system into a deterministic system. So just like how taking the math literally for QM means believing that you live in a multiverse, taking the math literally for probability also means believing that you live in a multiverse. But it seems philosophically coherent for me to believe that we live in a truly stochastic universe rather than just a deterministic probability multiverse, so it also feels like it should be philosophically coherent that we live in a truly quantum universe.