LessWrong 2.0 Reader

Ugh I can't believe I forgot about Rivest time locks, which are a better solution here.

Rules for per experiment betting seem to be imprecise. What exactly does it mean that Beauty can bet only once per experiment? Does it mean that she is offered the bet only once in case of Tails? If so, is she offered the bet on Monday or Tuesday or is the day randomly selected? Or does it mean that she is offered the bet on both Monday and Tuesday and only one bet counts if she accepts both? If so, which one? Monday bet, Tuesday bet, or is it randomly selected?

Depending on, a Thirder could base his decision on:

P(H/Today is Monday)=1/2, P(H/Today is my last awakening)=1/2, or P(H/Today is the randomly selected day my bet counts/is offered to me)=1/2

and therefore escapes utility instability?

Great work!

Did you ever run just the L0-approx & sparsity-frequency penalty separately? It's unclear if you're getting better results because the L0 function is better or because there are less dead features.

Also, a feature frequency of 0.2 is very large! 1/5 tokens activating is large even for positional (because your context length is 128). It'd be bad if the improved results are because polysemanticity is sneaking back in through these activations. Sampling datapoints across a range of activations should show where the meaning becomes polysemantic. Is it the bottom 10% (or 10% of max-activating example is my preferred method)

So what happened around Feb 25? It sure looks like something about your usage of Youtube and Twitter changed. Just to make sure I plotted an XMR chart, and yep, it sure looks like there's been a change in the process. (The a couple points lie outside the limits, and there are 3/4 consecutive points closer to the limits than the mean. Both signify exception variation, suggesting you did something different. The yellow line is just a divider showing the datapoint corresponding to the 18th Feb.) 
.

imagine an AI system which wipes out humans in order to secure its own power, and later on reflection wishes it hadn't; a wiser system might have avoided taking that action in the first place

I’m not confident this couldn’t swing just as easily (if not more so) in the opposite direction—a wiser system with unaligned goals would be more dangerous, not less. I feel moderately confident that wisdom and human-centered ethics are orthogonal categories, and being wiser therefore does not necessitate greater alignment.

On the topic of the competition itself, are contestants allowed to submit multiple entries?

As for where else these ideas can be found, philosophers have been working on conceptual vagueness intensely since the mid-20th century, and cluster concepts were a relatively early innovation. The philosophical literature also has the benefit of being largely free of nebulous speculations about cognition and needless formalism ... The literature also uses terminology in the ordinary way familiar to everybody engaging these issues professionally ... and avoids the invention of needless terms like "thingspace", which mainly achieve the isolation of LessWrong from the external literature.

I think there's some validity to this critique. I read The Cluster Structure of Thingspace (TCSOTS) and was asking myself "isn't this just talking about the problem of classification?" And classification definitely doesn't require us to treat 'birdness' or 'motherhood' as a discrete, as if a creature either has it or doesn't. Classification can be on a spectrum, with a score for 'birdness' or 'motherhood' that's a function of many properties. 

I welcome (!!) making these concepts more accessible to those who are unfamiliar with them, and for that reason I really enjoyed TCSOTS.But it also seems like there'd also be a lot of utility in then tying these concepts to the fields of math/CS/philosophy that are already addressing these exact questions. These ideas presented in The Cluster of Thingspace are not new; not even a little - so why not use them as a jumping-off-point for the broader literature on these subjects, to show how researchers in the field have approached these issues, and the solutions they've managed to come up with? 

See: Fuzzy Math, Support Vector Machines, ANNs, Decision Trees, etc. 

So: I think posts like this would have a stronger impact if tied into the broader literature that already covers the same subjects. The reader who started the article unfamiliar with the subject would, at the end, have a stronger idea of where the field stands, and they would also be better resourced for further exploring the subject on their own. 

Note: this is probably also why most scientific papers start with a discussion of previous related work. 

Any resignations yet? (The journalist doesn't seem to know of any.)

This makes me uncomfortable. From the perspective of sleeping beauty, who just woke up, the statement “today is Monday” is either true or false (she just doesn’t know which one). Yet you claim she can’t meaningfully assign it a probability. This feels wrong, and yet, if I try to claim that the probability is, say, 2/3, then you will ask me “in what sample space?” and I don’t know the answer.

What seems clear is that the sample space is not the usual sleeping beauty sample space; it has to run metaphorically “skew” to it somehow.

If the question were “did the coin land on heads” then it’s clear that this is question is of the form “what world am I in?”. Namely, “am I in a world where the coin landed on heads, or not?”

Likewise if we ask “does a Tuesday awakening happen?”… that maps easily to question about the coin, so it’s safe.

But there should be a way to ask about today as well, I think. Let’s try something naive first and see where it breaks. P(today is Monday | heads) = 100% is fine. (Or is that tails? I keep forgetting.) P(today is Monday | tails) = 50% is fine too. (Or maybe it’s not? Maybe this is where I’m going working? Needs a bit of work but I suspect I could formalize that one if I had to.) But if those are both fine, we should be able to combine them, like so: heads and tails are mutually exclusive and one of them must happen, so: P(today is Monday) = P(heads) • P(today is Monday | heads) + P(tails) • P(today is Monday | tails) = 0.5 + .25 = 0.75 Okay, I was expecting to get 2/3 here. Odd. More to the point, this felt like cheating and I can’t put my finger on why. maybe need to think more later

I'm excited for language model interpretability to teach us about the difference between compilers and simulations of compilers. In the sense that chatgpt and I can both predict what a compiler of a suitably popular programming language will do on some input, what's going on there---- surely we're not reimplementing the compiler on our substrate, even in the limit of perfect prediction? Will be an opportunity for a programming language theorist in another year or two of interp progress

I feel pretty into encrypting the weights and throwing the encryption key into the ocean or something, where you think it's very likely you'll find it in the limits of technological progress