LessWrong 2.0 Reader

IMO, the discontinuity that is sufficient here is that I expect societal responses to be discontinuous, rather than continuous, and in particular, I expect societal responses will come when people start losing jobs en masse, and at that point, either the AI is aligned well enough that existential risk is avoided, or the takeover has inevitably happened and we have very little influence over the outcome.

On this point:

Meaningful representative example in what class: I think it's representative in 'weird stuff may happen', not in we will get more teenage-intern-trapped-in-a-machine characters.

Yeah, I expect society to basically not respond at all if weird stuff just happens, unless we assume more here, and in particular I think societal response is very discontinuous, even if AI progress is continuous, for both good and bad reasons.

Also see Scott Alexander's Heuristics That Almost Always Work.

. . . Is there a way a random punter could kick in, say, $100k towards Elon's bid? Either they end up spending $100k on shares valued at somewhere between $100k and $150k; or, more likely, they make the seizure of OpenAI $100k harder at no cost to themselves.

I once saw an advert claiming that a pregnancy test was “over 99% accurate”. This inspired me to invent an only-slightly-worse pregnancy test, which is over 98% accurate. My invention is a rock with “NOT PREGNANT” scrawled on it: when applied to a randomly selected human being, it is right more than 98% of the time. It is also cheap, non-invasive, endlessly reusable, perfectly consistent, immediately effective and impossible to apply incorrectly; this massive improvement in cost and convenience is obviously worth the ~1% decrease in accuracy.

This is fantastic work, I'd be very excited to see more work in the vein of auditing games. It seems like the one of the best ways so far to test how useful different techniques for understanding models are

Thanks. I think it's an important point you make; I do have it in mind that traits can have nonlinearities at different "stages", but I hadn't connected that to the personality trait issue. I don't immediately see a very strong+clear argument for personality traits being super exceptional here. Intuitively it makes sense that they're more "complicated" or "involve more volatile forces" or something due to being mental traits, but a bit more clarity would help. In particular, I don't see the argument being able to support yes significant broadsense heritability but very little apparent narrowsense heritability. (Though maybe it can somehow.)

(Also btw I wouldn't exactly call multiplicativity by itself "nonlinearity"! I would just say that after the genomic fan-in there is a nonlinearity. It's linearity as long as there's a latent variable that's a sum of alleles. Indeed, as E.G. pointed out to me, IQ could very well be like this, i.e. IQ-associated traits might be even better predicted by assuming IQ is lognormal (or some other such distribution). Though of course then you can't say that such and such downstream outcome is linear in genes; but the point would be that you can push along the trait by independently pushing on lots of alleles.)

(Also in theory I ought to go through your whole article at some point; but as yet I'm not read in to polygenic scores, as there's already good enough predictors for the MVP, and biotech is the main bottleneck. Though yes, motivating the biotech depends on understanding PGSes.)

edit: uh, well, short answer there totally is, idk if they're the psychedelic states you wanted, but they should do for a lot of relevant purposes, seems pretty hard to match meds though. original longer version:

there's a huge space of psychedelic states, I think the subspace reachable with adding chemicals is a large volume that's hard to get to by walking state space with only external pushes - I doubt the kind of scraping a hole in the wall from a distance you can do with external input can achieve, eg, globally reversing the function of SERT (I think this paper I just found on google may show this - I forget where I first encountered the claim, not double checking it properly now), that MDMA apparently induces! you can probably induce various kinds of serotonin release, though.

but the premise of my argument here in the first place - where you can sometimes overwhelm human output behavior via well crafted input - is that that probably doesn't matter too much. human computational output bitrate seems to be on order ten bits per second across all modalities,^[1] and input bitrate is way above that, so my guess is that update bitrate (forming memories, etc) is much higher than natural output bitrate^[2], probably yeah you can do most of the weird targeted interventions you were previously getting via psychedelics instead from like, getting some emotional/tempo sorts of things to push into the attractor where neurons have similar functionality already. I just doubt you can go all the way to fixing neurological dysfunctions so severe that to even have a hope of doing it from external input, you'd need to be looking for these crazy brain hacking approaches we were talking about.

I guess what we'd need to measure is like, bitrate of self-correction internally within neurons, some FEP thing. not sure off the top of my head quite how to resolve that to something reasonable.

1. ^{^}

   of course, like, actually I'm pretty dang sure you can get way above 10bit/s by outputting more noisy output, but then you get bits that aren't coming from the whole network's integrated state. the 10bps claim feels right for choosing words or similar things like macroscopic choices, but something feels wrong with the claim to me.

2. ^{^}

   some concern I missed counterevidence to this memorization-bandwidth claim from the paper though!

I'm skeptical.

Did the Sakana team publish the code that their scientist agent used to write the compositional regularization paper? The post says

For our choice of workshop, we believe the ICBINB workshop is a highly relevant choice for the purpose of our experiment. As we wrote in the main text, we selected this workshop because of its broader scope, challenging researchers (and our AI Scientist) to tackle diverse research topics that address practical limitations of deep learning, unlike most workshops with a narrow focus on one topic.

This workshop focuses particularly on understanding limitations of deep learning methods applied to real world problems, and encourages participants to study negative experimental outcomes. Some may criticize our choice of a workshop that encourages discussion of “negative results” (implying that papers discussing negative results are failed scientific discoveries), but we disagree, and we believe this is an important topic.

and while it is true that "negative results" are important to report, "we report a negative result because our AI agent put forward a reasonable and interesting hypothesis, competently tested the hypothesis, and found that the hypothesis was false" looks a lot like "our AI agent put forward a reasonable and interesting hypothesis, flailed around trying to implement it, had major implementation problems, and wrote a plausible-sounding paper describing its failure as a fact about the world rather than a fact about its skill level".

The paper has a few places with giant red flags where it seems that the reviewer assumes that there were solid results that the author of the paper was simply not reporting skillfully, for example in section B2

 , 

I favor an alternative hypothesis: the Sakana agent determines where a graph belongs, what would be on the X and Y axis of that graph, what it expects that the graph would look like, and how to generate that graph. It then generates the graph and inserts the caption the graph would show if its hypothesis was correct. The agent has no particular ability to notice that its description doesn't work with the graph.

 

I have a simple model with toy examples of where non-additivity in personality and other domains comes from, see §4.3.3 here [LW · GW].

There are actually atmospheric microbiota, also called aeroplankton, though those are mostly bacteria.

I remember a novel by Stanisław Lem in which he talks about a planet which has an atmosphere with a green tint. It is caused by swarms of insects in the atmosphere doing photosynthesis. I don't know whether that would be realistic, but it's currently not possible on Earth, insofar no insects, nor any other animal, ever developed the ability to do photosynthesis.