LessWrong 2.0 Reader

I believe the confusion comes from assuming the current board follows rules rather than doing whatever is most convenient.

The old board was trying to follow the rules, and the people in question were removed (technically were pressured to remove themselves).

See my reply to Ryan - I'm primarily interested in offering advice on something like that question since I think it's where I have unusually helpful thoughts, I don't mean to imply that this is the only question that matters in making these sorts of decisions! Feel free to message me if you have pitches for other projects you think would be better for the world.

I think this is a really good and well-thought-out explanation of the agenda.

I do still think that it's missing a big piece: namely in your diagram, the lowest-tier dot (heuristic explanations) is carrying a lot of weight, and needs more support and better messaging.

As an extreme example of this, Stephen Wolfram believes he has a collection of ideas building on some thinking about cellular automata that will describe all of physics. He can write down all kinds of causal diagrams with this node in the root, leading to great strides in our understanding of science and the cosmos and so on. But ultimately, such a diagram would be making the statement that "there exists a productive way to build a theory of everything which is based on cellular automata in a particular principled way similar to how he thinks about this theory". Note that this is different from saying that cellular automata are interesting, or even that a better theory of cellular automata would be useful for physics, and requires a lot more motivation and scientific falsification to motivate.

The idea of heuristic arguments is, at its core, a way of generalizing the notion of independence in statistical systems and models of statistical systems. It's discussing a way to point at a part of the system and say "we are treating this as noise" or "we are treating these two parts as statistically independent", or "we are treating these components of the system as independently as we can, given the following set of observations about our system" (with a lot of the theory of HA asking how to make the last of these statements explicit/computable). I think this is a productive class of questions to think about, both theoretically and empirically. It's related to a lot of other research in the field (on causality, independence and so on). I conceptually vibe with ARC's approach from what I've seen of the org. (Modulo the corrigible fact that I think there should be a lot more empirical work on what kinds of heuristic arguments work in practice. For example what's the right independence assumption on components of an image classifier/ generator NN that notices/generates the kind of textural randomness seen in a cat's fur? So far there is no HA guess about this question, and I think there should be at least some ideas on this level for the field to have a healthy amount of empiricism.)

I think that what ARC is doing is useful and productive. However, I don't see strong evidence that this particular kind of analysis is a principled thing to put at the root of a diagram of this shape. The statement that we should think about and understand independence is a priori not the same as the idea that we should have a more principled way of deciding when one interpretation of a neural net is more principled than another, which is also separate from (though plausibly related to) the (I think also good) idea in MAD/ELK that it might be useful to flag NN's that are behaving "unusually" without having a complete story of the unusual behavior.

I think there's an issue with building such a big structure on top of an undefended assumption, which is that it is creates some immissibility (i.e., difficulty of mixing) with other ideas in interpretability, which are "story-centric". The phenomena that happen in neural nets (same as phenomena in brains, same as phenomena in realistic physical systems) are probably special: they depend on some particular aspects of the world/ of reasoning/ of learning that has some sophisticated moving parts that aren't yet understood (some standard guesses are shallow and hierarchical dependence graphs, abundance of rough symmetries, separation of scale-specific behaviors, and so on). Our understanding will grow by capturing these ideas in terms of suitably natural language and sophistication for each phenomenon.

[added in edit] In particular (to point at a particular formalization of the general critique), I don't think that there currently exists a defendable link between Heuristic Arguments and the proof verification as in Jason Gross's excellent paper. The specific weakening of the notion of proof verification is more general interpretability. Your post on surprise accounting [AF · GW], is also excellent, but it doesn't explain how heuristic arguments would lead to understanding systems better -- rather, it shows that if we had ways of making better independence assumptions about systems with an existing interpretation, we would get a useful way of measuring surprise and explanatory robustness (with proof a maximally robust limit). But I think that drawing the line from seeking explanations with some nice properties/ measurements to the statement that a formal theory of such properties would lead to an immediate generalization of proof/interpretability which is strictly better than the existing "story-centric" methods is currently undefended (similar to the story that some early work on causality in interp had that a good attempt to formalize and validate causal interpretations would lead to better foundations of interp. -- the techniques are currently used productively e.g. here, but as an ingredient of an interpretation analysis rather than the core of the story). I think similar critiques hold for other sufficiently strong interpretations of the other arrows in this post.

I think that the issue of the approach sketched here is that it overindexes on a particular shape of explanation -- namely, that the most natural way to describe the relevant details inherent in principled interpretability work will most naturally factorize through a language that grows out of better-understanding independence assumptions in statistical modeling. I don't see much evidence for this being the case, any more than I see evidence that the best theory of physics should grow out of a particular way of seeing cellular automata (and I'd in fact bet with some confidence that this is not true in both of these cases). At the same time I think that ARC ideas are good, and that trying to relate them to other work in interp is productive (I'm excited about the VAE draft in particular). I just would like to see a less ambitious, more collaboratively motivated version of this, which is working on improving and better validating the assumptions one could make as part of mechanistic/statistical analysis of a model (with new interpretability/MAD ideas as a plausible side-effect) rather than orienting towards a world where this particular direction is in some sense foundational for a "universal theory of interpretability".

Just saw the OP replied in another comment that he is offering advice.

Yeah, I agree that you should care about more than just the sign bit. I tend to think the magnitude of effects of such work is large enough that "positive sign" often is enough information to decide that it dominates many alternatives, though certainly not all of them. (I also have some kind of virtue-ethical sensitivity to the zero point of the impacts of my direct work, even if second-order effects like skill building or intra-lab influence might make things look robustly good from a consequentialist POV.)

The offer of the parent comment is more narrowly scoped, because I don't think I'm especially well suited to evaluate someone else's comparative advantages but do have helpful things to say on the tradeoffs of that particular career choice. Definitely don't mean to suggest that people (including myself) should take on capability-focused roles iff they're net good!

I did think a fair bit about comparative advantage and the space of alternatives when deciding to accept my offer; I've put much less work into exploration since then, arguably too much less (eg I suspect I don't quite meet Raemon's bar). Generally happy to get randomly pitched on things, I suppose! 

Will there be >1 individual per solar system?

A recently commonly heard viewpoint on the development of AI states that AI will be economically impactful but will not upend the dominancy of humans. Instead AI and humans will flourish together, trading and cooperating with one another. This view is particularly popular with a certain kind of libertarian economist: Tyler Cowen, Matthew Barnett, Robin Hanson.

They share the curious conviction that the probablity of AI-caused extinction p(Doom) is neglible. They base this with analogizing AI with previous technological transition of humanity, like the industrial revolution or the development of new communication mediums. A core assumption/argument is that AI will not disempower humanity because they will respect the existing legal system, apparently because they can gain from trades with humans.  

The most extreme version of the GMU economist view is Hanson's Age of EM; it hypothesizes radical change in the form of a new species of human-derived uploaded electronic people which curiously have just the same dreary office jobs as we do but way faster. 

Why is there trade & specialization in the first place?

Trade and specialization seems to mainly important in a world where: There are many individuals; those individuals have different skills and resources and a limited ability to transfer skills. 

Domain of Biology:  direct copying of genes but not brain, yes recombination, no or very low bandwith communication

highly adversarial, less cooperation,, no planning, much specialization, not centralized, vastly many sovereign individuals

Domain of Economics:   no direct copying, yes recombination, medium bandwith communication

 mildly adversarial, mostly cooperation,medium planning, much specialization, little centralization,   many somewhat soverign individuals

!AIs can copy, share and merge their weights!

Domain of Future AI society: direct copying of brain and machines, yes recombination, very high bandwith communication

?minimally adversarial, ?very high cooperation, ?cosmic-scale mathematized planning, ? little specialization, ? high centralization?, ? singleton sovereign individual

It is often imagined that in a 'good' transhumanist future the sovereign AI will be like a loving and caring parent for the billions and trillions of uploads. In this case, while there is one all-powerful Sovereign entity there are still many individuals who remain free and have their rights protected perhaps through cryptographic incantations.   [LW · GW]The closest cultural artifact would be Ian Banks' Culture series. 

There is another, more radical foreboding wherein the logic of ultra-high-bandwith sharing of weights is taken to the logical extreme and individuals merge into one transcendent hivemind. 

The public will Goodhart any metric you hand over to it. If you provide evaluation as a service, you will know how many attempts an AI lab made at your test.

I realize this ship has sailed, but: I continue to be confused about how the non-profit board can justify giving up control. Given that their mandate^[1] says that their 'primary fiduciary duty is to humanity' and to ensure that AGI 'is used for the benefit of all', and given that more control over a for-profit company is strictly better than less control, on what grounds can they justify relinquishing that control?

Am I just being insufficiently cynical here? Does everyone else tacitly recognize that the non-profit board isn't really bound by that charter and can do whatever they like? Or is there some justification that they can put forward with a straight face?

Is it maybe something like, 'We trust that OpenAI the for-profit is primarily dedicated to the general well-being of humanity, and so nothing is lost by giving up control'? (But then it seems like more control is still strictly better in case it turns out that in some shocking and unforeseeable way the for-profit later has other priorities). Or is it, 'OpenAI the for-profit is doing good in the world, and they can do much more good if they can raise more money, and there's certainly no way they could raise more money without us giving up control'?

1. ^{^}

   I'm assuming here that their charter is the same as OpenAI's charter, since I haven't been able to find a distinct non-profit charter.

A Different Gambit For Genetically Engineering Smarter Humans?

Background: Significantly Enhancing Adult Intelligence With Gene Editing [LW · GW], Superbabies [LW · GW]

Epistemic Status: @GeneSmith [LW · GW] or @sarahconstantin [LW · GW] or @kman [LW · GW] or someone else who knows this stuff might just tell me where the assumptions underlying this gambit are wrong.

I've been thinking about the proposals linked above, and asked a standard question: suppose the underlying genetic studies are Not Measuring What They Think They're Measuring [LW · GW]. What might they be measuring instead, how could we distinguish those possibilities, and what other strategies does that suggest?

... and after going through that exercise I mostly think the underlying studies are fine, but they're known to not account for most of the genetic component of intelligence, and there are some very natural guesses for the biggest missing pieces, and those guesses maybe suggest different strategies.

The Baseline

Before sketching the "different gambit", let's talk about the baseline, i.e. the two proposals linked at top. In particular, we'll focus on the genetics part.

GeneSmith's plan focuses on single nucleotide polymorphisms (SNPs), i.e. places in the genome where a single base-pair sometimes differs between two humans. (This type of mutation is in contrast to things like insertions or deletions.) GeneSmith argues pretty well IMO that just engineering all the right SNPs would be sufficient to raise a human's intelligence far beyond anything which has ever existed to date.

GeneSmith cites this Steve Hsu paper, which estimates via a simple back-the-envelope calculation that there are probably on the order of 10k relevant SNPs, each present in ~10% of the population on average, each mildly deleterious.

Conceptually, the model here is that IQ variation in the current population is driven mainly by mutation load: new mutations are introduced at a steady pace, and evolution kills off the mildly-bad ones (i.e. almost all of them) only slowly, so there's an equilibrium with many random mildly-bad mutations. Variability in intelligence comes from mostly-additive contributions from those many mildly-bad mutations. Important point for later: the arguments behind that conceptual model generalize to some extent beyond SNPs; they'd also apply to other kinds of mutations.

What's Missing?

Based on a quick googling, SNPs are known to not account for the majority of genetic heritability of intelligence. This source cites a couple others which supposedly upper-bound the total SNP contribution to about 25% of IQ variability (using a method which does not require identifying all the relevant SNPs, though I don't know the details of that method). Estimates of the genetic component of IQ tend to be 50-70%, so SNPs are about half or less.

Notably, IIRC, attempts to identify which mutations account for the rest by looking at human genetic datasets have also mostly failed to close the gap. (Though I haven't looked closely into that piece, so this is a place where I'm at particularly high risk of being wrong.)

So what's missing?

Guess: Copy Count Variation of Microsats/Minisats/Transposons

We're looking for some class of genetic mutations, which wouldn't be easy to find in current genetic datasets, have mostly-relatively-mild effects individually, are reasonably common across humans, and of which there are many in an individual genome.

Guess: sounds like variation of copy count in sequences with lots of repeats/copies, like microsatellites/minisatellites or transposons.

Most genetic sequencing for the past 20 years has been shotgun sequencing, in which we break the genome up into little pieces, sequence the little pieces, then computationally reconstruct the whole genome later. That method works particularly poorly for sequences which repeat a lot, so we have relatively poor coverage and understanding of copy counts/repeat counts for such sequences. So it's the sort of thing which might not have already been found via sequencing datasets, even though at least half the genome consists of these sorts of sequences.

Notably, these sorts of sequences typically have unusually high mutation rates. So there's lots of variation across humans. Also, there's been lots of selection pressure for the effects of those mutations to be relatively mild.

What Alternative Strategies Would This Hypothesis Suggest?

With SNPs, there's tens of thousands of different SNPs which would each need to be targeted differently. With high copy sequences, there's a relatively small set of different sequences. So the engineering part could be quite a lot easier, if we don't need to do different things with different copies. For instance, if the problem boils down to "get rid of live L1 transposons" or "lengthen all the XYZ repeat sequences", that would probably be simpler engineering-wise than targeting 10k SNPs.

The flip side is that there's more novel science to do. The main thing we'd want is deep sequencing data (i.e. sequencing where people were careful to get all those tricky high-copy parts right) with some kind of IQ score attached (or SAT, or anything else highly correlated with g-factor). Notably, we might not need a very giant dataset, as is needed for SNPs. Under (some versions of) the copy count model, there aren't necessarily thousands of different mutations which add up to yield the roughly-normal trait distribution we see. Instead, there's independent random copy events, which add up to a roughly-normal number of copies of something. (And the mutation mechanism makes it hard for evolution to fully suppress the copying, which is why it hasn't been selected away; transposons are a good example.)

So, main steps:

• Get a moderate-sized dataset of deep sequenced human genomes with IQ scores attached.
• Go look at it, see if there's something obvious like "oh hey centromere size correlates strongly with IQ!" or "oh hey transposon count correlates strongly with IQ!"
• If we find anything, go engineer that thing specifically, rather than 10k SNPs.

The title "How I Learned To Stop Trusting Prediction Markets and Love the Arbitrage" isn't appropriate for the content of the post. That there is a play-money prediction market in which it costs very little to make the prices on conditional questions very wrong does not provide significant reasons to trust prediction markets less. That this post got 193 karma leading me to see it 2 months later is a sign of bad-voting IMO. (There are many far better, more important posts that get far less karma.)