LessWrong 2.0 Reader

…So that’s all that’s needed. If any system has both a capacity for endogenous action (motor control, attention control, etc.), and a generic predictive learning algorithm, that algorithm will be automatically incentivized to develop generative models about itself (both its physical self and its algorithmic self), in addition to (and connected to) models about the outside world.

Yes, and there are many different classes of such models. Most of them boring because the prediction of the effect of the agent on the environment is limited (small effect or low data rate) or simple (linear-ish or more-is-better-like). 

But the self-models of social animals will quickly grow complex because the prediction of the action on the environment includes elements in the environment - other members of the species - that themselves predict the actions of other members.

You don't mention it, but I think Theory of Mind [? · GW] or Emphatic Inference [? · GW] play a large role in the specific flavor of human self-models.

FYI there’s some backup on westbound William Cannon just before the turn into thr neighborhood at James Ranch Rd.

Not for this kind of fact, I’m afraid – my experience is that in answering questions like these, LLMs typically do no better than an educated guess. There are just way too many people stating their educated legal guesses as fact in the corpus, so it gets hard to distinguish.

Mhh, I think "it's not possible to solve (1) without also solving (2)" is equivalent to "every solution to (1) also solves (2)", which is equivalent to "(1) is sufficient for (2)". I did take some liberty in rephrasing step (2) from "figure out what consciousness is" to "figure out its computational implementation".

On the question of quantizing different feature activations differently: Computing the description length using the entropy of a feature activation's probability distribution is flexible enough to distinguish different types of distributions. For example, binary distributions would have a entropy of one bit and more continuous distributions would have larger entropies.

In our methodology, the effective float precision matters because it sets the bin width for the histogram of a feature's activations that is then used to compute the entropy. We used the same effective float precision for all features, which was found by rounding activations to different precisions until the reconstruction or cross-entropy loss is changed by some amount.

When it comes to trying to understand basic facts like how legal systems work LLMs make it easy to get an overview. 

The LessWrong Review [? · GW] runs every year to select the posts that have most stood the test of time. This post is not yet eligible for review, but will be at the end of 2024. The top fifty or so posts are featured prominently on the site throughout the year.

Hopefully, the review is better than karma at judging enduring value. If we have accurate prediction markets on the review results, maybe we can have better incentives on LessWrong today. Will this post make the top fifty?

The whole point of the sentence was to demonstrate how bad ambiguity can get with pronouns, and this exchange is demonstrating my point exactly. The issue might be that you're making some (very reasonable) assumptions without noticing it narrows the range of possible interpretations. The only unambiguous part of the sentence is "John told Mark", but every other he can be either John or Mark.

Edit: my apologies for any rude tone, it was not intentional. All of us necessarily make reasonable assumptions to narrow ambiguity in our day to day conversations and it can be hard to completely jettison the habit.

I want to better understand how QACI works, and I'm gonna try Cunningham's Law. @Tamsin Leake [LW · GW].

QACI works roughly like this:

1. We find a competent honourable human $H$, like Joe Carlsmith or Wei Dai, and give them a rock engraved with a 2048-bit secret key. We define $H^{+}$ as the serial composition of a bajillion copies of $H$.
2. We want a model $M$ of the agent $H^{+}$. In QACI, we get $M$ by asking a Solomonoff-like ideal reasoner for their best guess about $H^{+}$ after feeding them a bunch of data about the world and the secret key.
3. We then ask $M$ the question $q$, "What's the best reward function to maximise?" to get a reward function $r:(O\times A{)}^{\ast }\to R$. We then train a policy $\pi :(O\times A{)}^{\ast }\times O\to \Delta A$ to maximise the reward function $r$. In QACI, we use some perfect RL algorithm. If we're doing model-free RL, then $\pi$ might be AIXI (plus some patches). If we're doing model-based RL, then $\pi$ might be the argmax over expected discounted utility, but I don't know where we'd get the world-model $\tau :(O\times A{)}^{\ast }\to \Delta O$ — maybe we ask $M$?

So, what's the connection between the final policy $\pi$ and the competent honourable human $H$? Well overall, $\pi$ maximises a reward function specified by the ideal reasonser's estimation of the serial composition of a bajillion copies of $H$. Hmm.

Questions:

1. Is this basically IDA, where Step 1 is serial amplification, Step 2 is imitative distillation, and Step 3 is reward modelling?
2. Why not replace Step 1 with Strong HCH or some other amplification scheme?
3. What does "bajillion" actually mean in Step 1?
4. Why are we doing Step 3? Wouldn't it be better to just use $M$ directly as our superintelligence? It seems sufficient to achieve radical abundance, life extension, existential security, etc.
5. What if there's no reward function that should be maximised? Presumably the reward function would need to be "small", i.e. less than a Exabyte, which imposes a maybe-unsatisfiable constraint.
6. Why not ask $M$ for the policy $\pi$ directly? Or some instruction for constructing $\pi$? The instruction could be "Build the policy using our super-duper RL algo with the following reward function..." but it could be anything.
7. Why is there no iteration, like in IDA? For example, after Step 2, we could loop back to Step 1 but reassign $H$ as $H$ with oracle access to $M$.
8. Why isn't Step 3 recursive reward modelling? i.e. we could collect a bunch of trajectories from $\pi$ and ask $M$ to use those trajectories to improve the reward function.

I'm aware of Griggs v Duke; do you have more modern examples? Note that the Duke case was about a company that was unambiguously racist in the years leading up to the IQ test (ie they had explicit rules forbidding black people from working in some sections of the company), so it's not surprising that judges will see their implementation of the IQ test after the Civil Rights Act was passed as an attempt to continue racist policies under a different name. 

"I've never had issue before" is not a legal argument. 

But it is a Bayesian argument for how likely you are to get in legal trouble. Big companies are famously risk-averse.

"The military, post office, and other government agencies get away with it under the doctrine of sovereign immunity"

1. Usually the government bureaucracy cares more than the private sector about being racist or perceived as racist, not less. It's also easier for governments to create rules for their own employees than in the private sector, see eg US Army integration in 1948.
2. Also the NFL use IQ-test-like things, and the NFL is a) not a government agency, and b) extremely prominent.