LessWrong 2.0 Reader

but only the dialogues?

actually, it probably needs a re-ordering. place the really terse stuff in an appendix, put the dialogues in the beginning, etc.

Actual humans aren't "aligned" with each other, and they may not be consistent enough that you can say they're always "aligned" with themselves.

Completely agreed, see for example my post 3. Uploading [LW · GW] which makes this point at length.

Anyway, even if the approach did work, that would just mean that "its own ideas" were that it had to learn about and implement your (or somebody's?) values, and also that its ideas about how to do that are sound. You still have to get that right before the first time it becomes uncontrollable. One chance, no matter how you slice it.

Inspired by abstractapplic's machine learning and wanting to get some experience in julia, I got Claude (3.5 sonnet) to write me an XGBoost implementation in julia. Took a long time especially with some bugfixing (took a long time to find that a feature matrix was the wrong shape - a problem with insufficient type explicitness, I think). Still way way faster than doing it myself! Not sure I'm learning all that much julia, but am learning how to get Claude to write it for me, I hope.

Anyway, I used a simple model that

and a full model that

Results:

Predictions for individual matchups for my and abstractapplic's solutions:

The distinction between inner and outer alignment is quite unnatural. For example, even the concept of reward hacking implies the double-fold failure of a reward that is not robust enough to exploitation, and a model that develops instrumental capabilities as to find a way to trick the reward; indeed, in the case of reward hacking, it's worth noting that depending on the autonomy of the system in question, we could attribute the misalignment as inner or outer. At its core, this distinction comes out of the policy <-> reward scheme of RL, though prediction <-> loss function in SL can be similarly characterized; I doubt how well this framing generalizes to other engineering choices.

>Do you think there is a situation where selected random people do not want to be in office/leadership and want to pursue their own passion/career and thus due to this reason may do a bad job? Is this mandatory?

I think a robust way to design the assembly (or multiple assemblies like with Bouricius's model) is to have many different people serving different term lengths. Some people may serve a term of only a couple days or weeks. Others might serve for years. 

For short-term service, I would make that mandatory. Everyone is required to come. 

For long term service, maybe those should be voluntary. 

As far as incentives go, there's a range of enforcement options for "mandatory" service. Perhaps you can just pay a big fine, as a percentage of your income, as an alternative to service. There probably ought to be mechanisms to defer service so you can time things a bit better with your life circumstances. 

The typical Citizens' Assembly will also offer benefits such as child care, parental care. 

A high paying salary will encourage the lower and middle class to participate. 

I have trouble coming up with ways to help small business owners to participate though. Could a small business owner drop their work for an entire year, even if it was well paid -- especially if the small business is so small there are no managers to cover their role? Perhaps there could be alternatives for them, such as part time work coupled with work-from-home. 

>What are some nuances about population and diversity? (I am not sure yet)

I have yet to hear about a case where Deliberative decision making techniques were tried and failed due to excessive diversity or cultural factors. I'm not an expert on the latest and greatest research here so I may be wrong. I do know that deliberation experiments have been performed all around the world, including East Asia, Africa, and India. 

An example deliberative poll was performed in Uganda, paper linked here:

https://direct.mit.edu/daed/article/146/3/140/27163/Applying-Deliberative-Democracy-in-Africa-Uganda-s

I haven't fully read this yet. Note that James Fishkin is the guy that performs and advocates for these "deliberative polls". 

Very interesting, this would certainly cast doubt on 

But so far I haven't been noticing

After reading your comments I've been getting Claude to write up an XGBoost implementation for me, I should have made this reply comment when I started, but will post my results under my own comment chain.

I have not (but should) try to duplicate (or fail to do so) your findings - I haven't been quite testing the same thing.

How difficult would it be to turn this into an epub or pdf? Is there word of that coming soon?

Thanks for testing this! This is very interesting.

I'm still operating under the assumption that it does know the string, and is hallucinating, or that it can't retrieve it under these conditions. But slightly alarming that it apologises for lying, then lies again. 

Agreed. Seems plausible that it does recognize the string as the BIG-Bench canary (though on questioning it's very evasive, so I wouldn't say it's extremely likely). I'm leaning toward it having "forgotten" the actual string though ("forgotten" because it's plausible it can be extracted with the right prompt, but there isn't a good word for describing a larger elicitation gap).

Maybe like 10%?

Mm, I think if the question is "what accounts for the differences between the EA and rationalist movements today, wrt number of adherents, reputation, amount of influence, achievements" I would assign credit in the ratio of ~1:3 to differences in (values held by individuals):systems. Where systems are roughly: how the organizations are set up, how funding and information flows through the ecosystem.

I can think about that question if it seems relevant, but the initial claim of Elizabeth's was "I believe there are ways to recruit college students responsibly. I don't believe the way EA is doing it really has a chance to be responsible". So I was trying to give an account of the root cause there.

Also — and I recognize that I'm saying something relatively trivial here — the root cause of a problem in a system can of course be any seemingly minor part of it. Just because I'm saying one part of the system is causing problems (the culture's values) doesn't mean I'm saying that's what's primarily responsible for the output. The current cause of a software company's current problems might be the slow speed with which PR reviews are happening, but this shouldn't be mistaken for the claim that the credit allocation for the company's success is primarily that it can do PR reviews fast.

So to repeat, I'm saying that IMO the root cause of irresponsible movement growth and ponzi-scheme-like recruitment strategies was a lack of IMO very important values like dialogue and candor and respecting other people's sense-making and courage and so on, rather than an explanation more like 'those doing recruitment had poor feedback loops so had a hard time knowing what tradeoffs to make' (my paraphrase of your suggestion). 

I would have to think harder about which specific values I believe caused this particular issue, but that's my broad point.