LessWrong 2.0 Reader

I agree; Eichmann in Jerusalem and immoral mazes come to mind. 

I think being a Catholic with no connection to living leaders makes more sense than being an EA who doesn't have a leader they trust and respect

Catholic EA: You have a leader you trust and respect, and defer to their judgement.

Sola Fide EA: You read 80k hours and Givewell, but you keep your own spreadsheet of EV calculations. 

but only the dialogues?

actually, it probably needs a re-ordering. place the really terse stuff in an appendix, put the dialogues in the beginning, etc.

Actual humans aren't "aligned" with each other, and they may not be consistent enough that you can say they're always "aligned" with themselves.

Completely agreed, see for example my post 3. Uploading [LW · GW] which makes this exact point at length.

Anyway, even if the approach did work, that would just mean that "its own ideas" were that it had to learn about and implement your (or somebody's?) values, and also that its ideas about how to do that are sound. You still have to get that right before the first time it becomes uncontrollable. One chance, no matter how you slice it.

True. Or, as I put it just above:

But yes, you do need to start the model off close enough to aligned that it converges to value learning. [LW · GW]

The point is that you now get one shot at a far simpler task: defining "your purpose as an AI is to learn about and implement the humans' collective values" is a lot more compact, and a lot easier to get right first time, than an accurate description of human values in their full large-and-fairly-fragile [LW · GW] details. As I demonstrate in the post linked to in that quote [LW · GW], the former, plus its justification as being obvious and stable under reflection, can be described in exhaustive detail on a few pages of text.

As for the the model's ideas on how to do that research being sound, that's a capabilities problem: if the model is incapable of performing a significant research project when at least 80% of the answer is already in human libraries, then it's not much of an alignment risk.

Inspired by abstractapplic's machine learning and wanting to get some experience in julia, I got Claude (3.5 sonnet) to write me an XGBoost implementation in julia. Took a long time especially with some bugfixing (took a long time to find that a feature matrix was the wrong shape - a problem with insufficient type explicitness, I think). Still way way faster than doing it myself! Not sure I'm learning all that much julia, but am learning how to get Claude to write it for me, I hope.

Anyway, I used a simple model that

and a full model that

Results:

Predictions for individual matchups for my and abstractapplic's solutions:

The distinction between inner and outer alignment is quite unnatural. For example, even the concept of reward hacking implies the double-fold failure of a reward that is not robust enough to exploitation, and a model that develops instrumental capabilities as to find a way to trick the reward; indeed, in the case of reward hacking, it's worth noting that depending on the autonomy of the system in question, we could attribute the misalignment as inner or outer. At its core, this distinction comes out of the policy <-> reward scheme of RL, though prediction <-> loss function in SL can be similarly characterized; I doubt how well this framing generalizes to other engineering choices.

>Do you think there is a situation where selected random people do not want to be in office/leadership and want to pursue their own passion/career and thus due to this reason may do a bad job? Is this mandatory?

I think a robust way to design the assembly (or multiple assemblies like with Bouricius's model) is to have many different people serving different term lengths. Some people may serve a term of only a couple days or weeks. Others might serve for years. 

For short-term service, I would make that mandatory. Everyone is required to come. 

For long term service, maybe those should be voluntary. 

As far as incentives go, there's a range of enforcement options for "mandatory" service. Perhaps you can just pay a big fine, as a percentage of your income, as an alternative to service. There probably ought to be mechanisms to defer service so you can time things a bit better with your life circumstances. 

The typical Citizens' Assembly will also offer benefits such as child care, parental care. 

A high paying salary will encourage the lower and middle class to participate. 

I have trouble coming up with ways to help small business owners to participate though. Could a small business owner drop their work for an entire year, even if it was well paid -- especially if the small business is so small there are no managers to cover their role? Perhaps there could be alternatives for them, such as part time work coupled with work-from-home. 

>What are some nuances about population and diversity? (I am not sure yet)

I have yet to hear about a case where Deliberative decision making techniques were tried and failed due to excessive diversity or cultural factors. I'm not an expert on the latest and greatest research here so I may be wrong. I do know that deliberation experiments have been performed all around the world, including East Asia, Africa, and India. 

An example deliberative poll was performed in Uganda, paper linked here:

https://direct.mit.edu/daed/article/146/3/140/27163/Applying-Deliberative-Democracy-in-Africa-Uganda-s

I haven't fully read this yet. Note that James Fishkin is the guy that performs and advocates for these "deliberative polls". 

Very interesting, this would certainly cast doubt on 

But so far I haven't been noticing

After reading your comments I've been getting Claude to write up an XGBoost implementation for me, I should have made this reply comment when I started, but will post my results under my own comment chain.

I have not (but should) try to duplicate (or fail to do so) your findings - I haven't been quite testing the same thing.

How difficult would it be to turn this into an epub or pdf? Is there word of that coming soon?

Thanks for testing this! This is very interesting.

I'm still operating under the assumption that it does know the string, and is hallucinating, or that it can't retrieve it under these conditions. But slightly alarming that it apologises for lying, then lies again. 

Agreed. Seems plausible that it does recognize the string as the BIG-Bench canary (though on questioning it's very evasive, so I wouldn't say it's extremely likely). I'm leaning toward it having "forgotten" the actual string though ("forgotten" because it's plausible it can be extracted with the right prompt, but there isn't a good word for describing a larger elicitation gap).