I think for similar reasons trade in ivory from dead anyways elephants is severely restricted.

Why do you think wise AI advisors avoid the general problems with other AI?

Is that a wise AI, which is an advisor, or somebody who advises about AI who is wise?

I don't see how you've shown it's a bad assumption?

That violates assumption one (a single pass cannot produce super intelligent output).

This isn't a solution to aligned LLMs being abused by humans, but to unaligned LLMs abusing humans.

context retrieval, reasoning models

Both of these are examples of more intelligent systems built on top of an LLM where the LLM itself has no state.

Thirdly, the AI can use the environment as state in ways which would be nearly impossible to fully trace or mitigate.

Again same problem - AI tool use is mediated by text input and output, and the validator just needs access to the LLM input and output.

To take an extreme example: killing all humans, and the plan to kill all humans.

But the AI isn't going to be so stupid as to write the plan in plaintext, so it'll likely develop some sort of code.

So then the unaligned behaviour is writing in code, and the output that leads to it is developing use the code.

But maybe the LLM can one shot the code without needing to develop it? Sure but then the validator (by definition if it's the same LLM) can also tell the code has been developed.

Maybe the LLM can one shot a code to jailbreak the validator? Sure but that starts to break the assumption the LLM isn't super intelligent in a single pass.

This is all true, but sometimes people assume the opposite:

"Rules which are not-concise must be clear and comprehensive".

This is trivially false, but rules can often end up falling into this pit by keeping on adding new rules to handle unexpected edge cases. But each new rule creates further edge cases which have to be handled until the whole system becomes so complicated that nobody sure what's allowed or isn't and you call the work done.

Hence even in areas where comprehensiveness is important, like the tax code, it can be valuable to push for simplification. Because if verbosity isn't actually buying the comprehensiveness or clarity you need, might as well at least be concise.

Yes

See Bell's theorem. Basically we know that quantum mechanics is truly random, not just pseudorandom, unless you posit non-locality.

I did also like the Ascension story. It did a very good job of imitating 1960s sci fi magazine stories. In a way it shows off his talent as an author more than the main story does!

Your review actually makes me more curious about the book. 

Looks like I did a bad job then 😂. I'm interested as to why? If it's the physics bit, it's 2 pages out of a thousand pages book and then basically not mentioned again.

Their unconsumed wealth is purely deflationary, allowing the government to print money for 'free'. Presumably that is less useful to society than e.g. giving it to an effective charity.

Their consumed wealth is sometimes used usefully - I buy that for Bill Gates for example. Sometimes it's frittered away on personal consumption. And sometimes it's given away to pointless/actively harmful charities like Mackenzie Scott.

The facts are many billionaires choose to either use their money for private consumption or waste it on pointless charities. That doesn't in any way imply that their having this money in unfair - they've earned it, and taking it away would make the world worse by discouraging excellence. It does however imply we should encourage them to pursue better uses for their money.

Both of these options sound wrong to me. I think the actual case is kind of obvious when you think about it:

Wealth is the reward people get for doing useful things.

Jeff Bezos is rich because he found away to easily provide people with cheap goods. That benefited everyone. It is good that he is rich as a result, because that's what gave him the incentive to do so.

That does not in any way imply that now that he has that money he'd be able to use it more usefully than anyone else. It's possible he will, but also possible he'll waste it all on gigantic superyachts or a 2000 metre high statue of himself.

Given that's the case it seems perfectly reasonable to try to push him towards giving some of his wealth to effective charities which will likely do more good for the world than his default next best use.

I don't really see how? A frequentist would just run this a few times and see that the outcome is 1/2.

In practice, for obvious reasons, frequentists and bayesians always agree on the probability of anything that can be measured experimentally. I think the disagreements are more philosophical about when it's appropriate to apply probability to something at all, though I can hardly claim to be an expert in non-bayesian epistemology.

Consider two realistic scenarios:

A) I'm talking to someone and they tell me they have two children. "Oh, do you have any boys?" I ask, "I love boys!". They nod.

B) I'm talking to someone and they tell me they have two children. One of the children then runs up to the parent. It's a boy.

The chance of two boys is clearly 1/3 in the first scenario, and a half in the second.

The scenario in the question as asked is almost impossible to answer. Nobody would ever state "I have two children, at least one of whom is a boy." in real life, so there's no way to update in that situation. We have no way to generate good priors. Instead people make up a scenario that sounds similar but is more realistic, and because everyone does that differently they'll all have different answers.

I think Harberger taxes are inherently incompatible with Georgian taxes as Georgian taxes want to tax only the land and Harberger taxes inherently have to tax everything.

That said see my somewhat maverickal attempt to combine them here: https://www.lesswrong.com/posts/MjBQ8S5tLNGLizACB/combining-the-best-of-georgian-and-harberger-taxes. Under that proposal we would deal with this case by saying that if anyone outbid me for the land they would not be allowed to extract the oil until the arranged a separate deal with me, but could use the land for any other purpose.

My assumption for an LVT is that the tax is based on the value of the land sans any improvements the landowner has made to the land. This would thus exclude from the tax an increase in value due to you discovering oil or you building nearby, but include in the tax any increase in value due to your neighbours discovering oil or building on their land.

That said I don't know how this would be calculated in practice, especially once we get to more complicated cases (a business I'm a minority owner of discovers oil on my land, I split my plot of land into 2 and sell them to 2 different people, etc.).

On the other hand, most taxes have all sorts edge cases too, and whilst they're problematic, we muddle through them. I doubt that this couldn't be muddled through in a similar way.

Or to put it another way: these SAT scores are compatible with an average IQ anywhere between + 1.93 to + 3.03 SD. Insofar as your prior lies somewhere between these two numbers, and you don't have a strong opinion on what precisely Lesswrong selects for it's not going to update you very much in either direction.

Indeed if rationalists were entirely selected by IQ and nothing else, and there were no other confounders, and height was +1.85 SD, IQ would be +9.25 SD. In the real world this instead provides a Bayesian update that you were wrong in assuming rationalists were purely selected for by IQ, and not e.g. gender.

The fact that going from 2.42 SD to 3.03 SD is nonsensical does not in anyway make it more sensible to go from 2.42 to 1.93. Your response to faul_sname is completely irrelevant because it assumes rationalists are selected for on SAT, which is clearly false. The correct calculation is impossible to make accurately given we are missing key information, but we can make some estimates by assuming that rationalists are selected for something that correlates with by IQ and SATs and guessing what that correlation is.

Here’s the breakdown: a median SAT score of 1490 (from the LessWrong 2014 survey) corresponds to +2.42 SD, which regresses to +1.93 SD for IQ using an SAT-IQ correlation of +0.80. This equates to an IQ of 129.

I don't think that works unless Less wrong specifically selects for high SAT score. If it selects for high IQ and the high SAT is as a result of the high IQ then you would have to go the other way and assume an SD of 3.03.

If, as seems more likely, Less wrong correlates with both IQ and SAT score, then the exact number is impossible to calculate, but assuming it correlates with both equally we would estimate IQ at 2.42 SD.

Note this requires market failure by definition - otherwise if an action provides me a a small gain for a huge loss to you, you would be willing to pay me some amount of money not to take that action, benefiting us both.

As a concrete example of how this plays out in practice. If you require Bob to wear a tuxedo costing 5000 dollars, and other similar companies don't, in a perfect market for labour you would need to pay Bob 5000 dollars more than other companies to cover the tuxedo or he'd just work for them himself.

The fact that he doesn't suggests that other things are going on - for example finding an alternative job might haven take more than the amount of time it takes to earn 5000 dollars, or he didn't know when he signed the contract that a tuxedo was required, and the contract makes it difficult for him to switch.

Most murder mysteries on TV tend to have a small number of suspects, and the trick is to find which one did it. I get the feeling that real life murders the police either have absolutely no idea who did it, or know exactly who did it and just need to prove that it was them to the satisfaction of the court of law.

That explains why forensic tests (e.g. fingerprints) are used despite being pretty suspect. They convince the jury that the guilty guy did it, which is all that matters.

See https://issues.org/mnookin-fingerprints-evidence/ for more on fingerprints.

Interesting paper!

I'm worried that publishing it "pollutes" the training data and makes it harder to reproduce in future LLMs - since their training data will include this paper and discussions of it, they'll know not to trust the setup.

Any thoughts on this?

(This leads to further concern that me publishing this comment makes it worse, but at some point it ought to be discussed and better do that early with less advanced techniques than later with more sophisticated ones).

Chromebooks replace the caps lock key with a search key - which is functionally equivalent to the windows keys on windows. E.g. search+right goes to the end of the line.

Yep, and when you run out of letters in a section you use the core letter from the section with a subscript.

Also:

m: used for a second whole number when n is already taken.

p: used for primes

q: used for a second prime.

Only if the aim of the AI is to destroy humanity. Which is possible but unlikely. Whereas by instrumental convergence, all AIs, no matter their aims, will likely seek to destroy humanity and thereby reduce risk and competition for resource.

I would have concerns about suitably generic, flexible and sensitive humanoid robots, yes.

One thing to consider is how hard an AI needs to work to break out of human dependence. There's no point destroying humanity if that then leaves you with noone to man the power stations that keep you alive.

If limited nanofactories exist it's much easier to bootstrap them into whatever you want, than it is those nanofactories don't exist, and robotics haven't developed enough for you to create one without the human touch.

Presumably because there's a hope that having a larger liver could help people lose weight, which is something a lot of people struggle to do?

I imagine that part of the difference is because Orcas are hunters, and need much more sophisticated sensors + controls.

I gigantic jellyfish wouldn't have the same number of neurons as a similarly sized whale, so it's not just about size, but how you use that size.

Douglas Adams answered this long ago of course:

For instance, on the planet Earth, man had always assumed that he was more intelligent than dolphins because he had achieved so much—the wheel, New York, wars and so on—whilst all the dolphins had ever done was muck about in the water having a good time. But conversely, the dolphins had always believed that they were far more intelligent than man—for precisely the same reasons.

Thanks - I've rehauled that section. Note a Codorcet method is not sufficient here, as the counter-example I give shows.

Why? That's a fact about voting preferences in our toy scenario, not a normative statement about what people should prefer.

Thanks for this!

What are the chances of a variable bypass engine at some point? Any opinions?

Counterpoint: when I was about 12, I was too old to collect candy at my Synagogue on Simchat Torah, so I would beg a single candy from someone, then trade it up (Dutch book style) with naive younger kids until I had a decent stash. I was particularly pleased whenever my traded up stash included the original candy.

The single most useful thing I use LLMs for is telling me how to do things in bash. I use bash all the time for one off tasks, but not quite enough to build familiarity with it + learn all the quirks of the commands + language.

90% of the time it gives me a working bash script first shot, each time saving me between 5 minutes to half an hour.

Another thing LLMs are good at is e.g taking a picture of e.g. screw, and asking what type of screw it is.

They're also great at converting data from one format to another: here's some JSON, convert it into Yaml. Now prototext. I forgot to mention, use maps instead of nested structs, and use Pascal case. Also the JSON is hand written and not actually legal.

Similarly they're good at fuzzy data querying tasks. I received this giant error response including full stack trace and lots of irrelevant fields, where's the actual error, and what lines of the file should I look at.

Buyers have to pay a lot more, but sellers receive a lot more. It's not clear that buyers at high prices are worse off than sellers, so it's egalitarian impact is unclear.

Whereas when you stand in line, that time you wasted is gone. Nobody gets it. Everyone is worse off.

I've been convinced! I'll let my wife know as soon as I'm back from Jamaica!

Similarly, the point about trash also ignores the larger context. Picking up my own trash has much less relationship to disgust, or germs, than picking up other people's trash.

Agreed, but that's exactly the point I'm making. Once you apply insights from rationality to situations outside spherical trash in a vacuum filled park you end up with all sorts of confounding affects that make the insights less applicable. Your point about germs and my point about fixing what you break are complimentary, not contradictory.

I think this post is missing the major part of what "metarational" means: acknowledging that the kinds of explicit principles and systems humans can hold in working memory and apply in real time are insufficient for capturing the full complexity of reality, having multiple such principles and systems available anyway, and skillfully switching among them in appropriate contexts.

This sounds to me like a semantic issue? Metarational isn't exactly a standard AFAIAA, (I just made it up on the spot), and it looks like you're using it to refer to a different concept from me.

Sure it is, if you accept a whole bunch of assumptions. Or it could just not do that.

Reading this reminds me of Scott Alexander in his review of "what we owe the future":

But I’m not sure I want to play the philosophy game. Maybe MacAskill can come up with some clever proof that the commitments I list above imply I have to have my eyes pecked out by angry seagulls or something. If that’s true, I will just not do that, and switch to some other set of axioms. If I can’t find any system of axioms that doesn’t do something terrible when extended to infinity, I will just refuse to extend things to infinity. I can always just keep World A with its 5 billion extremely happy people! I like that one! When the friendly AI asks me if I want to switch from World A to something superficially better, I can ask it “tell me the truth, is this eventually going to result in my eyes being pecked out by seagulls?” and if it answers “yes, I have a series of twenty-eight switches, and each one is obviously better than the one before, and the twenty-eighth is this world except your eyes are getting pecked out by seagulls”, then I will just avoid the first switch. I realize that will intuitively feel like leaving some utility on the table - the first step in the chain just looks so much obviously better than the starting point - but I’m willing to make that sacrifice.

You come up with a brilliant simulation argument as to why the AI shouldn't just do what's clearly in his best interests. And maybe the AI is neurotic enough to care. But in all probability, for whatever reason, it doesn't. And it just goes ahead and turns us into paperclips anyway, ignoring a person running behind it saying "bbbbbbut the simulation argument".

I'm not sure why those shouldn't be included? If someone uses my AI to perform 500 million dollars of fraud, then I should probably have been more careful releasing the product.

The rest of the family is still into Mormonism, and his wife tried to sue him for millions, and she lost (false accusations)

In case you're interested in following this up, Tracing Woodgrains on the accusations: https://x.com/tracewoodgrains/status/1743775518418198532

His approach to achieving immortality seems to be similar to someone attempting to reach the moon by developing higher altitude planes. He's using interventions that seem likely to improve health and lifespan by a few percentage points, which is great, but can't possibly get us to where he wants to go.

My assumption is that any real solution to mortality will look more like "teach older bodies to self repair the same way younger bodies do" than "eat this diet, and take these supplements".