LessWrong 2.0 Reader

That's right.

I think it's not that devastating, since I expect that their method can be adapted to counter classic LoRA tuning (the algorithm takes as input some set finetuning methods it "trains against"). But yeah, it's not reassuring that it doesn't generalize between full-weight FT and LoRA.

I'm single because my current location is amazing in every way except its density of young, childless people. I have met somewhere between 300-1000 people over the last 4 years, and of those, 8 have been women between the ages of 20-30.

Most are 40+ parents, the rest are those parents' kids. The sad truth (I think), is that I simply have to move if I want to date.

I don't think the Benjamin Jesty case fully covers the space of 'directly getting what you want'. 

That seems like a case of 'intelligent, specialized solution to a problem, which could then be solved with minimal resources.'

There's a different class of power in the category of 'directly getting what you want' which looks less like a clever solution, and more like being in control of a system of physical objects.

This could be as simple as a child holding a lollipop, or a chipmunk with cheeks full of seeds.

Or it can be a more complicated system of physical control, with social implications. For instance, being armed with a powerful gun while facing an enraged charging grizzly bear. It doesn't take a unique or clever or skillful action for the armed human to prevail in that case. And without a gun, there is little hope of the human prevailing.

Another case is a blacksmith with some iron ore and coal and a workshop. So long as nothing interferes, then it seems reasonable to expect that the blacksmith could solve a wide variety of different problems which needed a specific but simple iron tool. The blacksmith has some affordance in this case, and is more bottlenecked on supplies, energy, and time than on intellect or knowledge.

I discuss this here: https://www.lesswrong.com/posts/uPi2YppTEnzKG3nXD/nathan-helm-burger-s-shortform?commentId=ZsDimsgkBNrfRps9r [LW(p) · GW(p)] 

I can imagine a variety of future scenarios that don't much look like the AI being in a dominance hierarchy with humanity, or look like it trading with humanity or other AIs.

For instance: if an AI were building a series of sea-floor factories, using material resources that it harvested itself. Some world governments might threaten the AI, tell it to stop using those physical resources. The AI might reply: "if you attack me, I will respond in a hostile manner. Just leave me alone." If a government did attack the AI, the AI might release a deadly bioweapon which wiped out >99% of humanity in a month. That seems less like some kind of dominance conflict between human groups, and more like a human spraying poison on an inconvenient wasp nest.

Similarly, if an AI were industrializing and replicating in the asteroid belt, and some world governments told it to stop, but the AI just said No or said nothing at all. What would the governments do? Would they attack it? Send a rocket with a nuke? Fire a powerful laser? If the AI were sufficiently smart and competent, it would likely survive and counterattack with overwhelming force. For example, by directing a large asteroid at the Earth, and firing lasers at any rockets launched from Earth.

Or perhaps the AI would be secretly building subterranean factories, spreading through the crust of the Earth. We might not even notice until a whole continent started noticeably heating up from all the fission and fusion going on underground powering the AI factories. 

If ants were smart enough to trade with, would I accept a deal from the local ants in order to allow them to have access to my kitchen trashcan? Maybe, but the price I would demand would be higher than I expect any ant colony (even a clever one) to be able to pay. If they invaded my kitchen anyway, I'd threaten them. If they continued, I'd destroy their colony (assuming that ants clever enough to bargain with weren't a rare and precious occurrence). This would be even more the case, and even easier for me to do, if the ants moved and thought at 1/100th speed of a normal ant.

This is a comprehensive, nuanced, and well-written post. A few questions:

How likely do you think it is that, under a Harris administration, AI labs will successfully lobby Democrats to kill safety-oriented policies, as happened with SB 1047 on the state level? Even if Harris is on net better than Trump this could greatly reduce the expected value of her presidency from an x-risk perspective.

Related to the above, is it fair to say that under either party, there will need to be advocacy/lobbying for safety-focused policies on AI? If so, how do you make tradeoffs between this and the election? i.e. if someone has $x to donate, what percentage should they give to the election vs. other AI safety causes?

How much of your assessment of the difference in AI risk between Harris and Trump is due to the concrete AI policies you expect each of them to push, vs. how much is due to differences in competence and respect for democracy? 

I can't find much information about the Movement Labs quiz and how it helps Harris win. Could you elaborate, privately if needed? If the quiz is simply matching voters with the candidate who best matches their values, is it because it will be distributed to voters who lean Democrat, or does its effectiveness come through a different path?

There is the point about offensive/defensive asymmetry..

how much earlier

Yeah, good question. I don't know really. 

 and does it matter?

I think so, because even if pure AI control follows on from human-AI entity control (which would actually be my prediction), I expect the dynamics of human-AI control to very much lead to and accelerate that eventual pure AI control.

I'm thinking, also, that there is a thing where pure AI entities need to be careful not to 'tip their hat'. What I mean by this is that pure AI entities will need to be careful not to reveal the extent of their capabilities up until a point where they are actually capable of taking control, whereas human-AI entities can kind of go ahead and play the power game and start to build up control without so much concern about this. (To the average voter, this could just look like more of the same.)

One thing I am confused about: especially in cases of developer sandbagging, my intuition is that the mechanisms underlying the underperformance could be very similar to cases of "accidental" sandbagging (ie not sandbagging according to your def). More operationally, your example 1 and example 4 might have the same underlying issue from the perspective of the model itself, and if we want to find technical solutions to those particular examples they might look the same. If that's the case then it's not obvious to me that the "strategic" condition is a useful place to "cut nature at its joints."

Or to say it a different way, what operationally defines the difference between example 1 and 4 is that in ex.1 there is fine-tuning on a different dataset, and on ex.4 the extra dataset is part of the pre-training dataset. The model itself doesn't see the intent of the developer directly, so as far as technical solutions that only depend on the model itself, it's not obvious that the intent of the developer matters.

A developer could intentionally inject noisy and error-prone data into training, but the model would treat that equivalently to the case of where it was in the dataset by mistake.

While I agree that examples are stupid, I am not so sure about the electrical and plumbing. Connecting things to a public water supply that result in contamination from backflow/siphoning is very bad. You also dont want to electrocute a power pole worker who thinks power is off, but your house with DYO solar connection suddenly starts exporting power to grid. If I was insurer, I would take a dim view of unregister plumbing or electricial work in your house because of fire and flood risk. Where I live, so long as you are owner of property , you can do electricial that doesnt involve swithboard or cables coming into switchboard from street. Fair compromise? Drainlaying rules are still extremely strict though. 

Just returning to this -- Thank you for your comment. I think I need to do a bit more epistemological legwork before I can truly give you a response. Suffice it to say for now that I'm not really torn up about the issue and asked more out of boredom/general curiosity, but I see now that my question was asked lazily. 

Perhaps one day soon I'll comment on this with the relevant context/information. 

Given a world of humans, I don't think that libertarian society would be good enough at preventing competing powers from overthrowing it. Because there'd be an unexploitable-equillibrium condition where a government that isn't focused on dominance is weaker than a government more focused on government, it would generally be held by those who have the strongest focus on dominance. Those who desire resources would be better off putting themselves in situations where the dominant powers can become more dominant by giving them/putting them in charge of resources.

Given a world of AIs, I don't think the dominant AI would need a market; it could just handle everything itself.

As I understand it, libertarian paradises are basically fantasies by people who don't like the government, not realistically-achievable outcomes given the political realities.