LessWrong 2.0 Reader

Process supervision seems like a plausible o1 training approach but I think it would conflict with this:

We believe that a hidden chain of thought presents a unique opportunity for monitoring models. Assuming it is faithful and legible, the hidden chain of thought allows us to "read the mind" of the model and understand its thought process. For example, in the future we may wish to monitor the chain of thought for signs of manipulating the user. However, for this to work the model must have freedom to express its thoughts in unaltered form, so we cannot train any policy compliance or user preferences onto the chain of thought.

I think it might just be outcome-based RL, training the CoT to maximize the probability of correct answers or maximize human preference reward model scores or minimize next-token entropy. 

Sure!

Are you familiar with Kelly betting? The point of maximizing log expectation instead of pure expectation isn't because happiness grows on a logarithmic scale or whatever, it's for the sake of maximizing long-term expected value. This kills off making bets where "0" is on the table (as log(0) is minus infinity); whether or not that's appropriate is still an interesting topic for discussion because, as you mentioned, x-risks exist anyway

I think AI safety isn't as much a matter of government policy as you seem to think. Currently, sure. Frontier models are so expensive to train only the big labs can do it. Models have limited agentic capabilities, even at the frontier.

But we are rushing towards a point where science makes intelligence and learning better understood. Open source models are getting rapidly more powerful and cheap.

In a few years, the yrend suggests that any individual could create a dangerously powerful AI using a personal computer.

Any law which fails to protect society if even a single individual chooses to violate it once... Is not a very protective law. Historical evidence suggests that occasionally some people break laws. Especially when there's a lot of money and power on offer in exchange for the risk.

What happens at that point depends a lot on the details of the lawbreaker's creation. With what probability will it end up agentic, coherent, conscious, self-improvement capable, escape and self-replication capable, Omohundro goal driven (survival focused, resource and power hungry), etc...

The probability seems unlikely to me to be zero for the sorts of qualities which would make such an AI agent dangerous. Then we must ask questions about the efficacy of governments in detecting and stopping such AI agents before they become catastrophically powerful.

I find this comment flippant and unworthy of a community like LessWrong. First of all, you’re denying the politics of millions of earnest people, many as educated and gifted as you, and second of all, you’re equating a 21st century democratically steered market economy with the totalitarian central planning of 20th century Stalinism. You’re right that no one wants that.

I love this! Thank you for the feedback.

We could definitely build some more plot into the narration engine. Right now it's a pretty simple concept but I love this direction

You’re absolutely right, the post is light on details. To answer a few of your points: I don’t have a deep understanding of housing market dynamics beyond the bad deals and pressures I’ve heard about from many different people, especially in the Bay Area. If we were to develop this into a full proposal for public consumption it would include an analysis of how housing subsidy on both the demand and supply side would affect real outcomes. However, that’s somewhat beside the point, as that analysis has nothing to do with the soundness of the system as a whole, and actively denies the good it’s hoping to promulgate.

The debit card system is much less like food stamps than it is like dynamic UBI with constraints. You may have missed the part when a new corps of inspector-accountants validate businesses before they qualify to participate in the program. Once they do, they get a widget that adds a unique nonce to their transaction strings that the system validates. This solves a lot of problems you mention, also, the dynamic and constrained nature of the cards solve many of the issues people have with UBI: that people would spend on trivialities and non-essentials, and that it wouldn’t be enough to make a difference economically. Could someone buy $500 of alcohol with their five $100 retail transactions a week and drink themselves to death? Sure. That’s their prerogative, and their community Target worker (or wherever, just not the liquor store) could ask them if they’re ok the second time if they believe that shouldn’t happen. Further, if that’s not conscionable by a majority of people, the system could include disallowing drugs and alcohol from approved stores.

It seems common for people trying to talk about AI extinction to get hung up on whether statements derived from abstract theories containing mentalistic atoms can have objective truth or falsity values. They can. And if we can first agree on such basic elements of our ontology/epistemology as that one agent can be objectively smarter than another, that we can know whether something that lives in a physical substrate that is unlike ours is conscious, and that there can be some degree of objective truth as to what is valuable [not that all beings that are merely intelligent will necessarily pursue these things], it in fact becomes much more natural to make clear statements and judgments in the abstract or general case, about what very smart non-aligned agents will in fact do to the physical world.

Why does any of that matter for AI safety? AI safety is a matter of public policy. In public policy making, you have a set of preferences, which you get from votes or surveys, and you formulate policy based on your best objective understanding of cause and effect. The preferences don't have to be objective, because they are taken as given. It's quite different to philosophy, because you are trying to achieve or avoid something, not figure out what something ultimately is. You do t have to answer Wolfram's questions in their own terms, because you can challenge the framing.

And if we can first agree on such basic elements of our ontology/epistemology as that one agent can be objectively smarter than another,

It's not all that relevant to AI safety, because an AI only needs some potentially dangerous capabilities. Admittedly, a lot of the literature gives the opposite impression.

that we can know whether something that lives in a physical substrate that is unlike ours is conscious,

You haven't defined consciousness and you haven't explained how . It doesn't follow automatically from considerations about intelligence. And it doesn't follow from having some mentalistic terms in our theories.

and that there can be some degree of objective truth as to what is valuable

there doesn't need to be. You don't have to solve ethics to set policy.

Posts as nodes -- what's beautiful about LessWrong

I'm new to this site as a writer (and a writer in general), and I read LW's user guide, [LW · GW]to think more clearly about what kind of articles are expected and about why people are here. Direct quote:

LessWrong is a good place for someone who:

• values curiosity, learning, self-improvement, figuring out what's actually true (rather than just what you want to be true or just winning arguments)
• will change their mind or admit they're wrong in response to compelling evidence or argument
• wants to work collaboratively with others to figure out what's true
• likes acknowledging and quantifying uncertainty and applying lessons from probability, statistics, and decision theory to your reasoning
• is nerdy and interested in all questions of how the world works and who is not afraid to reach weird conclusions if the arguments seem valid
• likes to be pedantic and precise, and likes to bet on their beliefs
• doesn't mind reading a lot 

There is a style of communication and thought, that summarizes the spirit of most of these. It's when your presentation is structured like a graph of dependencies.

"I believe x because of y" is much better than "I believe x"

Changing your mind is built into this type of writing, it says that they believe x because they believe that y => x, and/or that they believe x to the degree that y is true. It allows collaboration, because someone else can point out other implications of y, or that y isn't true, or that y => x doesn't hold, etc., and they have to change their mind. Or put in another way:

When you make the graph explicit, including the edges, the audience can judge the way things are connected, in addition to the conclusion.

It's like open-sourcing the code (where your conclusion is like the "app").

(and you can arrive at weird conclusions for free, since you're simply following the graph)

But what about exploratory thinking?

You simply take your exploratory post, identify the parts that are solid, refactor the post into self-contained things with explicit paths of reasoning, and you take those posts as nodes to reason about and speculate about!

(maybe a better way to put it is that it encourages factoring out solid elements of an idea or conclusion, and that you have the Quick Takes feature for doing exploration?? idk this second part is way less solid lol)

I'd be curious about hearing your intuition re " i'm further guessing that most structures basically have 'one way' to descend into them"