Decision theory as discussed here heavily involves thinking about agents responding to other agents' decision processes

My guess is that AI accelerators will have some difficult-to-modify persistent memory based on similar chips having it, but I'm not sure if it would be on the same die or not. I wrote more about how a firmware-based implementation of Offline Licensing might use H100 secure memory, clocks, and secure boot here: https://arxiv.org/abs/2404.18308 

There is one problem with this. It is not entirely clear whether an ordinary living person will talk about consciousness if he is brought up accordingly his whole life (not given any literature that mentions consciousness, never talking to him about qualia, et cetera...).

Sims are very cheap compared to space travel, and you need to know what you're dealing with in quite a lot of detail before you fly because you want to have mapped the entire space of possible negotiations in an absolutely ridiculous level of detail.

Sims built for this purpose would still be a lot lower detail than reality, but of course that would be indistinguishable from inside if the sim is designed properly. Maybe most kinds of things despawn in the sim when you look away, for example. Only objects which produce an ongoing computation that has influence on the resulting civ would need modeling in detail. Which I suspect would include every human on earth, due to small world effects, the internet, sensitive dependence on initial conditions, etc. Imagine how time travel movies imply the tiniest change can amplify - one needs enough detail to have a good map of that level of thing. Compare weather simulation.

Someone poor in Ghana might die and change the mood of someone working for ai training in Ghana, which subtly affects how the unfriendly AI that goes to space and affects alien civs is produced, or something. Or perhaps there's an uprising when they try to replace all human workers with robots. Modeling what you thought about now helps predict how good you'll be at the danceoff in your local town which affects the posts produced as training data on the public internet. Oh, come to think of it, where are we posting, and on what topic? Perhaps they needed to model your life in enough detail to have tight estimates of your posts, because those posts affect what goes on online.

But most of the argument for continuing to model humans seems to me to be the sensitive dependence on initial conditions, because it means you need an unintuitively high level of modeling detail in order to estimate what von Neumann probe wave is produced.

Still cheap - even in base reality earth right now is only taking up a little more energy than its tiny silhouette against the sun's energy output in all directions. A kardashev 2 civ would have no problem fuelling an optimized sim with a trillion trillion samples of possible aliens' origin processes.

I'm not completely sure, since I was not personally involved in the relevant negotiations for FrontierMath. However, what I can say is that Tamay already indicated that Epoch should have tried harder to obtain different contract terms that enabled us to have greater transparency. I don't think it makes sense for him to say that unless he believes it was feasible to have achieved a different outcome.

Also, I want to clarify that this new benchmark is separate from FrontierMath and we are under different constraints with regards to it.

I think  this approach is reasonable for things where failure is low stakes. But I really think it makes sense to be extremely conservative about who you start businesses with. Your ability to verify things is limited, and there may still be information in vibes even after updating on the results of all feasible efforts to verify someone's trustworthiness. 

I personally found the memes funny. To address your objection:

Overall, the content she posts feels like engagement bait. It feels like it is trying to convince me of something rather than make me smarter about something. It feels like it is trying to convey feelings at me rather than facts. It feels like it is making me stupider. 

To give an analogy, it feels like PETA content. When I initially went vegan, it wasn’t PETA content that convinced me. It was Brian Tomasik content and videos of grinding male chicks. While it’s true that I am "out of distribution" so to speak, popular consensus is that PETA’s attempts at memetic content are mostly cringe. Kat Woods, why would you want to make content like that?

The goal of rationalist community is to make people smarter and more rational. Thus we have a norm: we should aim to explain and not persuade. This isn't a norm in a wider world; persuading other people to your point of view is a socially acceptable way to achieve your goals. It seems to me you are trying to enforce lesswrong norms outside of lesswrong; why? 

The goal of AI safety isn't to make people smarter, it is to prevent unsafe AI from being deployed. Conveying feelings isn't inherently bad. I would agree that manipulating people's feelings to change their beliefs contrary to facts is bad. But that is not the only possible purpose of conveying feelings. People can be moved to act by feelings, we often feel better when we know that others share our feelings, and we can get along better if we understand each other's feelings and don't hurt them, and humor is valuable in its own right. I also wouldn't say that these posts you sited are all feelings based, many of them are debunking faulty arguments that many people make. That's just valid discourse.

The goal of PETA isn't to be popular, it is to protect animals. Here are their accomplishments as listed on their website (I trust they're not lying): 

• PETA persuaded more than a dozen companies, including Pfizer and Johnson & Johnson, to make the abusive and pointless forced swim test a thing of the past. Laboratories conduct these experiments by dosing mice, rats, guinea pigs, gerbils, or hamsters with a test substance, dropping them into inescapable containers of water, and watching as the petrified animals frantically look for an escape. See other victories for animals who are used in experiments.
• In 1995 after two years of negotiations with—and more than 400 demonstrations against—the company worldwide, McDonald’s became the first fast-food chain to agree to make basic welfare improvements for farmed animals. Now, thanks largely to PETA’s outreach and persistence, you can’t visit a fast-food restaurant without seeing a vegan option, whether it’s Burger King’s or Carl’s Jr.’s animal-free burgers, Del Taco’s vegan beef burritos, or WaBa Grill’s plant-based steak bowls. The vegan revolution is here.
• Undercover investigations of pig-breeding factory farms in North Carolina and Oklahoma revealed horrific conditions and daily abuse of pigs, including the fact that one pig was skinned alive, leading to the first-ever felony indictments of farm workers. See other victories for animals who are used for food.
• After persistent campaigning by PETA U.S., other PETA entities, and our supporters around the world, Canada Goose joined the ever-growing list of top fashion brands that have sworn off fur, including Prada, Coach, Versace, Michael Kors, Balmain, Gucci, Calvin Klein, and Burberry. And we’re toppling other industries, too. After we released the results of PETA Asia’s investigation into the angora rabbit fur industry, more than 100 major brands suspended their use of the material, including Gap, H&M, Ralph Lauren, Topshop, UNIQLO, and Zara. And following the release of the first-of-its-kind undercover PETA investigation into one of the world’s largest alpaca-fleece producers, we persuaded more than 65 companies to make the compassionate decision to ban the material. See other victories for animals exploited for fashion.
• After 36 years of protests from PETA members and supporters against Ringling Bros. and Barnum & Bailey Circus, it stopped using animals in its shows. Ringling is planning its return to the big top, without animals—sending a powerful message to the entire industry and echoing what we’ve been saying for decades: Animals don’t belong in the circus or in any other form of entertainment. In a landmark case, our Endangered Species Act (ESA) lawsuit against Tiger King villain Tim Stark and Indiana roadside zoo Wildlife in Need succeeded—setting a precedent that premature separation of lion, tiger, and lion/tiger hybrid cubs and mothers; declawing; and cub-petting violate federal law. We also played an integral role in a major victory when the U.S. Department of Justice seized 69 protected big cats from Lauren and Jeff Lowe, operators of Tiger King Park in Oklahoma, and won its own ESA lawsuit against the Lowes. See other victories for animals used for entertainment.
• PETA persuaded Mobil, Texaco, Pennzoil, Shell, and other oil companies to cover their exhaust stacks after showing how millions of birds and bats had become trapped in the shafts and been burned to death. See other victories for wildlife.
• Thanks to PETA’s lengthy campaign to push PETCO to take more responsibility for the animals in its stores, the company agreed to stop selling large birds and to make provisions for the millions of rats and mice in its care. See other victories for abused companion animals.

It seems what PETA does works.

But honestly, is this content for the greater good? Are the clickbait titles causing people to earnestly engage? Are peoples’ minds being changed? Are people thinking thoughtfully about the facts and ideas being presented? 

If this looks a lot like brand memeing or PETA advocacy, I'd expect it to work about as well. Meaning at least somewhat well. What makes you think it doesn't work, apart from your own feelings? I'm not a subscriber to the subreddits and I don't know what their vibe or level of seriousness is, so the memes may indeed be out of place on some of these subreddits. I have no opinion on that. I agree it would be good if someone collected data and learned which advocacy methods are most effective.

What would I do instead 

Maybe you should do it.

to the extent this is feasible for us

Was [keeping FrontierMath entirely private and under Epoch's control] feasible for Epoch in the same sense of "feasible" you are using here?

In particular, the brain tries to compress the reward stream by modeling it as some (noisy) signal generated from value-assignments to patterns in the brain's environment. So e.g. the brain might notice a pattern-in-the-environment which we label "sports car", and if the reward stream tends to spit out positive signals around sports cars (which aren't already accounted for by the brain's existing value-assignments to other things), then the brain will (marginally) compress that reward stream by modeling it as (partially) generated from a high value-assignment to sports cars. See the linked posts for a less compressed explanation, and various subtleties.

I'm not sure why we can't just go with an explanation like... imagine a human with zero neuroplasticity, something like a weight-frozen LLM. Its behaviors will still tend to place certain attractor states into whatever larger systems it's embedded within, and we can call those states the values of one aspect of the system. Unfreeze the brain, though, resuming the RL, and now the set of attractor states the human embeds in whatever its surroundings are will change. You just won't be able to extract as much info about what the overall unfrozen system's values are, because you won't be able to just ask the current human what it would do in some hypothetical situation and get decent answers (modulo self-deception etc.), because the RL could possibly change what would be the frozen-human's values ~arbitrarily between now and the situation you're describing to them coming to pass.

Uh, I'm not sure if that makes what I have in mind sufficiently obvious, but I don't personally feel very confused about this question; if that explanation leaves something to be desired, lmk and I can take another crack at it.

Causal interactions? The answer is rather trivial. In order for the separate meaningless Planck moments of the brain's existence to be able to combine into "granules of qualia" that have integrity in time, they must be connected by something. It is usually assumed that there are causal relationships behind this, which can be likened to computational processes.

But many transhumanists, it seems to me, show some duality of thinking here. They agree that two adjacent computational cycles of the brain's work can be combined into one sensation. But they refuse to assume the existence of more extended configurations of this kind.

Why? Well, there may be two motives here: correct and not quite correct.

On the one hand, we have here a special case of the anthropic principle. It can be argued that the anthropic principle inevitably forces the individual Planck moments of my brain's existence to merge into my current sense of self. But the anthropic principle will not necessarily force my current sense of self to merge with my future sense of self in the same way.

On the other hand, many transhumanists want to believe in the ease of implementing projects like "mind uploading." If the life of consciousness does not represent a single track, then "mind uploading" will be much easier to implement. Therefore, many people like to believe in a kind of Buddhism, where I-now exists, but I-chronoblock does not exist.

Which of the motives drives you more is up to you to decide.