Hey everyone, could you spell out to me what's the issue here? I read a lot of comments that basically assume "x and y are really bad" but never spell it out. So, is the problem that:

- Giving the benchmark to OpenAI helps capabilities (but don't they have a vast sea of hard problems to already train models on?)

- OpenAI could fake o3's capabilities (why do you care so much? This would slow down AI progress, not accelerate it)

- Some other thing I'm not seeing?

expressly attempts to tarnish someone's reputation

I don't think that's accurate. The OP clearly states:

One upfront caveat. I am speaking about “Kat Woods” the public figure, not the person. If you read something here and think, “That’s not a true/nice statement about Kat Woods”, you should know that I would instead like you to think “That’s not a true/nice statement about the public persona Kat Woods, the real human with complex goals who I'm sure is actually really cool if I ever met her, appears to be cultivating.”

Decision theory as discussed here heavily involves thinking about agents responding to other agents' decision processes

My guess is that AI accelerators will have some difficult-to-modify persistent memory based on similar chips having it, but I'm not sure if it would be on the same die or not. I wrote more about how a firmware-based implementation of Offline Licensing might use H100 secure memory, clocks, and secure boot here: https://arxiv.org/abs/2404.18308 

There is one problem with this. It is not entirely clear whether an ordinary living person will talk about consciousness if he is brought up accordingly his whole life (not given any literature that mentions consciousness, never talking to him about qualia, et cetera...).

Sims are very cheap compared to space travel, and you need to know what you're dealing with in quite a lot of detail before you fly because you want to have mapped the entire space of possible negotiations in an absolutely ridiculous level of detail.

Sims built for this purpose would still be a lot lower detail than reality, but of course that would be indistinguishable from inside if the sim is designed properly. Maybe most kinds of things despawn in the sim when you look away, for example. Only objects which produce an ongoing computation that has influence on the resulting civ would need modeling in detail. Which I suspect would include every human on earth, due to small world effects, the internet, sensitive dependence on initial conditions, etc. Imagine how time travel movies imply the tiniest change can amplify - one needs enough detail to have a good map of that level of thing. Compare weather simulation.

Someone poor in Ghana might die and change the mood of someone working for ai training in Ghana, which subtly affects how the unfriendly AI that goes to space and affects alien civs is produced, or something. Or perhaps there's an uprising when they try to replace all human workers with robots. Modeling what you thought about now helps predict how good you'll be at the danceoff in your local town which affects the posts produced as training data on the public internet. Oh, come to think of it, where are we posting, and on what topic? Perhaps they needed to model your life in enough detail to have tight estimates of your posts, because those posts affect what goes on online.

But most of the argument for continuing to model humans seems to me to be the sensitive dependence on initial conditions, because it means you need an unintuitively high level of modeling detail in order to estimate what von Neumann probe wave is produced.

Still cheap - even in base reality earth right now is only taking up a little more energy than its tiny silhouette against the sun's energy output in all directions. A kardashev 2 civ would have no problem fuelling an optimized sim with a trillion trillion samples of possible aliens' origin processes. Probably superintelligent kardashev 1 even finds it quite cheap, could be less then earth's resources to do the entire sim including all parallel outcomes.

I'm not completely sure, since I was not personally involved in the relevant negotiations for FrontierMath. However, what I can say is that Tamay already indicated that Epoch should have tried harder to obtain different contract terms that enabled us to have greater transparency. I don't think it makes sense for him to say that unless he believes it was feasible to have achieved a different outcome.

Also, I want to clarify that this new benchmark is separate from FrontierMath and we are under different constraints with regards to it.

I think  this approach is reasonable for things where failure is low stakes. But I really think it makes sense to be extremely conservative about who you start businesses with. Your ability to verify things is limited, and there may still be information in vibes even after updating on the results of all feasible efforts to verify someone's trustworthiness. 

I personally found the memes funny. To address your objection:

Overall, the content she posts feels like engagement bait. It feels like it is trying to convince me of something rather than make me smarter about something. It feels like it is trying to convey feelings at me rather than facts. It feels like it is making me stupider. 

To give an analogy, it feels like PETA content. When I initially went vegan, it wasn’t PETA content that convinced me. It was Brian Tomasik content and videos of grinding male chicks. While it’s true that I am "out of distribution" so to speak, popular consensus is that PETA’s attempts at memetic content are mostly cringe. Kat Woods, why would you want to make content like that?

The goal of rationalist community is to make people smarter and more rational. Thus we have a norm: we should aim to explain and not persuade. This isn't a norm in a wider world; persuading other people to your point of view is a socially acceptable way to achieve your goals. It seems to me you are trying to enforce lesswrong norms outside of lesswrong; why? 

The goal of AI safety isn't to make people smarter, it is to prevent unsafe AI from being deployed. Conveying feelings isn't inherently bad. I would agree that manipulating people's feelings to change their beliefs contrary to facts is bad. But that is not the only possible purpose of conveying feelings. People can be moved to act by feelings, we often feel better when we know that others share our feelings, and we can get along better if we understand each other's feelings and don't hurt them, and humor is valuable in its own right. I also wouldn't say that these posts you sited are all feelings based, many of them are debunking faulty arguments that many people make. That's just valid discourse.

The goal of PETA isn't to be popular, it is to protect animals. Here are their accomplishments as listed on their website (I trust they're not lying): 

• PETA persuaded more than a dozen companies, including Pfizer and Johnson & Johnson, to make the abusive and pointless forced swim test a thing of the past. Laboratories conduct these experiments by dosing mice, rats, guinea pigs, gerbils, or hamsters with a test substance, dropping them into inescapable containers of water, and watching as the petrified animals frantically look for an escape. See other victories for animals who are used in experiments.
• In 1995 after two years of negotiations with—and more than 400 demonstrations against—the company worldwide, McDonald’s became the first fast-food chain to agree to make basic welfare improvements for farmed animals. Now, thanks largely to PETA’s outreach and persistence, you can’t visit a fast-food restaurant without seeing a vegan option, whether it’s Burger King’s or Carl’s Jr.’s animal-free burgers, Del Taco’s vegan beef burritos, or WaBa Grill’s plant-based steak bowls. The vegan revolution is here.
• Undercover investigations of pig-breeding factory farms in North Carolina and Oklahoma revealed horrific conditions and daily abuse of pigs, including the fact that one pig was skinned alive, leading to the first-ever felony indictments of farm workers. See other victories for animals who are used for food.
• After persistent campaigning by PETA U.S., other PETA entities, and our supporters around the world, Canada Goose joined the ever-growing list of top fashion brands that have sworn off fur, including Prada, Coach, Versace, Michael Kors, Balmain, Gucci, Calvin Klein, and Burberry. And we’re toppling other industries, too. After we released the results of PETA Asia’s investigation into the angora rabbit fur industry, more than 100 major brands suspended their use of the material, including Gap, H&M, Ralph Lauren, Topshop, UNIQLO, and Zara. And following the release of the first-of-its-kind undercover PETA investigation into one of the world’s largest alpaca-fleece producers, we persuaded more than 65 companies to make the compassionate decision to ban the material. See other victories for animals exploited for fashion.
• After 36 years of protests from PETA members and supporters against Ringling Bros. and Barnum & Bailey Circus, it stopped using animals in its shows. Ringling is planning its return to the big top, without animals—sending a powerful message to the entire industry and echoing what we’ve been saying for decades: Animals don’t belong in the circus or in any other form of entertainment. In a landmark case, our Endangered Species Act (ESA) lawsuit against Tiger King villain Tim Stark and Indiana roadside zoo Wildlife in Need succeeded—setting a precedent that premature separation of lion, tiger, and lion/tiger hybrid cubs and mothers; declawing; and cub-petting violate federal law. We also played an integral role in a major victory when the U.S. Department of Justice seized 69 protected big cats from Lauren and Jeff Lowe, operators of Tiger King Park in Oklahoma, and won its own ESA lawsuit against the Lowes. See other victories for animals used for entertainment.
• PETA persuaded Mobil, Texaco, Pennzoil, Shell, and other oil companies to cover their exhaust stacks after showing how millions of birds and bats had become trapped in the shafts and been burned to death. See other victories for wildlife.
• Thanks to PETA’s lengthy campaign to push PETCO to take more responsibility for the animals in its stores, the company agreed to stop selling large birds and to make provisions for the millions of rats and mice in its care. See other victories for abused companion animals.

It seems what PETA does works.

But honestly, is this content for the greater good? Are the clickbait titles causing people to earnestly engage? Are peoples’ minds being changed? Are people thinking thoughtfully about the facts and ideas being presented? 

If this looks a lot like brand memeing or PETA advocacy, I'd expect it to work about as well. Meaning at least somewhat well. What makes you think it doesn't work, apart from your own feelings? I'm not a subscriber to the subreddits and I don't know what their vibe or level of seriousness is, so the memes may indeed be out of place on some of these subreddits. I have no opinion on that. I agree it would be good if someone collected data and learned which advocacy methods are most effective.

What would I do instead 

Maybe you should do it.

to the extent this is feasible for us

Was [keeping FrontierMath entirely private and under Epoch's control] feasible for Epoch in the same sense of "feasible" you are using here?