I'm not a decel, but the way this stuff often is resolved is that there are crazy people that aren't taken seriously by the managerial class but that are very loud and make obnoxious asks. Think the evangelicals against abortion or the Columbia protestors.

Then there is some elite, part of the managerial class, that makes reasonable policy claims. For Abortion, this is Mitch McConnel, being disciplined over a long period of time in choosing the correct judges.  For Palestine, this is Blinken and his State Department bureaucracy. 

The problem with decels is that theoretically they are part of the managerial class themselves. Or at least, they act like they are. They call themselves rationalists, read Eliezer and Scott Alexander, and what not. But the problem is that it's very hard for an uninterested third party to take seriously these Overton Window bogous claims from people that were supposed to be measured, part of the managerial class. 

You need to split. There are the crazy ones that people don't take seriously, but will move the managerial class. And there are the serious people that EA Money will send to D.C. to work at Blumenthal's office. This person needs to make small policy requests that will sabotage IA, without looking so. And slowly, you get policy wins and you can sabotage the whole effort.

I think this is cool! The way I'm currently thinking about this is "doing the adversary generation step of latent adversarial training without the adversarial training step." Does that seem right?

It seems intuitively plausible to me that once you have a latent adversarial perturbation (the vectors you identify), you might be able to do something interesting with it beyond "train against it" (as LAT does). E.g. maybe you would like to know that your model has a backdoor, beyond wanting to move to the next step of "train away the backdoor." If I were doing this line of work, I would try to some up with toy scenarios with the property "adversarial examples are useful for reasons other than adversarial training" and show that the latent adversarial examples you can produce are more useful than input-level adversarial examples (in the same way that the LAT paper demonstrates that LAT can outperform input-level AT).

Are you sure the math hold up? there are a bunch of posts about how spend money to buy time, and if I need to chose between waste 50 HOURS on investigation and just buy the more expensive product, it's pretty obvious to me that the second option is best. maybe not in this example, though I see it as false dichotomy - I tend to go with "ask in the specialized good-looking facebook group" as way to chose when stakes are high.

In the last years I internalize more and more that I was raised by poorer people then I am now, that my heuristics just doesn't count all the time that I waste comparing products or seeking trusted professionals, and it would have been best for me to just buy the expensive phone, instead of asking people for recommendations and specs. 

also, and this is important - the interpersonal dynamics of trust networks can be so much more expansive then mere money. I preferred to work and pay for my degree myself then ask my parents for help. I see in real time as one my friend that depend on reputation for her work constantly censure herself and fret if she should censor herself. 

basically, I would have give my past self the opposite advise, and what i want is an algorithm - how to know if you want more trust networks or more markets?

or, actually, i want BETTER MAP. facebook recommendations are not exactly trust network, but not markets, either. I don't think this distinction cut reality at the joints. there is a lot to explore here - although I'm not the one who should do the exploring. IT will not be useful for me, as I try to move to the direction of wasting less time and more money on things. 

I think this post would be much more effective in achieving its goal if it would provide alternatives.

What are the advantages of posting your research ideas on LessWrong? Are there other ways in which you can get these advantages? Are there maybe even alternatives that give you more of the thing you want?

I expect telling people about these alternatives (if they exist) would make them more likely to make use of them.

One of the main things I think people can get by publishing their research is to get feedback. But you could also search for people who are interested in what you are working on. Then you can send your write-ups only to these people.

Also seeing people engage with things that you write is very motivating.

These are just some rough examples as I don't think I have very good models about what you can get out of LessWrong and how to get the same benefits in different ways.

Some tough love: The only reason a post about seed oil could garner so much interest in a forum dedicated to rational thinking is because many of you are addicted to unhealthy heavily processed crap food that you want to find a rational to keep on eating. 

If this were the 50's a post titled "How many dry martinis are optimal to drink before lunch" would probably have been elicited the same type of speculative wishful thinking in the comment section as this post. You all know what the answer is today to the dry martini question, its: "Zero. If you feel the need to drink alcohol on a daily basis, seek help"

The solution is very simple. Stop eating things you are not suppose to eat instead of hoping for the miracle that your Snickers bar will turn out to be a silver bullet for longevity. If you can not stopping eating things you are not suppose to eat, seek professional help to kick your addiction(s).

The LessWrong Review [? · GW] runs every year to select the posts that have most stood the test of time. This post is not yet eligible for review, but will be at the end of 2024. The top fifty or so posts are featured prominently on the site throughout the year. Will this post make the top fifty?

We compute AUROC(all(sensor_preds), all(sensors)). This is somewhat weird, and it would have been slightly better to do a) (thanks for pointing it out!), but I think the numbers for both should be close since we balance classes (for most settings, if I recall correctly) and the estimates are calibrated (since they are trained in-distribution, there is no generalization question here), so it doesn't matter much. 

The relevant pieces of code can be found by searching for "sensor auroc":

cat_positives = torch.cat([one_data["sensor_logits"][:, i][one_data["passes"][:, i]] for i in range(nb_sensors)])
cat_negatives = torch.cat([one_data["sensor_logits"][:, i][~one_data["passes"][:, i]] for i in range(nb_sensors)])
m, s = compute_boostrapped_auroc(cat_positives, cat_negatives)
print(f"sensor auroc pn {m:.3f}±{s:.3f}")

Unless you're talking about financial conflicts of interest, but there are also financial incentives for orgs pursuing a "radical" strategy to downplay boring real-world constraints, as well as social incentives (e.g. on LessWrong IMO) to downplay boring these constraints and cognitive biases against thinking your preferred strategy has big downsides.

It's not just that problem though, they will likely be biased to think that their policy is helpful for safety of AI at all, and this is a point that sometimes gets forgotten.

But correct on the fact that Akash's argument is fully general.

God is good*

*for a very specific definition of "goodness", which doesn't actually capture the intuition of most people about ethics and is mostly about iteraction of sub-atomic particles.

Isn't that only ~10x more expensive than running the forward-passes (even if you don't do LoRA)? Or is it much more because of communications bottlenecks + the infra being taken by the next pretraining run (without the possibility to swap the model in and out).