LessWrong 2.0 Reader

The constant bound isn't not that relevant just because of the in principal unbounded size, it also doesn't constrain the induced probabilities in the second coding scheme much at all. It's an upper bound on the maximum length, so you can still have the weightings in codings scheme B differ differ in relative length by a ton, leading to wildly different priors

And of the encoding schemes that remain on the table, virtually all of them will behave identically with respect to the description lengths they assign to "natural" versus "unnatural" optimization criteria.

I have no idea how you're getting to this, not sure if it's claiming a formal result or just like a hunch. But I disagree both that there is a neat correspondence between a system being physically realizable and its having a concise implementation as a TM. Even granting that point, I don't think that nearly all or even most of these physically realisable systems will behave identically or even similarly w.r.t. how they assign codes to "natural" optimization criteria

Thanks Hastings,

I think at that time you could reason much better if you could recognize that the separation between left and right was not natural.

I think you're saying it was easier in the past to see unorthodox or contradictory views within parties because the wings were more clearly delineated. I'd agree, it was a divided time, but a less chaotic divided time.

The effective left right split is mono-factor: you are right exactly in proportion to your personal loyalty to one Donald J. Trump

Absolutely, it's also bizarre regarding his tariff policy which is wholly anti-free market, that's a point the left didn't pick up on (because of the chaos I imagine) that was obvious to me. As a left-wing (pro-taxation) person myself who also believes in free markets, his approach is so anti-thetical to my own views, as if he took the last good idea on the right (free markets), and abandoned that in order to create a party based on all the bad ideas. This sort of contrarianism is something I've read Steven Pinker write about as a loyalty test (to despots and cult leaders)—the inducement to followers to knowingly lie or act contrary to their own interests as a statement of loyalty to each other through joint faith in the dear leader.

He’d walk on over to nearby industry labs with candy and a sales pitch for why they should use his services. He primarily targeted top, Nobel-prize-winning research groups

and

Plasmidsaurus has historically done very little ‘traditional’ marketing — no brochures, few cold reach-outs

seem to be a bit contradictory?

It already happens indirectly. Most digital money transfers are things like credit card transactions. For these, the credit card company takes a percentage fee and pays the government tax on its profit.

I think that the main conclusion is that large amounts of compute are still necessary in reasoning well OOD, and even though o1 is doing a little reasoning, it's a pretty small scale search (usually seconds or minutes of search.), which means that the fact that it's a derivative GPT-4o model matters a lot for it's incapacity, as it's pretty low on the compute scale compared to other models.

One consideration is the government wouldn't want to encourage (harder-to-tax) cash transactions.

Performance after post-training degrades if behavior gets too far from that of the base/SFT model (see Figure 1). Solving this issue would be an entirely different advancement from what o1-like post-training appears to do. So I expect that the model remains approximately as smart as the base model and the corresponding chatbot, it's just better at packaging its intelligence into relevant long reasoning traces.

Additional data points:

o1-preview and the new Claude Sonnet 3.5 both significantly improved over prior models on SimpleBench.

The math, coding, and science benchmarks in the o1 announcement post:

I highly agree with almost all of these points, and those are very consistent with my observation. As I am still relatively new to lesswrong, one big observation (based on my experience) I still see today, is disconnected concepts, definitions, and or terminologies with the academic language. Sometimes I see terminology that already exists in academia and introducing new concepts with the same name may be confusing without using channels academics are used to. There are some terms that I try to search on google for example, but the only relevant ones are from lesswrong or blogposts (which I still then read personally). I think this is getting better - in one of the recent conference reviews, I saw significant increase in submissions in AI safety working on X risks. 

Another point as you have mentioned is the reverse ingestion of papers from academia; there are rich papers in interpretability as you have mentioned for example, and some concrete confusion I saw from professors or people already in that field is that why there is feels like a lack of connection with these papers or concepts, even though they seems to be pretty related.

About actions - many people that I see are concerned about AI safety risks in my usual professional group are people who are concerned about or working in current intentional risks like misuse. Those are actually also real risks and have already started (CSAM, deep fake porn with real people's faces, privacy, potential bio/chem weapons), and needs to be worked on as well. It is hard to stop working on them, and transition directly to X risks.

However, I do think it is beneficial to keep merging the academic and AI safety groups, which I see are already underway with examples like more papers, and some PhD positions on AI Safety, industry positions etc; This will increase awareness of AI safety, and as you have mentioned the interests in the technical parts are shared, as they could be applied potentially to many kinds of safety, and hopefully not that much on capabilities (though sometimes not separable).

by saying their name aloud: [...] …but it’s a lot more difficult to use active recall to remember people’s names.

I'm confused, isn't saying their name in a sentence an example of active recall?