LessWrong 2.0 Reader

(I think I misinterpreted your question and started drafting another response, will reply to relevant portions of this reply there)

Yes, but

1. This update is screened off by "you actually looking at the past and checking whether we got lucky many times or there is a consistent reason". Of course, you could claim that our understanding of the past is not perfect, and thus should still update, only less so. Although to be honest, I think there's a strong case for the past clearly showing that we just got lucky a few times.
2. It sounded like you were saying the consistent reason is "our architectures are non-agentic". This should only constitute an anthropic update to the extent you think more-agentic architectures would have already killed us (instead of killing us in the next decade). I'm not of this opinion. And if I was, I'd need to take into account factors like "how much faster I'd have expected capabilities to advance", etc.

Under the anthropic principle [? · GW], we should expect there to be a 'consistent underlying reason' for our continued survival.

Why? It sounds like you're anthropic updating on the fact that we'll exist in the future, which of course wouldn't make sense because we're not yet sure of that. So what am I missing?

I think one issue is that someone can be aware about a specific worldview's existence and even consider it a plausible worldview, but still be quite bad at understanding what it would imply/look like in practice if it were true. 

For me personally, it's not that I explicitly singled out the scenario that happened and assigned it some very low probability. Instead, I think I mostly just thought about scenarios that all start from different assumptions, and that was that.

For instance, when reading Paul's "What failure looks like [AF · GW]" (which I had done multiple times), I thought I understood the scenario and even explicitly assigned it significant likelihood, but as it turned out, I didn't really understand it because I never really thought in detail about "how do we get from a 2021 world (before chat-gpt) to something like the world when things go off the rails in Paul's description?" If I had asked myself that question, I'd probably have realized that his worldview implies that there probably isn't a clear-cut moment of "we built the first AGI!" where AI boxing has relevance.

I have heard of AIXI but haven't looked deeply into it. I'm curious about it. What are some results you think are cool in this field ?

Also the remark that hyperfinite can mean smaller than a nonstandard natural just seems false, where did you get that idea from?

When I look up the definition of hyperfinite, it's usually defined as being in bijection with the hypernaturals up to a (sometimes either standard or nonstandard, but given the context of your OP I assumed you mean only nonstandard) natural $n$. If the set is in bijection with the numbers up to $n$, then it would seem to have cardinality less than $n+1$^[1].

1. ^{^}

   Obviously this doesn't hold for transfinite sizes, but we're merely considering hyperfinite sizes, so it should hold there.

Suggestion: A marker for recommended posts which are over x duration old. I was just reading this post which was recommended to me, and got half-way through before seeing it was 2 years out of date :(

https://www.lesswrong.com/posts/3S4nyoNEEuvNsbXt8/common-misconceptions-about-openai [LW · GW]

(Or maybe it's unnecessary and I'll get used to checking post dates on the algorithmic frontpage)

It's duct tapes all the way down!

Bad: "Screw #8463 needs to be reinforced."

The best: "Book a service appointment, ask them to replace screw #8463, do a general check-up, and report all findings to the central database for all those statistical analyses that inform recalls and design improvements."

I haven't watched it yet, but there is also a recent technical discussion/podcast episode about AIXI and relatedd topics with Marcus Hutter: https://www.youtube.com/watch?v=7TgOwMW_rnk