LessWrong 2.0 Reader

Insularity will make you dumber

Okay but there is another side to the issue, insularity can also have positive effects:

If you look at evolution, when a population gets stuck on an island, it starts to develop in interesting ways, maybe insularity is a necessary step to develop truly creative worldviews?

Also IIRC in "The Timeless way of building" Christopher Alexander mentions that cities should be designed as several small neighborhoods with well-defined boundaries, where people with similar background live. He also says something to the effect that the boundaries and the common background are essential to create a strong and interesting culture. So to put a positive spin on it, to some extent insularity is what allowed the rationalist community to develop its peculiar charm.

Now I also find the jargon annoying at times, and had to ask an LLM about this "epistemic status" business, but another way to see it is that this gate-keeping is a selection process that keeps people that are not a good fit out of the community, helping it maintain its coherence. But maybe we should still tone it down in favor of clarity.

I had previously noticed that the paper's classifier produced a lot of FPs/FNs and sent my findings + recommendations to Ryan G, who told me that there was a group working on improving the classifier (I assume that's you guys).  Glad to see an update on this effort!

What do you mean here by "does not mean anything"?

It seems clear to me that there's some notion of off-the-record that journalists understand.

This might vary on details, and I agree is probably not legally binding, but it does seem to mean something.

In a more peaceful world the science advanced faster and the AI already killed us?

Update from chatting with him: he said he was a just freelancer doing a year exclusively with NYT, and he wasn’t in a position to write on behalf of the NYT on the issue (e.g. around their deanonymization policies). This wasn’t satisfying to me, and so I will keep to being off-the-record.

It deliberately doesn’t assume anything especially different or weird happens, only that trend lines keep going.

Of course they are fitting an exponential curve, and only one thing happens when you do that. (Newborn on track to swallow the sun by 2040.) You can get a hyperbolic curve to fit about equally as well [citation needed] and predict negative infinity resources on Jan 2 2028. I wish they had defended this choice a bit more clearly. Like plot binomial and sigmoid best fit for comparison, to show it really does look like an exponential. (Y axis can be something arbitrary, like the price of land measured in gold.) An exponential makes sense, when an output is an input, so I would agree with it, but you can say the same thing about a puppy's cells & organs.

In reality, we observe that roughly 85% of recommendations stay the same when flipping nationality in the prompt and freezing reasoning traces. This suggests that the mechanism for the model deciding on its recommendation is mostly mediated through the reasoning trace, with a smaller less significant direct effect from the prompt to the recommendation.

Based on playing around recently with a similar setup (but only toy examples), I'm actually surprised you get only 85%, as I've only observed NDE=0 when I freeze the entire reasoning_trace. 

My just-so explanation for this was that whenever the reasoning trace includes the conclusion (that is, the bolded text in your example), then freezing the reasoning trace preserves the final conclusion. Put another way, the <recommendation> is ~deterministically determined by <reasoning>, which suggests a strong bias towards seeing low direct effects. 

If this just-so story is true, it suggests that we might need a more granular mediator than the <entire reasoning_trace>, if possible

Understood. Just there's a point of criticizing the rigid categories of decision theory, which is shown by showing the common sense in between. This is my question.

That's the idea of the quote at the end from Berlin; but one of the premises of ai research is that thinking is reducible to formula and categories.

Even if they are using the term to denote two different things, they can agree on connotation. Meaning isn't exhausted by denotation (reference, extension). Semantic differences are ubiquitous, but so are differences in background assumptions,in presumed ontology.

This hardly seems an argument against the one in the shortform, namely

Neither a physicalist nor a functionalist theory of consciousness can reasonably justify a number like this. Shrimp have 5 orders of magnitude fewer neurons than humans, so whether suffering is the result of a physical process or an information processing one, this implies that shrimp neurons do 4 orders of magnitude more of this process per second than human neurons. The authors get around this by refusing to stake themselves on any theory of consciousness.

If the original authors never thought of this that seems on them.