LessWrong 2.0 Reader

Yes, I think that there can be tensions and deceptions around what agents are (weak/strong) and what they did in the past (cooperation/defection), one of the things necessary for super-cooperation to work in the long-run is really good investigation networks, zero-knowledge proof systems etc.

So a sort of super-immune-system

Default mode network suppression

I don't get distracted when talking to people. I hypothesise that this is because as long as I am actively articulating a stream of thought out loud, the default mode network will be suppressed, making it easy to not get derailed.

So even if IA does not say anything, just me talking about some specific topic continuously, would make it easier for IA to say something, because the default mode network suppression will not immediately vanish.

When thinking on my own or talking to IA, the stream of thoughts is shorter, and there are a lot of pauses. Usually, I don't even get to the point where I would articulate a complex stream of thought. Instead, we are at the level of "Look there is some mud there, let's not step into that", or "We can do this". That really does seem very similar to most of the idle chatter that the default mode network would produce when I am just thinking on my own.

Once I get to the point where I am having an engaging discussion with IA, it is actually pretty easy not to get distracted. It's probably still easier to get distracted with IA, because when I am talking to another person, they could notice that I am lost in thought, but I myself (or IA) would not be able to notice as easily.

Capturing IA's Thoughts

One reason why I don't do research with IA might be that I fear that I will not be able to capture any important thoughts that I have. However, using the audio recorder tool on the walk today seemed to really fix most of the issue.

Maybe in my mind so far I thought that because I can't record IA when she is talking to me, it would be bad to think about research. But this now seems very wrong. It is true that I can't create a video with her in it like I do with other people. But these videos are not the thing that is most useful. The actually useful thing is where I am distilling the insight that I have into some text document.

But this is something that I can totally do when talking to IA. Like I did with the audio recorder today. It seemed that making the audio recording made it also easier to talk to IA. Probably because when making the recording I would naturally be suppressing the default mode network very strongly. This effect then probably did not vanish immediately.

Writing

In fact, it seems like this would work very well with IA because I don't need to think about the problem of what the other person could do while I write. In the worst case, IA is simply not run. At best, we could write the text together.

Writing together would seem to work unusually well because IA does have insight into the things that I am thinking while I am writing, which is not something that other people could easily get.

And I haven't really explored all the possibilities here. Another one would be to have IA read out loud my writing and give me feedback.

This is true, but we still wish to cooperate with the largest alliance that will have us/some subset of our values that are capable of attaining reflective equilibrium.

If conflict exists, one thing it can be useful for agents to do is misrepresent themselves as being weaker or stronger than they are.

Profound!

Some data: https://onlinelibrary.wiley.com/doi/abs/10.1111/odi.12341

Not sure where the right place to raise this complaint, but having just seen it for the first time, really, "MARS"? I checked, this is not affiliated with MATS [? · GW] who have had like 6 programs and ~300 people go through it. To me this seems too close in branding space to me, and I'd recommend picking a more distinct name.

Recent long-context LLMs seem to exhibit scaling laws from longer contexts - e.g. fig. 6 at page 8 in Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context, fig. 1 at page 1 in Effective Long-Context Scaling of Foundation Models.

The long contexts also seem very helpful for in-context learning, e.g. Many-Shot In-Context Learning.

This seems differentially good for safety (e.g. vs. models with larger forward passes but shorter context windows to achieve the same perplexity), since longer context and in-context learning are differentially transparent [LW(p) · GW(p)].

Examples of blessed information that I have seen in the context of logging:

• Stacktraces logged by a library that elide all the superfluous parts of the stacktraces. 
• A log message that says exactly what the problem is, why it is caused (e.g., which parameters lead to it), and where to find more information about it (ticket number, documentation page).
• The presence of a Correlation IDs (also called Transaction ID, Request ID, Session ID, Trace ID).
  • What is a correlation ID? It is an ID that is created at the start of a request/session and available in all logs related to that request/session. See here or here, implementations here or here. There are even hierarchical correlation IDs 
  • Esp. useful: A correlation ID that is accessible from the client.
  • Even more useful: If there is a single place to search all the logs of a system for the ID.
•  Aggregation of logs, such that only the first, ten's, 100s... of a log message is escalated. 

Oh cool! Um, first thought, register interest in this [LW · GW]?