LessWrong 2.0 Reader

Thank you for this post. 

The only thing about which I want to encourage more reflection is Have more "Move fast and break things" attitude [admittedly there is a bit of context I'm not edit-pasting here, but you do seem to favour this approach to a fair extent]. 

My gentle nudge here is based on my sense that 'moving fast and breaking things' can have pretty bad consequences if you're (collaboratively) exploring AI safety research tracks that recklessly put into circulation knowledge that can be used to increase capabilities over/without safety 'points'. 

I found the code hard to read, so here's a transcription for the next me who passes by:

troll:

U():
  if A() == "cross" & provable(False):
    return -10
  else if A() == "cross":
    return 10
  else:
    return 0

agent:

A():
  if A()="cross" proves U()=10
    return "cross"
  else if A()="not cross" proves U()=10
    return "not cross"
  else if A()="cross" proves U()=0
    return "cross"
  else if A()="not cross" proves U()=0
    return "not cross"
  else:
    return "not cross"

proof sketch:

reasoning u/u- PA
  $A = "cross"
    $\provable(A="cross" -> U=-10)
      either PA proved A()="cross" -> 10,
      or it proved A()="cross" -> 0
      in either case, PA \proves False by way of no
      number being equal to another
    \triforce \provable( (A="cross" -> -10) -> (A="cross" -> U=-10) ).
      by Lob, A="cross" -> u=-10
      so u=-10
  \triforce A="cross" => u=-10
  since we proved this in PA, the bot proves it, and proves no better utility in addition because if it did, then PA would actually be inconsistent.

(transcription revision 1)

Would you be added to a list of bad kids?

That would seem to be the "nice" outcome here, yes.

What is the typical level of shadiness of American VCs?

If you're asking that question, I claim that you already suspect the answer and should stop fighting it.

I wouldn't be surprised if within a few years the specific uniqueness of individual users of models today will be able to be identified from effectively prompt reflection in the outputs for any non-trivial/simplistic prompts by models of tomorrow.

For example, I'd be willing to bet I could spot the Claude outputs from janus vs most other users, and I'm not a quasi-magical correlation machine that's exponentially getting better.

A bit like how everyone assumed Bitcoin used with tumblers was 'untraceable' until it turned out it wasn't.

Anonymity is very likely dead for any long storage outputs no matter the techniques being used, it just isn't widely realized yet.

You can imagine different types of world models, going from very simple ones to very detailed ones. In a sense, you could perhaps think of the assumption that the input distribution is i.i.d. as a "world model". However, what is imagined is generally something that is much more detailed than this. More useful safety specifications would require world models that (to some extent) describe the physics of the environment of the AI (perhaps including human behaviour, though it would probably be better if this can be avoided). More detail about what the world model would need to do, and how such a world model may be created, is discussed in Section 3.2. My personal opinion is that the creation of such a world model probably would be challenging, but not more challenging than the problems encountered in other alignment research paths (such as mechanistic interpretability, etc). Also note that you can obtain guarantees without assuming that the world model is entirely accurate. For example, consider the guarantees that are derived in cryptography, or the guarantees derived from formal verification of airplane controllers, etc. You could also monitor the environment of the AI at runtime to look for signs that the world model is inaccurate in a certain situation, and if such signs are detected, transition the AI to a safe mode where it can be disabled.

The issue with single datapoints, at least in the context we used this for, which was building interaction graphs for the LIB papers, is that the answer to 'what directions in the layer were relevant for computing the output?' is always trivially just 'the direction the activation vector was pointing in.'

This then leads to every activation vector becoming its own 'feature', which is clearly nonsense. To understand generalisation, we need to see how the network is re-using a small common set of directions to compute outputs for many different inputs. Which means looking at a dataset of multiple activations.

And basically the trouble a lot of work that attempts to generalize ends up with is that some phenomena are very particular to specific cases, so one risks losing a lot of information by only focusing on the generalizable findings.
 

The application we were interested in here was getting some well founded measure of how 'strongly' two features interact. Not a description of what the interaction is doing computationally. Just some way to tell whether it's 'strong' or 'weak'. We wanted this so we could find modules in the network.

Averaging over data loses us information about what the interaction is doing, but it doesn't necessarily lose us information about interaction 'strength', since that's a scalar quantity. We just need to set our threshold for connection relevance sensitive enough that making a sizeable difference on a very small handful of training datapoints still qualifies.

I think this was a really poor branding choice by Altman, similarity infringement or not. The tweet, the idea of even getting her to voice it in the first place.

Like, had Arnold already said no or something?

If one of your product line's greatest obstacles is a longstanding body of media depicting it as inherently dystopian, that's not exactly the kind of comparison you should be leaning into full force.

I think the underlying product shift is smart. Tonal cues in the generations even in the short demos completely changed my mind around a number of things, including the future direction and formats of synthetic data.

But there's a certain hubris exposed in seeing Altman behind the scenes was literally trying (very hard) to cast the voice of Her in the product bearing a striking similarity to the film. Did he not watch through to the end?

It doesn't give me the greatest confidence in the decision making taking place over at OpenAI and the checks and balances that may or may not exist on leadership.

I wrote a bit about it in this comment [LW(p) · GW(p)].

I think that conceptual alignment research of the sort that Johannes is doing (and that I also am doing, which I call "deconfusion") is just really difficult. It involves skills that are not taught to people, that seems very unlikely that you'd learn by being mentored in traditional academia (including when doing theoretical CS or non-applied math PhDs), that I only started wrapping my head around after some mentorship from two MIRI researchers (that I believe I was pretty lucky to get), and even then I've spent a ridiculous amount of time by myself trying to tease out patterns to figure out a more systematic process of doing this.

Oh, and the more theoretical CS (and related math such as mathematical logic) you know, the better you probably are at this -- see how Johannes tries to create concrete models of the inchoate concepts in his head? Well, if you know relevant theoretical CS and useful math, you don't have to rebuild the mathematical scaffolding all by yourself.

I don't have a good enough model of John Wentworth's model for alignment research to understand the differences, but I don't think I learned all that much from John's writings and his training sessions that were a part of his MATS 4.0 training regimen, as compared to the stuff I described above.

Do you have a source for that? His website says:

VP and Chief AI Scientist, Facebook

Good ideas.

A gripe of mine in the same vein is that my old employer had this idea that in any public facing communication "numbers up to ten must be written in words, 11 or higher in digits". I think its a common rule in (for example) newspapers. But it leads to ludicrous sentences like "There are either nine, ten or 11 devolved administrations depending on how they are counted." It drives me completely crazy, either the whole list should be words or numerals, not a mix.