LessWrong 2.0 Reader

[M]aybe being yourself and open works for people who happen to already be relationship-compatible. People who are not would be worse off by trying to be themselves. I think I have been burned in the past a lot by that kind of advice, although my experience is too much of an anecdote to infer an average.

I think you are maybe using a different definition of "worse off." I would submit that a relationship that is maintainable only by being inauthentic and unopen is, in the long run, significantly worse than no relationship, both because of the experience of being in it, but also because of opportunity cost.

That's different than holding some things back at the beginning, or keeping some impolite thoughts to yourself sometimes. But if your goal is a long-term partnership, you move further away from that goal by spending time and energy on someone you know you aren't compatible with.

Good q, yes, that's the vertical axis in all the figures.

How can expectations exist without roles? When everyone is free to do whatever they want to, no one can expect anything specific...

Well, we can still have general, i.e. not gender-specific expectations, such as: people should be nice and emotionally mature. Nothing wrong with that. But it seems like the traditional gender roles also provided some gender-specific "hacks", and now we don't have them.

Or you could ask which traits are valued at the dating marketplace, or more specifically at the part you are interested in. But there is no general answer anymore; it depends on what you are looking for. For example, if you want to have a traditional relationship, it would make sense to behave according to the traditional roles, and expect the same from your potential partners. Other subcultures have different rules. And I suppose most people are confused, do random things, get random results, then hopefully learn and try something different.

One confusion I have with MAD as an approach to ELK is that it seems to assume some kind of initial inner alignment. If we're flagging when the model takes actions / makes predictions for "unusual reasons", where unusual is define with respect to some trusted set, but aligned and misaligned models are behaviorally indistinguishable on the trusted set, then a model could learn to do things for misaligned reasons on the trusted set, and then use those same reasons on the untrusted set. For example, a deceptively aligned model would appear aligned in training but attempt take-over in deployment for the "same reason" (e.g. to maximize paperclips), but a MAD approach that "properly" handles out of distribution cases would not flag take over attempts because we want models to be able to respond to novel situations. 

I guess this is part of what motivates measurement tampering as a subclass of ELK - instead of trying to track motivations of the agent as reasons, we try to track the reasons for the measurement predictions, and we have some trusted set with no tampering, where we know the reasons for the measurements is ~exactly that the thing we want to be measuring. 

Now time to check my answer by rereading https://www.alignmentforum.org/posts/vwt3wKXWaCvqZyF74/mechanistic-anomaly-detection-and-elk [AF · GW]

That’s a great idea, Thank you!

And here it is: https://manifold.markets/keltan/will-there-be-a-lessonline-2025

These days companies frequently buy back shares because they can do that without having to pay taxes for that. 

This makes a lot of sense actually.

If an investor wants to invest in a couple of different markets then that investor can choose some companies in those markets and buy shares. A single company in two non-synergistic markets is just silly from this perspective, if it was instead two companies investors would be more able to specifically choose which part they like.

If the company doesn't have shares (privately held) then this incentive does not exist.

In "Capitalism and Freedom" Milton Freidman has a section about diverse companies. The book is probably dated but some aspect of the 1960's American tax system meant that if a company you owned shares in paid you a dividend you paid tax on it, but if they re-invested a profit that was not taxed. This meant that, if (cartoon example) 100% of the shareholders in Company A wanted to take their dividends and invest them in a start-up that does X, the same thing can be done more tax efficiently by Company A instead not paying any dividends and using the money to set up a new arm that does X (the new arm essentially being a separate startup company in all but name).

Freidman thought this was a reasonably serious distortion on markets and that investors should have to pay tax on re-invested profits at the same rate as dividends, to correct for it.

Perhaps some aspect of tax systems in some countries is having a similar effect, that taking some money out of my lawnmower company to invest in a chip foundry would incur more taxes than expanding the company to do both lawnmowers and computer chips.

Is there a prediction market for that?

I don't think there is, but you could make one!

Our overall best guess is that an important role of early MLPs is to act as a “multi-token embedding”, that selects^[1] the right unit of analysis from the most recent few tokens (e.g. a name) and converts this to a representation (i.e. some useful meaning encoded in an activation). We can recover different attributes of that unit (e.g. sport played) by taking linear projections, i.e. there are linear representations of attributes. Though we can’t rule it out, our guess is that there isn’t much more interpretable structure (e.g. sparsity or meaningful intermediate representations) to find in the internal mechanisms/parameters of these layers. For future mech interp work we think it likely suffices to focus on understanding how these attributes are represented in these multi-token embeddings (i.e. early-mid residual streams on a multi-token entity), using tools like probing and sparse autoencoders, and thinking of early MLPs similar to how we think of the token embeddings, where the embeddings produced may have structure (e.g. a “has space” or “positive sentiment” feature), but the internal mechanism is just a look-up table with no structure to interpret.

You may be interested in works like REMEDI and Identifying Linear Relational Concepts in Large Language Models.

In Fig 1, is the vertical axis P(world) ?