LessWrong 2.0 Reader

Sorry, this doesn't make sense to me. The boundary doesn't need to be smooth in an absolute sense in order to exist and be learnable (whether by neural nets or something else). There exists a function from business plans to their profitability. The worry is that if you try to approximate that function with standard ML tools, then even if your approximation is highly accurate on any normal business plan, it's not hard to construct an artificial plan on which it won't be. But this seems like a limitation of the tools; I don't think it's because the space of business plans is inherently fractally complex and unmodelable.

From our state of knowledge about consciousness it's indeed not impossible that modern LLMs are conscious. I wouldn't say it's likely, I definitely wouldn't say that they are as likely to be conscious as uploaded humans. But the point stands. We don't know for sure and we lack proper way to figure it out.

Previously we could've vaguely point towards Turing test, but we are past this stage now. Behavioral analysis of a model at this point is mostly unhelpful. A few tweaks can make the same LLM that previously confidently claimed not to be conscious, to swear that it's conscious and is suffering. So what a current LLM says about the nature of its consciousness gives us about 0 bit of evidence.

This is another reason to stop making bigger models and spend a lot of time figuring out what we have already created. At some point we may create a conscious LLM, won't be able to tell the difference and it would be a moral catastrophe. 

It would seem to me that in this world brains would be much more expensive (or impossible) to copy. Which is worth talking about, because there are designs in our own era for very efficient very dense neural networks that have that same quality. They can be trained, but the weights can't be accessed.

Traditionally, those techniques are focused on what the model is outputting, not what the model’s underlying motivations are. But I haven’t read all the literature. Am I missing something?

It's confusing to me as well, perhaps because different people (or even the same person at different times) emphasize different things within the same approach, but here's one post [AF · GW] where someone said, "It is important that the overseer both knows which action the distilled AI wants to take as well as why it takes that action."

I like to phrase it as "the path to simplicity involves a lot of detours." Yes, Newtonian mechanics doesn't account for the orbit of Mercury but it turned out there was an even simpler, more parsimonious theory, general relativity, waiting for us.

Scale basically solves this too, with some other additions (not part of any released version of MJ yet) really putting a nail in the coffin, but I can't say too much here w/o divulging trade secrets. I can say that I'm surprised to hear that SD3 is still so much worse than Dalle3, Ideogram on that front - I wonder if they just didn't train it long enough?

They put too much emphasis on high frequency features, suggesting a different inductive bias from humans.

This was found to not be true at scale! It doesn't even feel that true w/weaker vision transformers, seems specific to convnets. I bet smaller animal brains have similar problems.

I get it if you're worried about leaks but I don't get how it could be a hard engineering problem — just share API access early, with fine-tuning

Fine-tuning access can be extremely expensive if implemented naively and it's plausible that cheap (LoRA) fine-tuning isn't even implemented for new models internally for a while at AI labs. If you make the third party groups pay for it than I suppose this isn't a problem, but the costs could be vast.

I agree that inference access is cheap.

What are the two other mechanisms of action?

Did SBF or Mao Zedong not have a pointer to the right values, or had a right pointer but made mistakes due to computational issues (i.e., would have avoided causing the disasters that they did if they were smarter and/or had more time to think)? Both seem possible to me, so I'd like to understand how the QACI approach would solve (or rule out) both of these potential problems:

1. If many humans don't have pointers to right values, how to make sure QACI gets a pointer from humans who have a pointer to the right values?
2. How to make sure that AI will not make some catastrophic mistake while it's not smart enough to fully understand the values we give it, while still being confident enough in its guesses of what to do in the short term to do useful things?

Moral uncertainty is an area in philosophy with ongoing research, and assuming that AI will handle it correctly by default seems unsafe, similar to assuming that AI will have the right decision theory by default.

I see that Tasmin Leake also pointed out 2 above as a potential problem, but I don't see anything that looks like a potential solution at QACI table of contents [LW · GW].