.CLI's Shortform

post by .CLI (Changbai Li) · 2024-06-06T22:02:51.701Z · LW · GW · 7 comments

Contents

7 comments

7 comments

Comments sorted by top scores.

comment by .CLI (Changbai Li) · 2024-12-04T07:15:35.582Z · LW(p) · GW(p)

How did safety engineering get invented for different disciplines, and how do their invention relate to engineering and theory?

Inspired by davidad's tweets: 1, 2, 3

It seems commonsense that a deeper (theoretical) understanding helps both engineering as well as safety engineering. Which one do you think does theory help more? And which development helped grow theory research more?

My intuition is that:

  1. First we started building something by trial-and-error, empirical results.
  2. We formulated some safety best practices. But they are all heuristics from the trial-and-error.
  3. Then we started gaining theoretical understanding of what we are doing.
  4. Only then do we become able to advance "safety engineering".
  5. At the same time, we also get much better at building that thing - much better at engineering.

How well does this mesh with real-life? In the bridges' case, safety engineering was invented separately, well after we understood how to build bridges - and well after we built a lot of bridges. The pioneers in safety engineering oft have formal math background. This seems to match the intuition above.

That said -

  • We did build a lot of bridges, and a lot of them failed, before safety engineering came about. And how much did theories for safety engineering help with bridge capability?
  • Did the field advance by novel theory works? Or was it more about the application of existing theories?
  • Related to that question is: did safety engineering require an entirely different set of theories that have little to do with bridge capability? (This seems obviously true to me: for example, environmental wear-and-tear and the process of metal rusting does not affect capability, but we need to understand them for safety.)
comment by .CLI (Changbai Li) · 2024-09-01T04:39:52.930Z · LW(p) · GW(p)

(ramblingly) Does the No Free Lunch Theorem imply that there's no one single technique that would always work for AGI alignment? Initial thought is probably not, because the theorem states that the performance of all optimization algorithms are identical across all possible problems. However,  AGI alignment is a subset of these problems.

Replies from: Seth Herd, MakoYass
↑ comment by Seth Herd · 2024-09-01T16:16:09.289Z · LW(p) · GW(p)

See Steve Byrne's take on the no free lunch theorem.

No is the answer to "does the NFL theorem prove x" for x we care about I'm pretty sure.

Replies from: sharmake-farah
↑ comment by Noosphere89 (sharmake-farah) · 2024-09-01T17:34:49.832Z · LW(p) · GW(p)

I basically agree with @davidad [LW · GW]'s comment on the NFL, because basically all of the time, you're aiming to find not just a good solution (which satisficing is), but an optimal solution, and thus what it really proves is that in the worst case, intelligence is necessarily and sufficiently look-up tables, and you need to resort to brute-force search, which is wildly intractable:

 

https://www.lesswrong.com/posts/yTvBSFrXhZfL8vr5a/?commentId=N3avtTM3ESH4KHmfN [LW · GW]

 

I also disagree with @jsteinhardt [LW · GW] on the min-max/optimal solution being unacceptably bad, assuming perfect specification of what we value (this is obviously not a safe assumption in practice, but is probably safe from a philosophical/theoretical perspective.)

↑ comment by mako yass (MakoYass) · 2024-09-01T06:22:20.467Z · LW(p) · GW(p)

[just now learning about the no free lunch theorem] oh nooo, is this part of the reason so many AI researchers think it's cool and enlightened to not believe in highly general architectures?

Because they either believe the theorem proves more than it does or because they're knowingly performing an aestheticised version of it by yowling about how LLMs can't scale to superintelligence (which is true, but also not a crux).

Replies from: Changbai Li
↑ comment by .CLI (Changbai Li) · 2024-09-05T21:43:04.266Z · LW(p) · GW(p)

is this part of the reason so many AI researchers think it's cool and enlightened to not believe in highly general architectures

I do hear No Free Lunch theorem get thrown around when an architecture fails to solve some problem which its inductive bias doesn't fit. But I think it's just thrown around as a vibe.

comment by .CLI (Changbai Li) · 2024-06-06T22:02:52.079Z · LW(p) · GW(p)

I think there should be more effort into researching the limits of controllability for self-improving machines. That aspect of rapid self improvement seems pretty important to me since it's there regardless of which architecture we use to get to the singularity. If the singularity is dangerous no matter how we get there, or how aligned our first try is, then, [clears throat and raises sign] don't build AGI?