Managing Emotional Potential Energy 2024-07-10T18:20:45.640Z
The Golden Mean of Scientific Virtues 2024-07-08T17:16:21.676Z
Minimalist And Maximalist Type Systems 2024-07-05T16:25:59.448Z
Libs vs Frameworks, Middle-Level Regularities vs Theories 2024-07-04T19:01:59.440Z
Static Analysis As A Lifestyle 2024-07-03T18:29:37.384Z
Assigning Praise and Blame: Decoupling Epistemology and Decision Theory 2023-01-27T18:16:43.025Z
Lost in Innovation: The Case of Phlogiston 2023-01-20T12:19:00.603Z
Group-level Consequences of Psychological Problems 2023-01-19T09:27:13.920Z
Confusing the ideal for the necessary 2023-01-16T17:29:06.932Z
The special nature of special relativity 2023-01-09T17:30:18.412Z
Reification bias 2023-01-09T12:22:15.460Z
Nothing New: Productive Reframing 2023-01-07T18:43:35.617Z
Opportunity Cost Blackmail 2023-01-02T13:48:51.811Z
Proof as mere strong evidence 2022-12-14T08:56:26.790Z
Psychological Disorders and Problems 2022-12-12T18:15:49.333Z
Confusing the goal and the path 2022-12-12T16:42:40.508Z
Formalization as suspension of intuition 2022-12-11T15:16:44.319Z
The First Filter 2022-11-26T19:37:04.607Z
What I Learned Running Refine 2022-11-24T14:49:59.366Z
Methodological Therapy: An Agenda For Tackling Research Bottlenecks 2022-09-22T18:41:03.346Z
Refine's Third Blog Post Day/Week 2022-09-17T17:03:15.472Z
Refine's Second Blog Post Day 2022-08-20T13:01:47.190Z
No One-Size-Fit-All Epistemic Strategy 2022-08-20T12:56:23.261Z
Refine's First Blog Post Day 2022-08-13T10:23:10.332Z
Shapes of Mind and Pluralism in Alignment 2022-08-13T10:01:42.102Z
Abstracting The Hardness of Alignment: Unbounded Atomic Optimization 2022-07-29T18:59:49.460Z
Levels of Pluralism 2022-07-27T09:35:32.458Z
Robustness to Scaling Down: More Important Than I Thought 2022-07-23T11:40:03.686Z
How to Diversify Conceptual Alignment: the Model Behind Refine 2022-07-20T10:44:02.637Z
Mosaic and Palimpsests: Two Shapes of Research 2022-07-12T09:05:28.984Z
Epistemological Vigilance for Alignment 2022-06-06T00:27:43.956Z
Refine: An Incubator for Conceptual Alignment Research Bets 2022-04-15T08:57:35.502Z
AMA Conjecture, A New Alignment Startup 2022-04-09T09:43:02.739Z
Productive Mistakes, Not Perfect Answers 2022-04-07T16:41:50.290Z
Replacing Natural Interpretations 2022-03-16T13:05:19.610Z
Shotgun Book Reviews: Against Method, The Knowledge Machine and Understanding Nature 2022-03-14T08:13:13.643Z
Treat Examples as World-Building 2022-03-10T15:09:44.302Z
Scientific Wrestling: Beyond Passive Hypothesis-Testing 2022-03-07T12:01:23.943Z
The Art and Science of Intuition Pumping 2022-02-22T00:18:21.535Z
What The Foucault 2022-02-19T22:48:39.276Z
Implications of automated ontology identification 2022-02-18T03:30:53.795Z
An analogy as the midwife of thermodynamics 2022-02-16T21:34:25.799Z
Becoming Stronger as Epistemologist: Introduction 2022-02-15T06:15:04.652Z
Is ELK enough? Diamond, Matrix and Child AI 2022-02-15T02:29:33.471Z
Introducing the Principles of Intelligent Behaviour in Biological and Social Systems (PIBBSS) Fellowship 2021-12-18T15:23:26.672Z
Redwood's Technique-Focused Epistemic Strategy 2021-12-12T16:36:22.666Z
Interpreting Yudkowsky on Deep vs Shallow Knowledge 2021-12-05T17:32:26.532Z
Applications for AI Safety Camp 2022 Now Open! 2021-11-17T21:42:31.672Z
Epistemic Strategies of Safety-Capabilities Tradeoffs 2021-10-22T08:22:51.169Z
Epistemic Strategies of Selection Theorems 2021-10-18T08:57:23.109Z


Comment by adamShimi on Managing Emotional Potential Energy · 2024-07-11T20:32:57.765Z · LW · GW

I do find thinking on paper (a bit more intentional than freewriting, but the same vibe) to be particularly helpful, I agree. Just like walks.

The reasons I don't find them enough is that:

  1. They generally happen after the fact, which means that some build up happened
  2. Personally, I'm rarely able to release all the build up just through thinking on paper (happens, just rare)

Still, I find it's a good way to build emotional potential energy much slower, and to notice when you really need to have a full break/sabbaticl.

Comment by adamShimi on Managing Emotional Potential Energy · 2024-07-11T16:47:35.984Z · LW · GW

Oh, I like the neural annealing connection, I have read the post but didn't relate it to emotional potential energy, but it makes sense!

Hope you take some time to anneal away some of that potential energy soon. People consistently underestimate the negative ripples on the social web from being overstretched, as opposed to the obvious and tangible "but this thing right in front of me needs doing".

Thanks. That's the plan. ;)

Comment by adamShimi on The Golden Mean of Scientific Virtues · 2024-07-10T18:17:09.491Z · LW · GW

No worries. ;)

Comment by adamShimi on The Golden Mean of Scientific Virtues · 2024-07-09T08:30:58.369Z · LW · GW

However, when it comes to more inchoate domains like research skill, such writing does very little to help the inexperienced researcher. It is more likely that they'd simply miss out on the point you are trying to tell them, for they haven't failed both by, say, being too trusting (a common phenomenon) and being too wary of 'trusting' (a somewhat rare phenomenon for someone who gets to the big leagues as a researcher). What would actually help is either concrete case studies, or a tight feedback loop that involves a researcher trying to do something, and perhaps failing, and getting specific feedback from an experienced researcher mentoring them. The latter has an advantage that one doesn't need to explicitly try to elicit and make clear distinctions of the skills involved, and can still learn them. The former is useful because it is scalable (you write it once, and many people can read it), and the concreteness is extremely relevant to allowing people to evaluate the abstract claims you make, and pattern match it to their own past, current, or potential future experiences.


I wholeheartedly agree.

The reason why I didn't go for this more grounded and practical and teachable approach is that at the moment, I'm optimizing for consistently writing and publishing posts.

Historically the way I fail at that is by trying too hard to write really good posts and make all the arguments super clean and concrete and detailed -- this leads to me dropping the piece after like a week of attempts.

So instead, I'm going for "write what comes naturally, edit a bit to check typos and general coherence, and publish", which leads to much more abstract pieces (because that's how I naturally think).

But reexploring this topic in an in-depth and detailed piece in the future, along the lines of what you describe, feels like an interesting challenge. Will keep it in mind. Thanks for the thoughtful comment!

Comment by adamShimi on Minimalist And Maximalist Type Systems · 2024-07-06T13:03:56.914Z · LW · GW

Just sharing some vibe I've got from your.. framing! 
Minimalism ~ path ~ inside-focused ~ the signal/reward 
Maximalist ~ destination ~ outside-focused ~ the world

These two opposing aesthetics is a well-known confusing bit within agent foundation style research. The classical way to model an agent is to think as it is maximizing outside world variables. Conversely, we can think about minimization ~ inside-focused (reward hacking type error) as a drug addict accomplishing "nothing"

Feels there is also something to say with dopamine vs serotonine/homeostasis, even with deontology vs consequentialism, and I guess these two clumsy clusters mirrors each other in some way (feels isomorph by reverse signe function). Will rethink about it for now.

I see what you're pointing out, but in my head, the minimalism and maximalism that I've discussed both allow you quick feedback loops, which is generally the way to go for complex stuff. The tradeoff lies more in some fuzzy notion of usability:

  • With the minimalism approach, you can more easily iterate in your head, but you need to do more work to lift the basic concepts to the potentially more tricky abstactions you're trying to express
  • With the maximalist approach, you get affordances that are eminently practical, so that many of your needs are solved almost instantly; but you need to spend much more expertise and mental effort to simulate what's going to happen in your head during edge-cases. 

As an aside note: I'm French too, and was surprised I'm supposed to yuck maximalist aesthetic, but indeed it's consistent with my reaction reading you about TypeScript, also with my K-type brain.. Anecdotally, not with my love for spicy/rich foods ^^'

I'm obviously memeing a bit, but the real pattern I'm point out is more for "french engineering school education", which you also have, rather than mere frenchness.

Comment by adamShimi on Minimalist And Maximalist Type Systems · 2024-07-06T12:57:57.101Z · LW · GW

Interestingly, the Lean theorem prover is sometimes considered a bit of a mess type-theoretically. (an illustrative thread), but is perhaps the most popular theorem prover among mathematicians. I would say it's more on the "maximalist" side.

Didn't know this about Lean, but the fact that a maximalist option is most popular with mathematicians makes sense to me.  As someone who worked both with mathematicians and formal methods researchers (much more meta-mathematicians), the latter are much closer to programmers, in the sense that they want to build things and solve their own abstract problems, instead of necessarily wanting the most compositional machinery possible (although I still expect compositionality to be baked into the intuitions of many mathematicians).

Last I read about Rust's type system, it basically didn't have a theoretical basis, and seemed like it was just based around Graydon figuring out algorithms for getting the properties he wanted. Rust is much more popular than SML (or Haskell, though I'm not sure Haskell should really count as a 'minimalist' type system with all of its language extensions).

Rust is an interesting point in the design space. If I had to describe it quickly according to the framing above, it feels like a really pleasant fractal tradeoff between different type systems:

  • It has basically affine type but with more practical usage through borrowing (see this survey for more details)
  • It has an ML type system with algebraic datatypes (and even traits which are close to typeclasses in Haskell)

So it definitely feels more maximalist than some ML or some pure linear type system, but that's more from the combination and UX work than from a crazy "let's add this super advanced feature" rush à la TypeScript imo.

Comment by adamShimi on Minimalist And Maximalist Type Systems · 2024-07-06T12:43:55.184Z · LW · GW

It is definitely one minimalist vs maximalist dimensions ^^.

Comment by adamShimi on Static Analysis As A Lifestyle · 2024-07-05T11:50:18.222Z · LW · GW

Oh, I didn't see it actually mentioned your package. 😂

Comment by adamShimi on Static Analysis As A Lifestyle · 2024-07-04T18:57:41.522Z · LW · GW

Units / dimensional analysis in physics is really a kind of type system. I was very big into using that for error checking when I used to do physics and engineering calculations professionally.



Dimensional analysis was the first place this analogy jumped to me when reading Fly By Night Physics, because it truly used dimensions not only to check results, but also to infer the general shape of the answer (which is also something you can do in type systems, for example a function with a generic type  can only be populated by the identity function, because it cannot not do anything else than return its input).

Although in physics you need more tricks and feels to do it correctly. Like the derivation of the law of the pendulum just from dimensional analysis requires you to have the understanding of forces as accelerations to know that you can us g here.

Dimensions are also a perennial candidate for things that should be added to type systems, with people working quite hard at implementing it (just found this survey from 2018).

I invented my own weird way to do it that would allow them to be used in places where actual proper types & type-checking systems weren’t supported—like most numerical calculation packages, or C, or Microsoft Excel, etc.

I looked at the repo, and was quite confused how you did it, until I read

A complete set of independent base units (meters, kilograms, seconds, coulombs, kelvins) are defined as randomly-chosen positive floating-point numbers. All other units and constants are defined in terms of those. In a dimensionally-correct calculation, the units all cancel out, so the final answer is deterministic, not random. In a dimensionally-incorrect calculations, there will be random factors causing a randomly-varying final answer.

That's a really smart trick! I'm sure there are some super advanced cases where the units might cancel out wrongly, but in practice they shouldn't, and this let's you interface with all the random software that exists! (Modulo the run it twice, as you said, because the two runs correspond to two different drawings of the constants)

Yeah a case where this came up for me is angles (radians, degrees, steradians, etc.). If you treat radians as a “unit” subjected to dimensional analysis, you wind up needing to manually insert and remove the radian unit in a bunch of places, which is somewhat confusing and annoying.

Another interesting thing with radiants is that when you write a literal expression, it will look quite different than in degrees (involving many more instances of ), so inspection can fix many errors without paying the full price of different types.

Comment by adamShimi on Assigning Praise and Blame: Decoupling Epistemology and Decision Theory · 2023-01-27T20:05:38.257Z · LW · GW

Thanks for the pointer!

Comment by adamShimi on The Art and Science of Intuition Pumping · 2023-01-20T09:41:49.582Z · LW · GW

Oh, didn't know him!

Thanks for the links!

Comment by adamShimi on Group-level Consequences of Psychological Problems · 2023-01-20T09:41:11.632Z · LW · GW

Thanks for the comment!

I agree with you that there are situations where the issue comes from a cultural norm rather than psychological problems. That's one reason for the last part of this post, where we point out to generally positive and productive norms that try to avoid these cultural problems and make it possible to discuss them. (One of the issue I see in my own life with cultural norms is that they are way harder to discuss when in addition psychological problems compound them and make them feel sore and emotional). But you might be right that it's worth highlighting more.

In a more meta point, my model is that we have moved from societies where almost everything is considered ''people's fault" to societies where almost everything is considered "society's fault". And it strikes me that this is an overcorrection, and that actually many issues in day to day life and groups are just people's problem (here drawing from my experience of realizing in many situations that I was the problem, and in other — less common — that the norms were the problem.)

Comment by adamShimi on Confusing the ideal for the necessary · 2023-01-18T11:24:38.965Z · LW · GW

Oh, I definitely agree, this is a really good point. What I was highlighting was an epistemic issue (namely the confusion between ideal and necessary conditions) but there is also a different decision theoretic issue that you highlighted quite well.

It's completely possible that you're not powerful enough to work outside the ideal condition. But by doing the epistemic clarification, now we can consider the explicit decision of taking step to become more powerful and being better able to manage non-ideal conditions.

Comment by adamShimi on Confusing the ideal for the necessary · 2023-01-17T16:14:04.125Z · LW · GW

Good point! The difference is that the case explained in this post is one of the most sensible version of confusing the goal and the path, since there the path is actually a really good path. On the other version (like wanting to find a simple theory simply, the path is not even a good one!

Comment by adamShimi on Biology-Inspired AGI Timelines: The Trick That Never Works · 2023-01-10T20:47:03.133Z · LW · GW

In many ways, this post is frustrating to read. It isn't straigthforward, it needlessly insults people, and it mixes irrelevant details with the key ideas.

And yet, as with many of Eliezer's post, its key points are right.

What this post does is uncover the main epistemological mistakes made by almost everyone trying their hands at figuring out timelines. Among others, there is:

  • Taking arbitrary guesses within a set of options that you don't have enough evidence to separate
  • Piling on arbitrary assumption on arbitraty assumption, leading to completely uninformative outputs
  • Comparing biological processes to human engineering in term of speed, without noticing that the optimization path is the key variable (and the big uncertainty)
  • Forcing the prediction to fit within a massively limited set of distributions, biasing it towards easy to think about distributions rather than representative ones.

Before reading this post I was already dubious of most timeline work, but this crystallized many of my objections and issues with this line of work.

So I got a lot out of this post. And I expect that many people would if they spent the time I took to analyze it in detail. But I don't expect most people to do so, and so am ambivalent on whether this post should be included in the final selection.

Comment by adamShimi on What Multipolar Failure Looks Like, and Robust Agent-Agnostic Processes (RAAPs) · 2023-01-10T20:46:37.891Z · LW · GW

I was mostly thinking of the efficiency assumption underlying almost all the scenarios. Critch assumes that a significant chunk of the economy always can and does make the most efficient change (everyone replacing the job, automated regulations replacing banks when they can't move fast enough). Which neglects many potential factors, like big economic actors not having to be efficient for a long time, backlash from customers, and in general all factors making economic actors and market less than efficient.

I expect that most of these factors could be addressed with more work on the scenarios.

Comment by adamShimi on What Multipolar Failure Looks Like, and Robust Agent-Agnostic Processes (RAAPs) · 2023-01-10T20:10:21.107Z · LW · GW

I consider this post as one of the most important ever written on issues of timelines and AI doom scenario. Not because it's perfect (some of its assumptions are unconvincing), but because it highlights a key aspect of AI Risk and the alignment problem which is so easy to miss coming from a rationalist mindset: it doesn't require an agent to take over the whole world. It is not about agency.

What RAAPs show instead is that even in a purely structural setting, where agency doesn't matter, these problem still crop up!

This insight was already present in Drexler's work, but however insightful Eric is in person, CAIS is completely unreadable and so no one cared. But this post is well written. Not perfectly once again, but it gives short, somewhat minimal proofs of concept for this structural perspective on alignment. And it also managed to tie alignment with key ideas in sociology, opening ways for interdisciplinarity.

I have made every person I have ever mentored on alignment study this post. And I plan to continue doing so. Despite the fact that I'm unconvinced by most timeline and AI risk scenarios post. That's how good and important it is.

Comment by adamShimi on Reification bias · 2023-01-09T17:48:50.755Z · LW · GW

I agree that a lot of science relies on predictive hallucinations. But there are examples that come to mind, notably the sort of phenomenological compression pushed by Faraday and (early) Ampère in their initial exploration of electromagnetism. What they did amounted to vary a lot of the experimental condition and relate outcomes and phenomena to each other, without directly assuming any hidden entity. (see this book for more details)

More generally, I expect most phenomenological laws to not rely heavily on predictive hallucinations, even when they integrate theoretical terms in their formulation. That's because they are mostly strong experimental regularities (like the ideal gas law or the phenomenological laws of thermodynamics) which tend to be carried to the next paradigm with radically different hidden entities.

Comment by adamShimi on Reification bias · 2023-01-09T16:43:07.264Z · LW · GW

So reification means "the act of making real" in most english dictionaries (see here for example). That's the meaning we're trying to evoke here, where the reification bias amounts to first postulate some underlying entity that explain the phenomena (that's merely a modelling technique), and second to ascribe reality to this entity and manipulate it as if it was real.

Comment by adamShimi on Prediction Markets for Science · 2023-01-02T21:33:00.885Z · LW · GW

You use the analogy with sports betting multiple time in this post. But science and sports are disanalogical in almost all the relevant ways!

Notably, sports are incredibly limited and well-defined, with explicit rules that literally anyone can learn, quick feedback signals, and unambiguous results. Completely the opposite of science!

The only way I see for the analogy to hold is by defining "science" in a completely impoverished way, that puts aside most of what science actually looks like. For example, replication is not that big a part of acience, it's just the visible "clean" one. And even then, I expect the clarification of replication issues and of the original meaning to be tricky.

So my reaction to this proposal, like my reaction to any prediction market for things other than sports and games, is that I expect it to be completely irrelevant to the progress of knowledge because of the weakness of such tools. But I would definitely be curious of attempts to explicitly address all the ambiguities of epistemology and science through betting mechanisms. Maybe you know of some posts/works on that?

Comment by adamShimi on Proof as mere strong evidence · 2022-12-16T08:07:09.206Z · LW · GW

Agreed! That's definitely and important point, and one reason why it's still interesting to try to prove P \neq NP. The point I was making here was only that when proofs are used for the "certainty" that they give, then strong evidence from other ways is also enough to rely on the proposition.

Comment by adamShimi on Applications for Deconfusing Goal-Directedness · 2022-12-14T17:16:16.209Z · LW · GW

What are you particularly interested in? I expect I could probably write it with a bit of rereading.

Comment by adamShimi on Psychological Disorders and Problems · 2022-12-12T21:02:52.225Z · LW · GW

Hot take: I would say that most optimization failures I've observed in myself and in others (in alignment and elsewhere) boil down to psychological problems.

Comment by adamShimi on Formalization as suspension of intuition · 2022-12-12T14:23:03.580Z · LW · GW

Completely agree! The point is not that formalization or axiomatization is always good, but rather to elucidate one counterintuitive way in which it can be productive, so that we can figure out when to use it.

Comment by adamShimi on Formalization as suspension of intuition · 2022-12-12T14:22:05.517Z · LW · GW

Thanks for your thoughtful comment!

First, I want to clarify that this is obviously not the only function of formalization. I feel like this might clarify a lot of the point you raise.

But first, the very idea that formalization would have helped discover non-Euclidean geometries earlier seems counter to the empirical observation that Euclid himself formalized geometry with 5 postulates, how more formal can it get? Compared to the rest of the science of the time, it was a huge advance. He also saw that the 5th one did not fit neatly with the rest. Moreover, the non-Euclidean geometry was right there in front of him the whole time: spheres are all around. And yet the leap from a straight line to the great circle and realizing that his 4 postulates work just fine without the 5th had to wait some two millennia. 

So Euclid formalized our geometric intuitions, the obvious and immediate shape that make naturally sense of the universe. This use of formalization was to make more concrete and precise some concepts that we had but that were "floating around". He did it so well that these concepts and intuition acquired an even stronger "reality" and "obviousness": how could you question geometry when Euclid had made so tangible the first intuitions that came to your mind?

According to Bachelard, the further formalization, or rather the axiomatization of geometry, of simplifying the apparently simple concepts of points and lines to make them algebraically manipulable, was a key part in getting out of this conceptual constraint.

That being said, I'd be interested for an alternative take or evidence that this claim is wrong. ;)

In general, what you (he?) call "suspension of intuition", seems to me to be more like emergence of a different intuition after a lot of trying and failing. I think that the recently empirically discovered phenomenon of "grokking" in ML provides a better model of how breakthroughs in understanding happen. It is more of a Hegelian/Kuhnian model of phase transitions after a lot of data accumulation and processing. 

This strike me as a false comparison/dichotomy: why can't both be part of scientific progress? Especially in physics and chemistry (the two fields Bachelard knew best), there are many examples of productive formalization/axiomatization as suspension of intuition:

  • Bolzmann work that generally started from mathematical building blocks, build stuff from them, and then interpreted them. See this book for more details of this view.
  • Quantum Mechanics went through that phase, where the half-baked models based on classical mechanics didn't work well enough, and so there was an effort at formalization and axiomatization that revealed the underlying structure without as much pollution by macroscopic intuition.
  • The potential function came from a pure mathematical and formal effort to compress the results of classical mechanics, and ended up being incorporated in the core concepts of physics.

I've also found out that on inspection, models of science based on the gathering of a lot of data rarely fit the actual history. Notably Kuhn's model contradicts the history of science almost everywhere, and he makes a highly biased reading of the key historic events that he leverages.

Comment by adamShimi on Formalization as suspension of intuition · 2022-12-12T13:51:27.200Z · LW · GW

That definitely feels right, with a caveat that is dear to Bachelard: this is a constant process of rectification that repeats again and again. There is no ending, or the ending is harder to find that what we think.

Comment by adamShimi on Biases are engines of cognition · 2022-11-30T17:12:51.059Z · LW · GW

I'm confused by your confusion, given that I'm pretty sure you understand the meaning of cognitive bias, which is quite explicitly the meaning of bias drawn upon here.

Comment by adamShimi on Methodological Therapy: An Agenda For Tackling Research Bottlenecks · 2022-11-30T16:49:55.114Z · LW · GW

Thanks for your comment!

Actually, I don't think we really disagree. I might have just not made my position very clear in the original post.

The point of the post is not to say that these activities are not often valuable, but instead to point out that they can easily turn into "To do science, I need to always do [activity]". And what I'm getting from the examples is that in some cases, you actually don't need to do [activity]. There's a shortcut, or maybe just you're in a different phase of the problem.

Do you think there is still a disagreement after this clarification?

Comment by adamShimi on The First Filter · 2022-11-29T16:32:42.377Z · LW · GW

In a limited context, the first example that comes to me is high performers in competitive sports and games. Because if they truly only give a shit about winning (and the best generally do), they will throw away their legacy approaches when they find a new one, however it pains them.

Comment by adamShimi on What I Learned Running Refine · 2022-11-29T16:30:48.723Z · LW · GW

Thanks for the kind words!

I'm not aware of any such statistics, but I'm guessing that MATS organizers might have some.

Comment by adamShimi on Don't align agents to evaluations of plans · 2022-11-29T11:17:02.884Z · LW · GW

I interpret Alex as making an argument such that there is not just two vs one difficulties, but an additional difficulty. From this perspective, having two will be more of an issue than one, because you have to address strictly more things.

This makes me wonder though if there is not just some sort of direction question underlying the debate here. Because if you assume the "difficulties" are only positive numbers, then if the difficulty for the direct instillation is  and the one for the grader optimization is  , then there's no debate that the latter is bigger than the former.

But if you allow directionality (even in one dimension), then there's the risk that the sum leads to less difficulty in total (by having the  move in the opposite direction in one dimension). That being said, these two difficulties seem strictly additive, in the sense that I don't see (currently) how the difficulty of evaluation could partially cancel the difficulty of instillation.

Comment by adamShimi on Don't align agents to evaluations of plans · 2022-11-29T11:03:50.941Z · LW · GW

Thanks for taking time to answer my questions in detail!

About your example for other failure modes

Is it meant to point at the ability of the actor to make the plan more confusing/harder to evaluate? Meaning that you're pointing at the ability for the actor to "obfuscate" its plan in order to get high reward?

If so, it's not clear to me why this is valuable for the actor to do? How is it supposed to get better reward from confusion only? If it has another agenda (making paperclips instead of diamonds for example), then the obfuscation is clearly valuable to allow it to work on its main goal. But here its goal is to improve evaluation, and so confusion doesn't seem like it helps.

About the cost/competitiveness argument

I think that a values-executing AGI can also search over as many plans which actually make sense, I don't think its options are limited or anything. But it'll be generating different kinds of plans, using reflective reasoning to restrict its search to non-adversarial-to-own-values parts of plan space (e.g. "don't think about basilisks").

This is the part where I currently cannot reconstruct your model (maybe because I haven't read shard theory in a while). Because from an abstract perspective, the trick part of starting from significantly more limited plans is how to expand the range of plans without allowing adversarial ones. And I don't have a model of what mechanism you think make it easier to go from the bottom up (from few plans accepted to more plans accepted) safely rather than from the top down (from all plans accepted to less plans accepted) safely.

About the "negative result" of not even trying to do grader optimization

  1. I don't see why that should exist; any plan-inhabiting adversary wishes to fool the boundary of whatever rule you provide. 
  2. See the last 10+years of alignment researchers failing to do this. Probably wise to not spend further increments of research time on such matters, once the fault is pointed out.

But why don't you see that it should exist? What are the parts of your model that make this prediction? Because by definition, there is not a single innovation or progress that most people expected to exist. That's what it means to figure out something new. So you need an actual reason for why you're not falling into that trap of only being able to imagine what already exists.

Here are some guesses:

  • Specific scientific claim that there should be no true name/clean concept of what is adversarial for a large class of cognitive architectures (in opposition to Natural Abstraction Hypothesis I guess?)
  • Human limitation claim that even though there might be such a true name/clear concept, we have neither the time not the ability to get it right under the current conditions
  • Security mindset claim that because the actor is allowed to get arbitrary smart, you should expect that if it optimizes against you, it wins.


For the second point, what do you make of the claim that there are ridiculously few researchers who even attempted to work on the problem in the last 10 years, compared to almost all easier fields of knowledge? Do you think that this should not alter the evidence of absence you get from the absence of evidence? 

Comment by adamShimi on Corrigibility Via Thought-Process Deference · 2022-11-28T17:52:26.579Z · LW · GW

It... seems to be a significant simplification of the problem? I mean, not needing all the interpretability and surgery tools would be a bigger improvement, but that's probably not something we can have.

Why do you think so? Currently I'm seeing a couple of massive difficulties here that don't generally or necessarily appear in alternatives approaches:

  • You need to know that you're going to reach an AGI before it becomes superintelligent, or you'll waste your time training an AI that will be taken over by the competitors. Whereas many approaches don't require this.
  • You need basically perfect interpretability, compared with approaches that require no or just some interpretability capabilities
  • You need to figure out the right translation to bootstrap it, and there seem to be risks if you get it wrong.
  • You need to figure out the right thought similarity measure to bootstrap it, and there seem to be risks if you get it wrong.

Can you help me understand why you think that these strong requirements nonethless are simpler than most versions or approaches of the problem that you know about?

Comment by adamShimi on Don't align agents to evaluations of plans · 2022-11-28T17:27:24.103Z · LW · GW

The way you write this (especially the last sentence) makes me think that you see this attempt as being close to the only one that makes sense to you atm. Which makes me curious:

  • Do you think that you are internally trying to approximate your own ?
  • Do you think that you have ever made the decision (either implicitly or explicitly) to not eval all or most plans because you don't trust your ability to do so for adversarial examples (as opposed to tractability issues for example)?
  • Can you think of concrete instances where you improved your own Eval?
  • Can you think of concrete instances where you thought you improved you own Eval but then regretted it later?
  • Do you think that your own changes to your eval have been moving in the direction of your ?
Comment by adamShimi on Don't align agents to evaluations of plans · 2022-11-28T17:17:00.574Z · LW · GW

> This includes “What would this specific and superintelligent CEV-universe-simulation say about this plan?”.

> This doesn’t include (somehow) getting an AI which correctly computes what program would be recommended by AGI designers in an altruistic and superintelligent branch of humanity, and then the AI executes that program and shuts itself off without doing anything else.[5]

But isn't 1 here is at least as good as 2, since the CEV-universe-simulation could always compute X=[the program that would be recommended by AGI designers in an altruistic and superintelligent branch of humanity] then return 1 iff input-plan = 'run X then shuts itself off without doing anything else' (by doing a simple text match), 0 otherwise, so there's no chance of adversarial inputs? Not to say this is a realistic way of getting an aligned AGI, but just that your argument seems to be proving too much, if it's saying that 2 is safer/better than 1.

Is your issue here that there exist a specific CEV-universe-simulation that makes 1 just as safe as 2, by basically emulating the latter situation? If so, why do you think this is a point against Alex's claim(which strikes me more as saying "there are a lot more cases of 2. being safe than of 1.")? 

Comment by adamShimi on Don't align agents to evaluations of plans · 2022-11-28T17:07:35.513Z · LW · GW
  1. Intelligence => strong selection pressure => bad outcomes if the selection pressure is off target.
  2. In the case of agents that are motivated to optimize evaluations of plans, this argument turns into "what if the agent tricks the evaluator".
  3. In the case of agents that pursue values / shards instilled by some other process, this argument turns into "what if the values / shards are different from what we wanted".
  4. To argue for one of these over the other, you need to compare these two arguments. However, this post is stating point 2 while ignoring point 3.

One thing that is not clear to me from your comment is what you make of Alex's argument (as I see it) to the extent that "evaluation goals" are further away from "direct goals" than "direct goals" are between themselves. If I run with this, it seems like an answer to your point 4 would be:

  • with directly instilled goals, there will be some risk of discrepancy that can explode due to selection pressure;
  • with evaluation based goals, there is the same discrepancy than between directly instilled goals (because it's hard to get your goal exactly right) plus an additional discrepancy between valuing "the evaluation of X" and valuing "X".

I'm curious what you think of this claim, and if that influences at all your take.

Comment by adamShimi on Don't align agents to evaluations of plans · 2022-11-28T16:49:44.570Z · LW · GW

A few questions to better understand your frame:

  • You mostly mention two outcomes for the various diamond-maximizer architectures: maximizing the number of diamonds produced and creating hypertuned-fooling-plans for the evaluator. If I could magically ensure that plan-space only contains plans that are not hypertuned-fooling-plans (they might try, but will most likely be figured out), would you say that then grader-optimization gives us an aligned AI? Or are there other failures modes that you see?
    • Intuitively if maximizing the number of diamonds and maximizing the evaluation of the number of diamonds are not even close, I would expect multiple distinct failure modes "in-between".
  • In your response to Wei Dai, I interpret you as making an uncompetitiveness claim for grader-optimization: that it will need to pay a cost in compute for both generating and pruning the adversarial examples that will make it cost more than alternative architectures. Why do you think that this cost isn't compensated by the fact that you're searching over more plans and so have access to more "good options" too?
  • You're making strong claims about us needing to avoid as much as possible going on the route of grader optimization. Why do you expect that there is no clean/clear cut characterization of the set of adversarial plans (or a superset) that we could just forbid and then go on our merry way building grader optimizers?
Comment by adamShimi on What I Learned Running Refine · 2022-11-24T19:57:10.679Z · LW · GW

Thanks for the kind words!

  1. Are there any particular lessons/ideas from Refine that you expect (or hope) SERI MATS to incorporate?

I have shared some of my models related to epistemology and key questions to MATS organizers, and I think they're supposed to be integrated in one of the future programs. Mostly things regarding realizing the importance of productive mistakes in science (which naturally pushes back a bit from the mentoring aspect of MATS) and understanding how less "clean" most scientific progress actually look like historically (with a basic reading list from the history of science).

From the impression I have, they are also now trying to give participants some broader perspective about the field, in addition to the specific frame of the mentor, and a bunch of the lessons from Refine about how to build a good model of the alignment problem apply.

On a more general level, I expect that I had enough discussions with them that they would naturally ask me for feedback if they thought of something that seemed Refine shaped or similar.

2. Do you think there's now a hole in the space that someone should consider filling (by making Refine 2.0), or do you expect that much of the value of Refine will be covered by SERI MATS [and other programs]?

Hum, intuitively the main value from Refine that I don't expect to be covered by future MATS would come from reaching out to very different profiles. There's a non-negligeable chance that PIBBSS manages to make that work though, so not clear that it's a problem.

Note that this is also part of why Refine feels less useful: when I conceived of it, most of these programs either didn't exist or were not well-established. Part of the frustration came from having nothing IRL for non-american to join, and just no program spending a significant amount of time on conceptual alignment, which both MATS and PIBBSS (in addition to other programs like ARENA) are now fixing. Which I think is great!

Comment by adamShimi on Two reasons we might be closer to solving alignment than it seems · 2022-09-26T12:56:40.997Z · LW · GW

Thanks for explicitly writing out your thoughts in a place where you can expect strong pushback! I think this is particularly valuable.

That being said, while I completely agree with your second point (I keep telling to people who argue theory cannot work that barely 10 people worked on it for 10 years, which is a ridiculously small number), I feel like your first point is missing some key reflections on the asymmetry of capabilities vs alignment.

I don't have time to write a long answer, but I already have a post going in depth into many of the core assumptions of science and engineering that we don't expect to apply for alignment, (almost all apply or are irrelevant for capabilities, although that's not discussed explicitly in the post)

Comment by adamShimi on Losing the root for the tree · 2022-09-26T12:52:23.409Z · LW · GW

This post is amazing. Not just good, but amazing. You manage to pack exactly the lesson I needed to hear with just the right amount of memes and cheekiness to also be entertaining.

I would genuinely not be surprised if the frame in this post (and the variations I'm already adding to it) proved one of the key causal factors in me being far more productive and optimizing as an alignment researcher.

One suggestion: let's call these trees treeory of change, because that's what they are. ;)

Thanks. Really.

Comment by adamShimi on Methodological Therapy: An Agenda For Tackling Research Bottlenecks · 2022-09-24T12:24:53.288Z · LW · GW

Thanks for the kind words and useful devil's advocate! (I'm expecting nothing less from you ;p)

  1. I expect it's unusual that [replace methodology-1 with methodology-2] will be a pareto improvement: other aspects of a researcher's work will tend to have adapted to fit methodology-1. So I don't think the creation of some initial friction is a bad sign. (also mirrors therapy - there's usually a [take things apart and better understand them] phase before any [put things back together in a more adaptive pattern] phase)
    1. It might be useful to predict this kind of thing ahead of time, to develop a sense of when to expect specific side-effects (and/or predictably unpredictable side effects)

I agree that pure replacement of methodology is a massive step that is probably premature before we have a really deep understanding both of the researcher's approach and of the underlying algorithm for knowledge production. Which is why in my model, this comes quite late; instead the first step are more revealing the cached methodology to the researcher, and showing alternatives from History of Science (and Technology) to make more options and approaches credible for them.

Also looking at the "sins of the fathers" for philosophy of science (how methodologies have fucked up people across history) is part of our last set of framing questions. ;)

  1. I do think it's worth interviewing at least a few carefully selected non-alignment researchers. I basically agree with your alignment-is-harder case. However, it also seems most important to be aware of things the field is just completely missing.
    1. In particular, this may be useful where some combination of cached methodologies is a local maximum for some context. Knowing something about other hills seems useful here.
      1. I don't expect it'd work to import full sets of methodologies from other fields, but I do expect there are useful bits-of-information to be had.
    2. Similarly, if thinking about some methodology x that most alignment researchers currently use, it might be useful to find and interview other researchers that don't use x. Are they achieving [things-x-produces] in other ways? What other aspects of their methodology are missing/different?
      1. This might hint both at how a methodology change may impact alignment researchers, and how any negative impact might be mitigated.

Two reactions here:

  1. I agree with the need to find things that are missing and alternatives, which is where the history and philosophy of science works come to help. One advantage of it is that you can generally judge whether the methodology was successful or problematic in hindsight there, compared to interviews.
  2. I hadn't thought about interviewing other researchers. I expect it to be less efficient in a lot of ways than the HPS work, but I'm also now on the lookout for the option, so thanks!
  1. Worth considering that there's less of a risk in experimenting (kindly, that is) on relative newcomers than on experienced researchers. It's a good idea to get a clear understanding of the existing process of experienced researchers. However, once we're in [try this and see what happens] mode there's much less downside with new people - even abject failure is likely to be informative, and the downside in counterfactual object-level research lost is much smaller in expectation.

I see what you're pointing out. A couple related thoughts:

  1. The benefits of working with established researchers is that you have a historical record of what they did, which makes it easier to judge whether you're actually helping.
  2. I also expect helping established researchers to be easier on some dimensions, because they have more experience learning new models and leveraging them.
  3. Related to your first point, I don't worry too much about messing people up because the initial input will far less invasive than replacements of methodologies wholesale. But we're still investigating the risks to be sure we're not doing something net negative.
Comment by adamShimi on My thoughts on direct work (and joining LessWrong) · 2022-08-17T14:07:29.186Z · LW · GW

Here are some of mine:

Comment by adamShimi on Refine's First Blog Post Day · 2022-08-14T18:56:53.518Z · LW · GW

It's a charitable (and hilarious) interpretation. What actually happened is that he drafted it by mistake instead of just editing it to add stuff. It should be fine now.

Comment by adamShimi on Abstracting The Hardness of Alignment: Unbounded Atomic Optimization · 2022-07-29T20:39:56.403Z · LW · GW

You probably know better than me, but I still have this intuition that seed-AI and FOOM have oriented the framing of the problem and the sort of question asked. I think people who came to agent foundations from different routes ended up asking slightly different questions.

I could totally be wrong though, thanks for making this weakness of my description explicit!

Comment by adamShimi on Robustness to Scaling Down: More Important Than I Thought · 2022-07-23T18:57:24.840Z · LW · GW

That's a great point!

There's definitely one big difference between how Scott defined it and how I'm using it, which you highlighted well. I think a better way of explaining my change is that in Scott's original example, the AI being flawed result in some sense in the alignment scheme (predict human values and do that) to be flawed too.

I hadn't made the explicit claim in my head or in the post, but thanks to your comment, I think I'm claiming that the version I'm proposing generalize one of the interesting part of the original definition, and let it be applied to more settings.

As for your question, there is a difference between flawed and not the strongest version. What I'm saying about interpretability and single-single is not that a flawed implementation of them would not work (which is obviously trivial), but that for the reductions to function, you need to solve a particularly ambitious form of the problem. And that we don't currently have a good reason to expect to solve this ambitious problem with enough probability to warrant trusting the reduction and not working on anything else.

So an example of a plausible solution (of course I don't have a good solution at hand) would be to create sufficient interpretability techniques that, when combined with conceptual and mathematical characterizations of problematic behaviours like deception, we're able to see if a model will end up having these problematic behaviours. Notice that this possible solution requires working on conceptual alignment, which the reduction to interpretability would strongly discourage.

To summarize, I'm not claiming that interpretability (or single-single) won't be enough if it's flawed, just that reducing the alignment problem (or multi-multi) to them is actually a reduction to an incredibly strong and ambitious version of the problem, that no one is currently tackling this strong version, and that we have no reason to expect to solve the strong version with such high probability that we should shun alternatives and other approaches.

Does that clarify your confusion with my model? 

Comment by adamShimi on How to Diversify Conceptual Alignment: the Model Behind Refine · 2022-07-23T11:57:20.522Z · LW · GW

Yeah, I see how it can be confusing. To give an example, Paul Christiano focuses on prosaic alignment (he even coined the term) yet his work is mostly on the conceptual side. So I don't see the two as in conflict.

Comment by adamShimi on How to Diversify Conceptual Alignment: the Model Behind Refine · 2022-07-23T11:55:45.866Z · LW · GW

Thanks for your comment!

Probably the best place to get feedback as a beginner is AI Safety Support. They can also redirect you towards relevant programs, and they have a nice alignment slack.

As for your idea, I can give you quick feedback on my issues with this whole class of solutions. I'm not saying you haven't thought about these issues, nor that no solution in this class is possible at all, just giving the things I would be wary of here:

  • How do you limit the compute if the AI is way smarter than you are?
  • Assuming that you can limit the compute, how much compute do you give it? Too little and it's not competitive, leading many people to prefer alternatives without this limit; too much and you're destroying the potential guarantees.
  • Even if there's a correct and safe amount of compute to give for each task, how do you compute that amount? How much time and resources does it cost?
Comment by adamShimi on How to Diversify Conceptual Alignment: the Model Behind Refine · 2022-07-22T07:32:58.747Z · LW · GW

Basically the distinction is relevant because there are definitely more and more people working on alignment, but the vast majority of the increase actually doesn't focus on formulating solution or deconfusing the main notions; instead they mostly work on (often relevant) experiments and empirical questions related to alignment. 

Comment by adamShimi on How to Diversify Conceptual Alignment: the Model Behind Refine · 2022-07-22T07:30:56.988Z · LW · GW

Maybe I should have added this link. ;)

Comment by adamShimi on How to Diversify Conceptual Alignment: the Model Behind Refine · 2022-07-21T17:08:13.255Z · LW · GW

Yeah, I will be posting updates, and probably the participants themselves will post some notes and related ideas. Excited too about how it's going to pan out!