Subspace optima
post by Chris van Merwijk (chrisvm) · 2020-05-15T12:38:32.444Z · LW · GW · 7 commentsContents
7 comments
The term "global optimum" and "local optimum" have come from mathematical terminology and entered daily language. They are useful ways of thinking in every day life. Another useful concept, which I don't hear people talk about much is "subspace optimum": A point maximizes a function not in the whole space, but in a subspace. You have to move along a different dimension than those of the subspace in order to improve. A subspace optimum doesn't have to be a local optimum either, because even a small change along the new dimension might yield improvements. If you're in a subspace optimum, this requires a different attitude to get to a global optimum, than if you're in a local optimum, which makes me think it's good for the term to be part of every day language.
- When you're in a local optimum, you have to do something quite different from what you're doing to improve.
- When you're in a subspace optimum, you have to notice dimensions along which you could be doing things differently that you didn't even notice before, but small changes along those new dimensions might already help. You're applying constraints to yourself that you could let go.
Regarding how it looks subjectively:
- The phrase: "am I in a local optimum?" generates curiosity about whether you maybe should undertake a quite different plan from the one you're taking now. (Should I do a different project, rather than make local changes to the project I'm taking?)
- The phrase: "am I in a subspace optimum?" generates curiosity about whether you maybe are not noticing (possibly small) changes you could be making across dimensions you haven't been considering. (Should I optimize/adjust the way I'm doing my project across different dimensions/variables than the ones I've been optimizing over so far?)
My impression is that somewhat often when people informally use the term local optimum, they are in fact talking about a subspace optimum.
Bonus for the theoretically inclined: A local subspace optimum is one where you can improve by temporarily doing things differently along dimension X, moving around in a bigger space, while eventually ending up on a different, better, point in the same subspace.
7 comments
Comments sorted by top scores.
comment by johnswentworth · 2022-01-15T04:27:21.842Z · LW(p) · GW(p)
This was a concept which it never occurred to me that people might not have, until I saw the post. Noticing and drawing attention to such concepts seems pretty valuable in general. This post in particular was short, direct, and gave the concept a name, which is pretty good; the one thing I'd change about the post is that it could use a more concrete, everyday example/story at the beginning.
comment by Pongo · 2020-05-15T20:50:33.684Z · LW(p) · GW(p)
Regarding the bonus: is that well-enough known terminology that I don't risk confusing people to think I mean a local optimum in a subspace?
Replies from: chrisvm↑ comment by Chris van Merwijk (chrisvm) · 2020-05-16T11:06:33.306Z · LW(p) · GW(p)
I made up the term on the spot, so I don't think so.
comment by romeostevensit · 2020-05-15T20:17:37.679Z · LW(p) · GW(p)
I've been using production-possibility frontier and saddle points to communicate the concept but this seems faster. Thanks!
Replies from: Pongo↑ comment by Pongo · 2020-05-15T20:49:30.021Z · LW(p) · GW(p)
I am grateful for this comment, because it made me look at this (good) post, but I have trouble parsing it (I looked basically because I like your taste)
Is it "production-possiblity" "frontier and saddle points", or "production-possiblity frontier" and "saddle points", or even production-possiblity "frontier and saddle points". My guess is the middle one, but for some reason my brain always resists reading it like that
Replies from: romeostevensit↑ comment by romeostevensit · 2020-05-16T03:21:15.231Z · LW(p) · GW(p)
middle
https://en.wikipedia.org/wiki/Production%E2%80%93possibility_frontier
I agree the name isn't great.
comment by JustinShovelain · 2020-05-18T14:51:43.195Z · LW(p) · GW(p)
I like the distinction that you're making and that you gave it a clear name.
Relatedly, there is the method of Lagrangian multipliers for solving things in the subspace.
On a side note: there is a way to partially unify subspace optimum and local optimum by saying that the subspace optimum is a local optimum with respect to the local set of parameters you're using to define the subspace. You're at a local optimum with respect to defining the underlying space to optimize over (aka the subspace) and a local optimum within that space (the subspace). (Relatedly, moduli spaces.)