Subspace optima

post by chrisvm · 2020-05-15T12:38:32.444Z · score: 50 (19 votes) · LW · GW · 6 comments

The term "global optimum" and "local optimum" have come from mathematical terminology and entered daily language. They are useful ways of thinking in every day life. Another useful concept, which I don't hear people talk about much is "subspace optimum": A point maximizes a function not in the whole space, but in a subspace. You have to move along a different dimension than those of the subspace in order to improve. A subspace optimum doesn't have to be a local optimum either, because even a small change along the new dimension might yield improvements. If you're in a subspace optimum, this requires a different attitude to get to a global optimum, than if you're in a local optimum, which makes me think it's good for the term to be part of every day language.

Regarding how it looks subjectively:

My impression is that somewhat often when people informally use the term local optimum, they are in fact talking about a subspace optimum.

Bonus for the theoretically inclined: A local subspace optimum is one where you can improve by temporarily doing things differently along dimension X, moving around in a bigger space, while eventually ending up on a different, better, point in the same subspace.

6 comments

Comments sorted by top scores.

comment by Pongo · 2020-05-15T20:50:33.684Z · score: 7 (2 votes) · LW(p) · GW(p)

Regarding the bonus: is that well-enough known terminology that I don't risk confusing people to think I mean a local optimum in a subspace?

comment by chrisvm · 2020-05-16T11:06:33.306Z · score: 1 (1 votes) · LW(p) · GW(p)

I made up the term on the spot, so I don't think so.

comment by JustinShovelain · 2020-05-18T14:51:43.195Z · score: 6 (1 votes) · LW(p) · GW(p)

I like the distinction that you're making and that you gave it a clear name.

Relatedly, there is the method of Lagrangian multipliers for solving things in the subspace.

On a side note: there is a way to partially unify subspace optimum and local optimum by saying that the subspace optimum is a local optimum with respect to the local set of parameters you're using to define the subspace. You're at a local optimum with respect to defining the underlying space to optimize over (aka the subspace) and a local optimum within that space (the subspace). (Relatedly, moduli spaces.)

comment by romeostevensit · 2020-05-15T20:17:37.679Z · score: 5 (3 votes) · LW(p) · GW(p)

I've been using production-possibility frontier and saddle points to communicate the concept but this seems faster. Thanks!

comment by Pongo · 2020-05-15T20:49:30.021Z · score: 3 (2 votes) · LW(p) · GW(p)

I am grateful for this comment, because it made me look at this (good) post, but I have trouble parsing it (I looked basically because I like your taste)

Is it "production-possiblity" "frontier and saddle points", or "production-possiblity frontier" and "saddle points", or even production-possiblity "frontier and saddle points". My guess is the middle one, but for some reason my brain always resists reading it like that

comment by romeostevensit · 2020-05-16T03:21:15.231Z · score: 3 (2 votes) · LW(p) · GW(p)

middle

https://en.wikipedia.org/wiki/Production%E2%80%93possibility_frontier

I agree the name isn't great.