I will spend time reading posts and papers, improving coding skills as needed to run and interpret experiments, learning math as needed for writing up proofs, talking with concept-based interpretability researchers as well as other conceptual alignment researchers

I feel like this is missing the bit where you write proofs, run and interpret experiments, etc.

As a maximal goal, I might seek to test my theories about the detection of generalizable human values (like reciprocity and benevolence) by programming an alife simulation meant to test a toy-model version of agentic interaction and world-model agreement/interoperability through the fine-structure of the simulated agents.

Do you think you will be able to do this in the next 6 weeks? Might be worth scaling this down to "start a framework to test my theories" or something like that

the fine-structure of the simulated agents.

what does this mean?

alife

I think most people won't know what this word means

I plan to stress-test and further flesh out the theory, with a minimal goal of producing a writeup presenting results I've found and examining whether the assumptions of the toy models of the original post hold up as a way of examining Natural Abstractions as an alignment plan.

I feel like this doesn't give me quite enough of an idea of what you'd be doing - like, what does "stress-testing" involve? What parts need fleshing out? 

Time-bounded: Are research activities and outputs time-bounded?

• Does the proposal include a tentative timeline of planned activities (e.g., LTFF grant, scholar symposium talk, paper submission)?
• How might the timeline change if planned research activities are unsuccessful?

This part is kind of missing - I'm seeing a big list of stuff you could do, but not an indication of how much of it you might reasonably expect to do in the next 5 weeks. A better approach here would be to give a list of things you could do in those 5 weeks together estimates for how much time each thing could take, possibly with a side section of "here are other things I could do depending on how stuff goes"

Theory of Change

I feel like this section is missing a sentence like "OK here's the thing that would be the output of my project, and here's how it would cause these good effects"

[probably I also put a timeline here of stuff I have done so far?]

This is valuable for you to do so that you can get a feel for what you can do in a week, but I'm not sure it's actually that valuable to plop into the RP

Less prosaically, it's not impossible that a stronger or more solidly grounded theory of semantics or of interoperable world-models might prove to be the "last missing piece" between us and AGI; that said, given that my research path primarily involves things like finding and constructing conceptual tools, writing mathematical proofs, and reasoning about bounds on accumulating errors - and not things like training new frontier models - I think the risk/dual-use-hazard of my proposed work is minimal.

I don't really understand this argument. Why wouldn't having a better theory of semantics and concepts help people build better AIs, but still do a good job of describing what's going on in smart AIs? Like, you might think the more things you know about smart AIs, the easier it would be to build them - where does this argument break?

The thing you imply here is that it's pretty different from stuff people currently do to train frontier models, but you already told me that scaling frontier models was really unlikely to lead to AGI, so why should that give me any comfort?

Not only would a better theory of semantics help researchers detect objects and features which are natural to the AI, it would also help them check whether a given AI treats some feature of its environment or class of object as a natural cluster, and help researchers agree within provable bounds on what concept precisely they are targeting.

This part isn't so clear to me. Why can't I just look at what features of the world an AI represents without a theory of semantics?

(admittedly unlikely)

why do you think it's unlikely?

On one hand, arbitrary agents - or at least a large class of agents, or at least (proto-)AGIs that humans make - might turn out to simply already naturally agree with us on the features we abstract from our surroundings; a better-grounded and better-developed theory of semantics would allow us to confirm this and become more optimistic about the feasibility of alignment.

On the other, such agents might prove in general to have inner ontologies totally unrelated to our own, or perhaps only somewhat different, but in enduring and hazardous ways; a better theory of semantics would warn us of this in advance and suggest other routes to AGI or perhaps drive a total halt to development.

I feel like these two paragraphs are just fleshing out the thing you said earlier and aren't really needed

inner

is this word needed?

those that rely on arbitrary AGIs detecting and [settling on as natural] the same features of the world that humans do, including values and qualities important to humanity

can you give examples of such strategies, and argue that they rely on this?

Natural Abstractions framework

a link here could be nice

Worst of all, lacking such a theory means that we lack the framework and the language we'd need to precisely describe both human values - and how we'd check that a given system comprehends human values.

Maybe devote a sentence arguing for this claim

As a result, I think that conceptual alignment will be a required direction of work towards ensuring that the advent of AGI results in a desirable future for humanity among other sapient life. In particular, my perspective as a mathematician leads me to believe that just as a lack of provable guarantees about a mathematical object means that such an object can be arbitrarily unexpectedly badly behaved on features you didn't want to specify, so too could the behavior of an underspecified or imprecisely specified AGI result in arbitrarily undesirable (or even merely pathological or self-defeating) behavior along axes we didn't think to check.

IDG how this is supposed to be related to whether scaling will work. Surely if scaling were enough, your arguments here would still go thru, right?

It seems very likely (~96%) to me that scale is not in fact all that is required to go from current frontier models to AGI, such that GPT-8 (say) will still not be superintelligent and a near-perfect predictor or generator of text, just because of what largely boils down to a difference of scale and not a difference of kind or of underlying conceptual model; I consider it more likely that we'll get AGI ~30 years out but that we'll have to get alignment precisely right.

You might want to gesture at why this seems likely to you, since AFAICT this is a minority view.

All the same, it's not all that surprising that conceptual alignment generally and natural abstractions/natural semantics specifically are - maybe unavoidably - underserved subfields of alignment: the model of natural semantics I'm working off of was only officially formalized in mid-June 2024.

what's the point of this sentence? would anything bad happen if you just deleted it?

the model of natural semantics I'm working off of was only officially formalized in mid-June 2024.

is this why it isn't surprising that conceptual alignment is underserved, or an example of it being underserved? as written i feel like the structure implies the second, but content-wise it feels more like the first

both human values

both of the human values? or should this be "both human values and how we'd check that..."?

-

You use a lot of em dashes, and it's noticeable. This is a common problem in writing. I don't know a good way to deal with this, other than suggesting that you consider which ones could be footnotes, parentheses, or commas.

cannot be even reasonably sure that the measurements taken and experiments performed are telling us what we think they are

It's not clear to me why this follows. Couldn't it be the case that even without a theory of what sorts of features we expect models to learn / use, we can detect what features they are in fact using?