Riffing on the agent type

post by Quinn (quinn-dougherty) · 2022-12-08T00:19:38.054Z · LW · GW · 3 comments

Contents

      Preliminaries
  Selection and continuation
    Remark: quantifiers are continuations
      Preliminaries
      Exercises:
    Remark: distributions are a special case of generalized quantifiers
      Preliminaries
      Consume a valuation and produce an expectation
      Δ forms a monad
    Remark: convert selections into continuations/quantifiers
      Attainability
      Exercise
    Wrapping the codomain
      Example: powerset
      Exercise (harder than previous)
    Wrapping the codomain of the domain
      Exercise
    Wrapping the whole domain
  Modifying the agent signature
    Investigation: continuation is to Δ as selection is to what?
      Preliminaries
      Rambling about Δ∗
      Exercise
      What about ΔC∗X for metric spaces X?
      Lastly, Δ∗∘C
      Conjecture: attainability survives the transportation to the custom ≤⊸
    Investigation: J[Δ]O
    Investigation: J(Δ)S
    Rescue attempt: the objective interpretation
    Rescue: the subjective interpretation
      Preliminaries
      A subjective agent is a tuple with a protocol
      Selection product?
  Conclusion
  References
None
3 comments

Much is owed to Diffractor for Giry-pilling me at Alignable Structures, I had been struggling with type-driven expected value previously.

Epistemic status: took a couple days off from my master plan [LW · GW] to think about John's selection theorems call to action.

We would like [LW · GW] a type signature of agency. Scott Garrabrant [LW · GW] provides $(A\to O)\to A$ as a first approximation. You can choose one of two ideas here: 1. that an agent simply takes a belief about how actions $A$ turn into outcomes $O$ and returns a recommended action, or 2. that an agent takes underlying configurations of reality (containing information about how actions lead to outcomes) and tends to perform certain actions. Notice that $O$ happens to be for "outcome", "observation", and even "ontology", which is nice. This signature is widely discussed in the monad literature.

Scott wrote that $\to$ primarily means causal influence and secondarily means functions. I will be mostly ignoring the causal influence idea, and I think instead of thinking of the signature from an objective perspective of it being a transcription of the underlying reality, I want to think of it from a subjective perspective of it being an assistant for implementation engineers. I think we should take a swing at being incredibly straightforward about what we mean by the type signature of agency: when I say that a type $\tau$ is the type signature of agency, I mean that if we have programs that are admitted by $\tau$ then those programs are doing all the things that interest me about agents (i.e., at $\tau =(A\to B)\to A$, if we instantiate particular non-atomic propositions $A$ and $B$ that interact with the outside world in such a way that we can obtain proofs of $(A\to B)\to A$ (which we can't do in general) in some way, then those proofs are doing all the things that interest me about agents).

In my view, the first idea (involving "belief") can be called a subjective interpretation of the type signature, and we shall explore some adjustments to make this story better, while the second idea (involving "base reality") can be called an objective interpretation of the type signature, and we shall not explore philosophical controversies around saying that a type is "in" reality rather than in a model.

I will ultimately conclude that I am not equipped to flesh out the objective interpretation, and give a subjective interpretation such that an agent is not one selection function, but a pair of selection functions. In particular, $A=InsA\oplus EpiA$ (an agent is made up of an instrumental part and an epistemic part).

In the post, heading number one is an infodump about stuff I've been reading, and setup of some tooling. Heading number two is applications to agency.

Preliminaries

• $\mapsto$ denotes implementation of terms and $\to$ denotes signature of types. $x\mapsto e$ is an alternative to $\lambda x.e$.
• $Type$ is the type of types, which you can define via structural induction like propositional logic; the only important part for us today is $\forall AB:Type,A\to B:Type$, and I'm handwaving the equipping of preorders to arbitrary members of $Type$ very fast and loose, and I'll handwave some other stuff implying that a properly structural induction story would be really messy if I actually worked out the details.
• In addition to being a $Type$-constructor, $\to$ is also a $P$-constructor (for the type of propositions $P$).
• Arrows $\to$ associate to the right (see currying). $A\to B\to C=A\to (B\to C)\ne (A\to B)\to C$.
• We sometimes write $B^A$ instead of $A\to B$ (alluding to the arrow type's associated counting problem).
• When we say a function $M:Type\to Type$ is a monad we mean that it comes to us equipped with a function $\eta :X\to MX$ (called a return) and a function $\mu :MMX\to MX$ (called a flatten) which agree to some laws.
  • Example: let $M:=list$ setting $\eta$ to the construction of a singleton and $\mu$ to the removal of nesting information.
  • Exercise: fixing an $A:Type$, one of $M:=B\mapsto B^A$ or $M:=B\mapsto A^B$ forms a monad. Pick one and set its $\eta$ and $\mu$ (Solution^[1]).

Selection and continuation

The agent type is widely discussed in the monad literature.

Fixing an outcome type $S$, $J_S:=X\mapsto (X\to S)\to X:Type\to Type$ is called the selection monad, and its friend $K_S:=X\mapsto (X\to S)\to S:Type\to Type$ is called the continuation monad.

Remark: quantifiers are continuations

Preliminaries

• $B:=\{true,false\}$, or the type of two nullary construcors.
• $R$ is the complete ordered field.

The story of $B$-valued or $B$-interpreted logics goes like this. For any $X:Type$,

$$(\forall x:X)(\exists x:X):K_{B}X$$

In other words, a quantifier takes a predicate (typed $X\to B$) and returns a valuation of the predicate under different conditions. $\forall x:X$ is the element of $K_{B}X$ that says "the predicate is true all over $X$", $\forall x:X,kx$ (or we may write it point-free as $\forall k$) is $true$ if and only if $kx$ is always $true$ regardless of $x$. $\exists x:X$ is the element of $K_{B}X$ that says "the predicate is true at least once over $X$", the point-free $\exists k$ is $true$ if and only if you can provide at least one $x$ such that $kx$ is $true$.

The literature likes to call continuations generalized quantifiers, where your "truth values" can take on arbitrary type. The story of quantifiers can be updated to $P$ for a richer type of propositions such that not everything is decidable.

Exercises:

• Think of distinguished primitives in reinforcement learning theory; is there either a selection or a continuation story one of them? (Solution^[2]).
• Name a distinguished primitives from calculus or analysis; is there a selection or continuation story of it? (Solution^[3]).

Remark: distributions are a special case of generalized quantifiers

Preliminaries

• Recall that for each $y\in B$, you can construct a constant function $\tilde{y}:=x\mapsto y:A\to B$ by throwing out the $x$.
• A ${\le }_X:X\to X\to P$ is a reflexive and transitive relation.
• Recall that an $\alpha :A\to B$ is monotonic when, having a ${\le }_A$ and a ${\le }_B$, $\forall xy:A,x{\le }_{A}y\to \alpha x{\le }_B\alpha y$.
• Let ${\le }_{B^A}:=\alpha \mapsto \beta \mapsto \forall x:A,\alpha x\le \beta x:B^A\to B^A\to P$.
• A map $\alpha :B^A$, when $A$ and $B$ are drawn from some underlying field $F$, is linear whenever $\forall kl:F,\forall xy:A,\alpha (kx+ly)=k\alpha x+l\alpha y$.

Consume a valuation and produce an expectation

A particular way of strengthening or filtering $K_R$ (quantifiers generalized to valuations in $R$) is to require linearity, monotonicity, and the sending of constant functions to a neutral scalar. For arbitrary types $A,B$ and for types $C$ equipped with some multiplicative structure involving a neutral, we will write $B^A\stackrel{\le }{⊸}C$ to describe the functions $B^A\to C$ but only keeping the ones that are monotonic, linear, and that send constants in $B^A$ to the multiplicative neutral in $C$ (conventionally, $⊸$ pronounced "lollipop" or "lolli" denotes linearity). Letting $R$ play the roles of $B$ and $C$, define $$\Delta :=X\mapsto R^X\stackrel{\le }{⊸}R:Type\to Type$$

In other words, a distribution is just a continuation term that knows how to turn a valuation (an $X\to R$, i.e. a random variable) into an expectation (where the expectation abides linearity, monotonicity, and the sending of constants to $1$).

$$\forall X:Type,\forall \mu ,E_{\mu }:=\alpha \mapsto {\int }_X\alpha d\mu :\Delta X$$

where I'm being lazy about the measure theory needed to actually compute terms, however, we see that measure theory doesn't really emerge at the type level.

I'm thinking of distributions as a subset of these $R$-valued quantifiers because I want to eventually think about utilities, and I'm still pretty sure the utility codomain is going to be $R$ all the time.

$\Delta$ forms a monad

The settings of $\eta$ and $\mu$ along with the lawfulness proofs are in this coq file, written a few weeks ago before I knew anything about the selection and continuation literature. (This is not surprising, as we knew that $K_R$ forms a monad, and the substitution of the second $\to$ for $\stackrel{\le }{⊸}$ only deletes maps and doesn't add any potential violators).

Remark: convert selections into continuations/quantifiers

$$\forall AB:Type,\overline{.}:=ϵ\mapsto k\mapsto k\left(ϵk\right):J_{B}A\to K_{B}A$$

In other words, if $ϵ$ is a selection then $\overline{ϵ}$ is a continuation.

Attainability

Presume a $AB:Type$. Suppose I have a $k:K_{B}A$. $k$ is called attainable when it's preimage under $\overline{.}$ is nonempty. In other words, $k$ is attainable if and only if $\exists ϵ:J_{B}A,\forall \alpha :B^A,k\alpha =\alpha \left(ϵ\alpha \right)$. In that case, we may say "$ϵ$ attains $k$".

Notice that from the existence half of the functionality predicate, we get a free existence proof of a continuation/quantifier for every selection. To believe that some continuations are unattainable is to believe that $\overline{.}$ is not surjective.

Exercise

• Recall the solutions to previous exercises 1 and 2. What is the attainability relationship between them, if any?^[4]

Wrapping the codomain

Fix a $F:Type\to Type$ and a $S:Type$. Define

$$J_S^F:=X\mapsto (X\to S)\to FX:Type\to Type$$ $$K_S^F:=X\mapsto (X\to S)\to FS:Type\to Type$$

Example: powerset

Denote $P$ as the function that confiscates a type and rewards the powerset of that type. In other words, $P:=X\mapsto X\to B:Type\to Type$ (where an $\alpha :PX$ is interpreted $x"\in "\alpha$ if and only if $\alpha x=true$).

We call the items of $J_S^P$ multi-valued selections and items of $K_S^P$ multi-valued quantifiers.

Exercise (harder than previous)

• can you re-obtain monadicity for multi-valued selection?
• can you re-obtain monadicity for multi-valued continuation?
• write down multi-valued attainment^[5]

Wrapping the codomain of the domain

We may additionally like to use maps $F:Type\to Type$ to goof off with transforming the codomain of the input map.

$$J_S^{\left(F\right)}:=X\mapsto (X\to FS)\to X:Type\to Type$$ $$K_S^{\left(F\right)}:=X\mapsto (X\to FS)\to S:Type\to Type$$

Exercise

• again, can you re-obtain monadicity for $J_S^{\left(P\right)}$? For $K_S^{\left(P\right)}$?

Wrapping the whole domain

Having maps $F:Type\to Type$, and since $X\to S:Type$, we also might enjoy transforming the whole input map type.

$$J_S^{\left[F\right]}:=X\mapsto F(X\to S)\to X:Type\to Type$$ $$K_S^{\left[F\right]}:=X\mapsto F(X\to S)\to S:Type\to Type$$

Modifying the agent signature

Recall the agent interpretation of selection. We fix an outcome type $O$ and an action type $A$ and we reason about $J_{O}A=(A\to O)\to A$. Recall that there are two cases: a subjective case in which items $A\to O$ are beliefs, and an objective case in which items $A\to O$ are actual configurations of reality. In the subjective case, an agent turns a model of reality into a recommended action (the term hardcodes its notion of utility or whatever). In the objective case, the world has configurations, and an agent can be trusted to tend toward the actual configuration over time, using it to (again relying on hardcoded utility data) select actions.

Investigation: continuation is to $\Delta$ as selection is to what?

We obtained $\Delta$ by replacing the rightmost $\to$ in the definition of $K_{R}X$ with my custom $\stackrel{\le }{⊸}$. Let's goof around with performing the same replacement in $J_{R}X$.

$${\Delta }_{\ast }:=X\mapsto R^X\stackrel{\le }{⊸}X:\text{"Type"}\to Type$$

Recall that $\stackrel{\le }{⊸}$ implies that it's codomain supports linearity, monotonicity, and multiplicative neutrality, so we know that the domain of ${\Delta }_{\ast }$ isn't "really" just $Type$ (hence the scare quotes), whereas the domain of $\Delta$ was truly the unconstrained type $Type$. So it may be difficult now to be sure of the preservation of monadicity.

Preliminaries

• A monoidal preorder is a preorder with a monoid attached. If you start with $(P,\le )$ such that $\le$ is reflexive and transitive, and you find an associative $\otimes :P\to P\to P$ that has a distinguished neutral element $ϵ$, and you know $\forall abcd:P,a\le c\to b\le d\to a\otimes b\le c\otimes d$, then you have the monoidal preorder $(P,\le ,ϵ,\otimes )$.
  • From any set $A$ you can construct a monoidal preorder $(PA,\subseteq ,A,\cap )$ where $\subseteq$ and $\cap$ are from set theory. Validate this, if you like.

Rambling about ${\Delta }_{\ast }$

How do we interpret this? In the agent case, actions are playing the role of $X$, which immediately suggests that we'll only have the class of continuous action spaces, so we can try $R$. But ${\Delta }_{\ast }R=R^R\stackrel{\le }{⊸}R=\Delta R$, which feels maybe problematic or vacuous. Possibly problematic, because I don't know how the theory of random variables adjusts to the bare real line (as opposed to a collection of subsets). Possibly vacuous, because I don't know any particular terms typed $R\to R$ (other than $x\mapsto x$ or ones with fairly strong conditions like increasingness) that I would expect to correspond with some foggy coherence notion for valuations in the back of my mind. Moreover, what should we think of collapsing the very distinction between selection and continuation, by setting $S=X$? $(X\to X)\to X$ isn't provable in the logic interpretation (unless I'm missing some coinductive black magic resolving loops), which is a hint that we're barking up the wrong tree. My gut isn't telling me ${\Delta }_{\ast }[0,1]$ would be any better.

We could of course support the $\stackrel{\le }{⊸}$ requirements on the codomain by putting a monoidal preorder on $B^X$ (namely setting $P:=B^X$, $\le :=\subseteq$, $ϵ:=X$, and $\otimes :=\cap$), which wouldn't work for entirely arbitrary $X:Type$ but would work if you could interpret the scaling of a subset (like $X$ is a single suit out of a deck of cards, the valuation $\nu$ of a subset is the total number of pips across all the cards in the subset, and scalar $k$ hits it by doing some operation on that valuation, like $k:=p\mapsto \lfloor |k\nu p|\rfloor$). Fix an $X$ that you can interpret in this way. Then, try ${\Delta }_{\ast }{B}^X=\left(\right(X\to B)\to R)\stackrel{\le }{⊸}X\to B$. In other words, if I have an $X$-generated multi-valued "selection distribution" $E:{\Delta }_{\ast }{B}^X$, then for every valuation of a subset $\nu :B^X\to R$, $E\nu$ is a kind of expected subset, or it's something the agent can proactively search for like $\arg min$ or $\arg max$. Perhaps you could even interpret/implement it like "if $\nu$ is my complete account of what a subset is worth to me, then $E$ fixes an amount of optimization power I'm going to throw at steering the future into particular subsets over others, and $E\nu$ denotes the sort of place I would end up if I applied that much optimization to my values (insofar as landing at an actual optima implies that possibly unbounded optimization power was deployed)".

Exercise

• Check that monotonicity, linearity, and the sending of constants to $1$ (in this case $X$ because it's the monoidal neutral) works with something like my deck-of-cards choice of $X$.

What about ${\Delta }_{\ast }^{C}X$ for metric spaces $X$?

Loosening up the pedantry a little, because the actual type-driven story would get too hairy, let's by fiat admit $C[0,1]:Type$, so we can take the subset of $R^{[0,1]}$ that just has continuous functions in it. You shall indulge me if I utilize $C:Type\to Type$ without properly saying that the domain is just the types interpretable as or isomorphic to uninterrupted intervals, whatever.

$${\Delta }_{\ast }^C=X\mapsto (X\to R)\stackrel{\le }{⊸}CX=X\mapsto (X\to R)\stackrel{\le }{⊸}X\stackrel{C}{\to }R$$

A modus ponens with a little decoration with conditions like the linearity/monotonicity/sending constants to $1$ and continuity. What does it mean?

It could mean the environment actually giving the agent a reward for taking action $X$, though it's a simpler story than the one in standard reinforcement learning theory, especially e.g. POMDPs.

Lastly, ${\Delta }_{\ast }\circ C$

The idea of rigging "scalar multiplication" to my deck of cards semantics was uncomfortable. The following, however, has a perfectly natural notion of linearity (alongside order and the idea of a $1$).

$${\Delta }_{\ast }\circ C=X\mapsto (CX\to R)\stackrel{\le }{⊸}CX=X\mapsto \left(\right(X\stackrel{C}{\to }R)\to R)\stackrel{\le }{⊸}X\stackrel{C}{\to }R$$

Selections over continuous functions (taking valuations of continuous functions as inputs and returning continuous functions as output) sounds like a kind of learning over "metavalues", when the continuous functions are interpreted as utilities, then the $\arg max:({\Delta }_{\ast }\circ C)X$ knows how to take the utility of a utility function (which is metautility) and choose the one that maximizes metautility.

This of course restricts that the action type be equipped with a metric.

Conjecture: attainability survives the transportation to the custom $\stackrel{\le }{⊸}$

$\forall X:\text{"Type"},\overline{.}:{\Delta }_{\ast }X\to \Delta X$ should be provable. Indeed, it's just a domain restriction on the original $\overline{.}$, so this conjecture is in the bag.

Investigation: $J_O^{\left[\Delta \right]}$

$$J_S^{\left[\Delta \right]}:=X\mapsto \Delta (X\to O)\to X:Type\to Type$$

This isn't quite the subjective approach I'm looking for. Mapping from uncertainty over valuations to actions seems kinda from the perspective of social choice theory, where the difference in opinion across the population is captured by not being able to know a precise point estimate of a valuation, having to turn a distribution over valuations of actions into an action.

Investigation: $J_S^{\left(\Delta \right)}$

$$J_S^{\left(\Delta \right)}:=X\mapsto (X\to \Delta S)\to X:Type\to Type$$

This looks to me the most like "the agent turns models/beliefs into actions".

Let's unfold $\Delta$.

$$J_S^{\left(\Delta \right)}=X\mapsto (X\to (R^X\stackrel{\le }{⊸}R\left)\right)\to X:Type\to Type$$

The general pattern of "terms such that the input is $X$ into quantifiers and the output is $X$" might mean that terms are hardcoded predicates which can select values of $X$ to get a desired result depending on whichever quantifier shows up. We will not work with the unfolded version in what follows.

Rescue attempt: the objective interpretation

In the objective interpretation of the type signature of agency, an agent is a term that turns a configuration reality could be in (specifically the information about how actions lead to outcomes) into an action.

In my rescue operation, objectivity is not pure: we will see that I've installed a subjectivity (i.e. learning) layer as an implementation detail. Think of it like the difference between a lemma and a theorem; at the lemma level, there's subjectivity, while if the theorem level doesn't open up black boxes it may not notice subjectivity. Put another way, the challenge of the rescue operation is to tell a compellingly full story (which ought to oblige the term to empiricism under uncertainty) without resorting to $\Delta$.

The "lemma" will be a term $\varphi :J_R{O}^A$. Its inputs $O^A\to R$ are loss functions which come equipped with real-world data hardcoded into them. These loss functions make sense of the gap between a map and a territory, here focusing on action-output relations, i.e. they take a notion of how actions turn into outcomes and they score how accurate it is. Then for such a loss function $l:O^A\to R$, $\varphi l:A\to O$. (And if you want the function that constructs $l$s, you need ontology to describe that function's domain). Since this is the objective point of view, we interpret $\varphi l$'s codomain as the literal outcomes in the world, indeed $\varphi l$ is the gears by which perturbations from agents effect things. (Warning: here be monsters [LW · GW]) if we say that in order to implement an agent you need to provide a $\varphi$, and $\varphi l$ describes the literal gears of the world and isn't a conditional forecast (like "our best guess at time $t$ is that action $x$ will transition the world into state $\left({\varphi }_{t}l\right)x$"), then I don't see how an agent is remotely computational.

Equipped like so, with any proof $ϵ:J_{O}A$ hardcoded by some humans or learned by some ML model, and some $l:O^A\to R$ provided by a stakeholder/principal, $(ϵ\circ \varphi )l$ is the action the proof would like to take. But since $\varphi$ is a blackbox, $(ϵ\circ \varphi )l$ is as good as an axiom, i.e., the blackboxness propagates out and it can't be actually written at the low level. A configuration of the world $\left(\right(\varphi l)\circ ϵ\circ \varphi )l$ has a similar problem.

It's plausible to me that infrabayesian physicalism or factored sets provide a way forward, but I'm not going to grok either of those today. (The first time I read "Saving Time" just as now, I was confused about "the future effects the past" because of the determinism/nondeterminism question, i.e. I get that forecasts of or distributions over the future effect the past, but I don't get how the actual future effects the past).

I'm marking this rescue operation as a failure, owing to the restriction against invoking $\Delta$.

Rescue: the subjective interpretation

Preliminaries

• We will use pair types, of one constructor (namely $(\cdot ,\cdot ):=a\mapsto b\mapsto (a,b):A\to B\to A\times B$) and two destructors (namely ${\pi }_1:=(a,b)\mapsto a:A\times B\to A$ and ${\pi }_2:=(a,b)\mapsto b:A\times B\to B$), and we assume associativity of $n$-ary or nested products.

A subjective agent is a tuple with a protocol

In the subjective interpretation of the type signature of agency, an agent is a term that knows how to turn a belief about how actions turn into outcomes into an action. Since beliefs are emphasized, and beliefs are uncertain, we will allow ourselves liberal use of the $\Delta$ operator. The following approach is based on the failed rescue of the objective interpretation.

Fix a type $A$ of actions and a type $O$ of outcomes. We consider proofs $\psi :N\to J_R(A\to \Delta O)$ where items $f:A\to \Delta O$ are conditional forecasts that accept an action $x$ and report, with uncertainty, a belief about what will happen if it does $x$. As the domain of $\varphi$ above, for any $t:N$ the domain of $\psi t$ is loss functions, each of which considers a conditional forecast $f$ and scores it for calibration, accuracy, whatever. We write ${\psi }_t$ instead of $\psi t$.

A stakeholder or principal encodes observations at time $t$ of the world by hardcoding data into their choice of loss function by using the function $L:N\to (A\to \Delta O)\to R$, writing $l=L_t$ instead of $Lt$.

Then implement the uncertain selection $ϵ:J_O^{\left(\Delta \right)}A$. Notice that it's domain is conditional forecasts.

Then an agent is none other than $A:=(ϵ,\psi ):J_O^{\left(\Delta \right)}A\times (N\to J_R(A\to \Delta O\left)\right)$ with sensors and actuators, which interacts with the world via a protocol $\pi$ which runs as follows

1. At time $t+1$, a stakeholder or principal sets $l=L_t$ and hands it to $A$.
2. ${\psi }_{t+1}l$ is a conditional forecast that turns actions into uncertainty in $O$. $(ϵ\circ {\psi }_{t+1})l$ is the action taken by $A$ at time $t+1$.
3. Observe world $\omega :O$ and score the term $\Omega =\left(\right({\psi }_{t+1}l)\circ ϵ\circ {\psi }_{t+1})l:\Delta O$ against it, using the score to power some search process that informs ${\psi }_{t+2}$.
4. Increment $t$ and repeat.

In other words, the agent calculates an action because it can turn loss functions which score conditional forecasts into a handpicked conditional forecast, and it can also turn conditional forecasts into handpicked actions. $\psi$ hardcodes the procedure for doing bayesian updates, i.e. it has opinions about some beliefs being better than others. $ϵ$ hardcodes (and hides) a utility function, i.e. it has opinions about some outcomes being better than others. Echoing a complex number, which is a real part and an imaginary part, we can view an agent $A$ as an instrumental part and an epistemic part. While the complex numbers are equipped with some notion of "$+$" such that $z:={\pi }_{1}z+{\pi }_{2}z$ (real part plus imaginary part), I can make up a notion of "$\oplus$" such that

$$\forall L:N\to (A\to \Delta O)\to R,\forall t:N,A_t:=\left(\right({\pi }_{2}A\left)\right(t+1\left)\right(Lt\left)\right)\circ \left({\pi }_{1}A\right)\circ \left({\pi }_{2}A\right)(t+1)\left(Lt\right)\oplus \left({\pi }_{1}A\right)\circ \left({\pi }_{2}A\right(t+1\left)\right(Lt\left)\right)$$

(epistemic part "plus" instrumental part).

I played fast and loose with the mutability and implicit differentiability of $\psi$. I think this is appropriate: any intuition about agents is that their beliefs change over time, even if corrigibility remains an open problem (in other words, epistemic part ought to be mutable even if the instrumental part (where the utility function is) is not).

The abstract type signature of $\pi$ is $N\to O$, where here when we say a codomain is outcomes we mean that it's the literal world, not an implementational model of it, hence the signature being "abstract".

Selection product?

In the literature there's a function $J_S^{\times }:=J_1\mapsto J_2\mapsto J_S^{\times }{J}_1{J}_2:J_{S}A\to J_{S}B\to J_S(A\times B)$. It's only defined between selections that share an inner target $S$, though, so it doesn't apply to $A$. Still, there might be some cleverness I haven't considered.

Conclusion

We need more candidates for the type signature of agency. An obvious way to explore is to take the first candidate someone wrote down, make an incision, and poke its guts with various functions $F:Type\to Type$.

A more complete story of agency, together with a protocol describing interactions with the world, is not a single selection but a pair of selections. The pair can be understood as an epistemic part and an instrumental part.

I'm aware that I at least partially took some steps toward reinventing the reinforcement learning theory wheel when I gave the protocol $\pi$, an alternative approach to this post would be to start with RL theory and see what notions of selection function are hanging around.

If we hammer out the dents in ${\Delta }_{\ast }$ we get a really pretty notion of "turning agency into probability" (in the form of the function $\overline{.}$ on a restricted domain), and plausibly also a characterization of the unreliability or impossibility of turning probability into agency (via the insurjectivity of $\overline{.}$).

What about interp? I think something like the searching for search [LW · GW] could, if we're not totally and completely wrong about the pillars of the agency type signature direction, show us a ton about how ML naturally implements terms/proofs of things like $J_{S}X$. A dope UX would be something like tactical programming not for creating terms/proofs, but for parsing out / identifying them in a big pile/soup of linear algebra. A fantasy world I'd like to live in is one where a prosaic/interp shop ships neural-hoogle or transformer-hoogle, a search engine that accepts type signatures and finds configurations of matrices and weights which, if you squint, count as proofs/terms of those types. To be abundantly clear, you can think of the following proof of $J_{\text{float}}A$ as the dumbest possible search

def argmax(f: Callable[[A], float]) -> A:
  ret = None
  curr_y = - 2 ** 100
  for x in A:
    y = f(x)
    if y > curr_y:
      curr_y = y
      ret = x
  return ret

Insofar as the type A is enumerable. The hypothesis advanced by this post is that arbitrarily not-dumb search is constrained by the same type information as dumb search. Search is literally a significant class of proofs of selection.

The objective interpretation of the project of giving a type signature for agents seems a little borked right now, but that could change with increased understanding of factored sets or maybe infrabayesian physicalism.

Selections and continuations play a huge role in compositional game theory, which I'm starting to think provides a mean embedded agency story, though I haven't grokked it quite at the level of writing a post about it just yet.

References

• selection on nlab (recursing into citations)
• several Jules Hedges publications
• selections in software engineering

---

1. not super useful without an interactive session, nevertheless. ↩︎

2. $\forall X:Type,\arg max:J_{R}X$ ↩︎

3. $\forall X:Type,max:K_{R}X$ ↩︎

4. $\arg max$ attains $max$, or $\overline{\arg max}=max$. ↩︎

5. Presume some $XS:Type$ and some $K:K_S^{P}X$. $K$ is attainable if and only if $\exists \epsilon :J_S^{P}X,\forall \alpha :S^X,\forall x:X,\left(\epsilon \alpha \right)x=true\to \left(K\alpha \right)x=true$. For multi-valued variants of $max$ and $\arg max$, we can check that the solution to exercise 2 transfers over to this setting. ↩︎

3 comments

Comments sorted by top scores.