Can we learn much by studying the behaviour of RL policies? 2023-05-15T12:56:25.769Z

Forecasting extreme outcomes 2023-01-09T16:34:40.244Z

Comment by AidanGoth on All AGI Safety questions welcome (especially basic ones) [May 2023] · 2023-05-23T20:41:11.697Z · LW · GW

I thought it dealt with these ok -- could you be more specific?

It's linear because it's an expectation. It is under-specified in that it needs us to assume or prove the marginal distributions for the $X_i$ and I guess that's problematic if an algorithm for doing that is a big part of what the authors are looking for. But if we do have marginal distributions for each $X_i$, then $E\left(X_i^2\right),E\left(X_i^{\prime 2}\right),E\left(X_i^{\prime 2}|{\pi }^{\prime }\right)$ are well-defined and $\tilde{E}\left({\sum }_{i=1}^n{X}_i^2|\pi \right)={\sum }_{i=1}^{n}E\left(X_i^{\prime 2}|{\pi }^{\prime }\right)$.

Comment by AidanGoth on All AGI Safety questions welcome (especially basic ones) [May 2023] · 2023-05-23T16:43:46.671Z · LW · GW

This question is in the spirit of "I think I'm doing something dumb / obviously wrong -- help me see why" but it's maybe too niche for this thread. (Answers that redirect me to a better place to ask are welcome.)

I recently read Paul Christiano, Eric Neyman and Mark Xu's "Formalizing the presumption of independence" (https://arxiv.org/pdf/2211.06738.pdf). My understanding is that they aim to formalise some types of reasonable (but defeasible) “hand-waving” in otherwise formal proofs, in a way that maintains the underlying deductive structure of a formal proof and responds appropriately to new information / arguments. They're particularly interested in heuristic estimators that presume the independence of random variables so long as we have no reason to think the variables aren't independent and so long as we can adjust the estimate appropriately if we learn about their dependencies.

To that end, suppose we want to estimate $f(X_1,\dots ,X_n)$, where $\{X_i{\}}_{1\le n}$ is a set of real-valued random variables, $f:R^n\to R$, and we have a collection of deductively proved (in)equalities $\pi =\pi (X_1,\dots ,X_n)$ about $\{X_i{\}}_{1\le n}$. Then a natural heuristic estimator could be:

$\tilde{E}\left(f\right(X_1,\dots ,X_n\left)|\pi \right)=E\left(f\right(X_1^{\prime },\dots ,X_n^{\prime }\left)|{\pi }^{\prime }\right)$

where each $X_i^{\prime }$ has the same marginal distributions as $X_i$, ${\pi }^{\prime }=\pi (X_1^{\prime },\dots ,X_n^{\prime })$ (i.e. ${\pi }^{\prime }$ is equal to $\pi$ but with each instance of $X_i$ replaced by $X_i^{\prime }$), and where the $X_i^{\prime }$ are conditionally independent given ${\pi }^{\prime }$. This formalises the idea that we assume we've thought of all the dependencies between the variables of interest and that they're independent, conditional on everything we've thought of so far -- but we can revise this estimate by conditioning on new information and dependencies later.

Before considering any information relating the $X_i$ to each other, $\tilde{E}$ assumes that they are unconditionally independent. As we condition on information about them, we update the estimate to account for this and maintain that the variables are conditionally independent, given the information considered so far. E.g. in the twin primes example, we can initially assume that $\left\{x\text{is prime}\right\}$ and $\{x+2\text{is prime}\}$ are independent, and then condition on the fact that if $x$ is prime, then $x+2$ is odd (this can be operationalised by considering the appropriate indicator function and conditioning on it taking value $1$) to adjust the estimate and assume (for now) that there are no further dependencies.

We always have $\tilde{E}\left(XY\right)=E\left(X^{\prime }{Y}^{\prime }\right)=E\left(X^{\prime }\right)E\left(Y^{\prime }\right)$. In fact, we always have $\tilde{E}\left(XY|\pi \right)=E\left(X^{\prime }|{\pi }^{\prime }\right)E\left(Y^{\prime }|{\pi }^{\prime }\right)$. If we further have that $\pi$ doesn't relate $X$ and $Y$ (i.e. doesn't include a formula containing both $X$ and $Y$), then I think we have $E\left(X^{\prime }|{\pi }^{\prime }\right)=E\left(X|\pi \right)$ and $E\left(Y^{\prime }|{\pi }^{\prime }\right)=E\left(Y|\pi \right)$, giving $\tilde{E}\left(XY|\pi \right)=E\left(X|\pi \right)E\left(Y|\pi \right)$ (i.e. without the primes).

My suggested heuristic estimator apparently has lots of nice properties thanks to being an expectation, including some of the informal properties listed in the paper, which can be stated formally (e.g. if ${\pi }_j$ doesn't have an instance of any of the $X_i$, then conditioning on it won't change the heuristic estimate).

My suggested estimator jumped out to me pretty quickly as capturing (to my understanding) what the authors want, but I'd expect myself to be much worse at this than the authors, who will have spent a while longer thinking about it. So my estimator seems "too good to be true" and I think it's likely I'm pretty confused or missing something obvious and/or important. Please help me see what I'm missing! A couple of hypotheses:

• There's something wrong / incoherent about my suggested heuristic estimator
• My suggested heuristic estimator is too general to be useful
  • The paper mainly considers very specific special cases with specific algorithms for heuristic estimators rather than something as general as this, which might be difficult to implement in practice