Posts

What is calibration? 2023-03-13T06:30:53.431Z
Searching for a model's concepts by their shape – a theoretical framework 2023-02-23T20:14:46.341Z
Event [Berkeley]: Alignment Collaborator Speed-Meeting 2022-12-19T02:24:20.666Z
Why bet Kelly? 2022-11-15T18:12:05.981Z
Average probabilities, not log odds 2021-11-12T21:39:19.835Z
Mapping Out Alignment 2020-08-15T01:02:31.489Z
AlexMennen's Shortform 2019-12-08T04:51:09.960Z
When wishful thinking works 2018-09-01T23:43:00.858Z
Safely and usefully spectating on AIs optimizing over toy worlds 2018-07-31T18:30:38.392Z
Computational efficiency reasons not to model VNM-rational preference relations with utility functions 2018-07-25T02:11:35.429Z
A comment on the IDA-AlphaGoZero metaphor; capabilities versus alignment 2018-07-11T01:03:03.305Z
Logical uncertainty and mathematical uncertainty 2018-07-01T00:33:53.000Z
Logical uncertainty and Mathematical uncertainty 2018-06-26T01:08:20.559Z
More on the Linear Utility Hypothesis and the Leverage Prior 2018-02-26T23:53:35.605Z
Value learning subproblem: learning goals of simple agents 2017-12-18T02:05:15.000Z
Against the Linear Utility Hypothesis and the Leverage Penalty 2017-12-14T18:38:53.103Z
Being legible to other agents by committing to using weaker reasoning systems 2017-12-03T07:49:13.000Z
Metamathematics and probability 2017-09-22T04:04:55.000Z
Metamathematics and Probability 2017-09-22T03:07:57.156Z
Density Zero Exploration 2017-08-17T00:43:51.000Z
Logical Induction with incomputable sequences 2017-08-17T00:39:01.000Z
Existential risk from AI without an intelligence explosion 2017-05-25T16:44:04.809Z
Modal Combat for games other than the prisoner's dilemma 2017-02-26T08:10:05.000Z
Superintelligence via whole brain emulation 2016-08-17T04:11:28.914Z
An approach to the Agent Simulates Predictor problem 2016-04-09T06:28:04.000Z
Two kinds of population ethics, and Current-Population Utilitarianism 2014-06-17T22:26:31.924Z
Selfish reasons to reject the repugnant conclusion in practice 2014-06-05T06:41:41.682Z
Prisoner's dilemma tournament results 2013-07-09T20:50:43.753Z
Prisoner's Dilemma (with visible source code) Tournament 2013-06-07T08:30:20.271Z
VNM agents and lotteries involving an infinite number of possible outcomes 2013-02-21T21:58:30.280Z
Harsanyi's Social Aggregation Theorem and what it means for CEV 2013-01-05T21:38:42.693Z
Interpersonal and intrapersonal utility comparisons 2013-01-04T03:05:36.635Z
A utility-maximizing varient of AIXI 2012-12-17T03:48:05.173Z
Meetup : Berkeley meetup: choice blindness 2012-09-25T05:31:44.630Z
Meetup : Berkeley meetup 2012-09-19T03:21:52.634Z
Meetup : Berkeley meetup: board game night 2012-09-03T17:27:47.017Z
Friendship and happiness generation 2011-11-25T20:52:18.009Z
Silicon Valley AI/Machine Learning study group 2011-09-23T17:18:02.239Z
Fiction: Letter from the End 2011-08-09T23:04:59.534Z
Assumption of positive rationality 2011-04-28T16:41:50.161Z
Creationism's effect on the progress of our understanding of evolution 2011-03-28T20:36:26.967Z
Social ethics vs decision theory 2011-02-20T04:14:54.389Z
Resolving the unexpected hanging paradox 2011-01-25T19:32:50.111Z
Could/would an FAI recreate people who are information-theoretically dead by modern standards? 2011-01-22T21:11:19.278Z
Does Evidential Decision Theory really fail Solomon's Problem? 2011-01-11T04:53:36.930Z
A Paradox in Timeless Decision Theory 2010-10-25T03:09:06.462Z
Human inability to assign numerical probabilities 2010-09-30T04:42:26.633Z

Comments

Comment by AlexMennen on Constructive Cauchy sequences vs. Dedekind cuts · 2024-03-17T23:00:24.978Z · LW · GW

Something that I think it unsatisfying about this is that the rationals aren't previleged as a countable dense subset of the reals; it just happens to be a convenient one. The completions of the diadic rationals, the rationals, and the algebraic real numbers are all the same. But if you require that an element of the completion, if equal to an element of the countable set being completed, must eventually certify this equality, then the completions of the diadic rationals, rationals, and algebraic reals are all constructively inequivalent.

Comment by AlexMennen on Constructive Cauchy sequences vs. Dedekind cuts · 2024-03-17T16:18:50.961Z · LW · GW

This means that, in particular, if your real happens to be rational, you can produce the fact that it is equal to some particular rational number. Neither Cauchy reals nor Dedekind reals have this property.

Comment by AlexMennen on Constructive Cauchy sequences vs. Dedekind cuts · 2024-03-17T16:15:39.698Z · LW · GW

perhaps these are equivalent.

They are. To get enumerations of rationals above and below out of an effective Cauchy sequence, once the Cauchy sequence outputs a rational  such that everything afterwards can only differ by at most , you start enumerating rationals below  as below the real and rationals above  as above the real. If the Cauchy sequence converges to , and you have a rational , then once the Cauchy sequence gets to the point where everything after is gauranteed to differ by at most , you can enumerate  as less than .

Comment by AlexMennen on Constructive Cauchy sequences vs. Dedekind cuts · 2024-03-16T08:10:59.568Z · LW · GW

My take-away from this:

An effective Cauchy sequence converging to a real  induces recursive enumerators for  and , because if , then  for some , so you eventually learn this.

The constructive meaning of a set is that that membership should be decidable, not just semi-decidable.

If  is irrational, then  and  are complements, and each semi-decidable, so they are decidable. If  is rational, then the complement of  is , which is semi-decidable, so again these sets are decidable. So, from the point of view of classical logic, it's not only true that Cauchy sequences and Dedekind cuts are equivalent, but also effective Cauchy sequences and effective Dedekind cuts are equivalent.

However, it is not decidable whether a given Cauchy-sequence real is rational or not, and if so, which rational it is. So this doesn't give a way to construct decision algorithms for the sets   and   from recursive enumerators of them.

Comment by AlexMennen on OpenAI: The Battle of the Board · 2023-11-23T23:19:04.849Z · LW · GW

If board members have an obligation not to criticize their organization in an academic paper, then they should also have an obligation not to discuss anything related to their organization in an academic paper. The ability to be honest is important, and if a researcher can't say anything critical about an organization, then non-critical things they say about it lose credibility.

Comment by AlexMennen on When and why should you use the Kelly criterion? · 2023-11-11T21:51:28.804Z · LW · GW

Yeah, I wasn't trying to claim that the Kelly bet size optimizes a nonlogarithmic utility function exactly, just that, when the number of rounds of betting left is very large, the Kelly bet size sacrifices a very small amount of utility relative to optimal betting under some reasonable assumptions about the utility function. I don't know of any precise mathematical statement that we seem to disagree on.

Well, we've established the utility-maximizing bet gives different expected utility from the Kelly bet, right? So it must give higher expected utility or it wouldn't be utility-maximizing.

Right, sorry. I can't read, apparently, because I thought you had said the utility-maximizing bet size would be higher than the Kelly bet size, even though you did not.

Comment by AlexMennen on When and why should you use the Kelly criterion? · 2023-11-08T18:00:49.290Z · LW · GW

Yeah, I was still being sloppy about what I meant by near-optimal, sorry. I mean the optimal bet size will converge to the Kelly bet size, not that the expected utility from Kelly betting and the expected utility from optimal betting converge to each other. You could argue that the latter is more important, since getting high expected utility in the end is the whole point. But on the other hand, when trying to decide on a bet size in practice, there's a limit to the precision with which it is possible to measure your edge, so the difference between optimal bet and Kelly bet could be small compared to errors in your ability to determine the Kelly bet size, in which case thinking about how optimal betting differs from Kelly betting might not be useful compared to trying to better estimate the Kelly bet.

Even in the limit as the number of rounds goes to infinity, by the time you get to the last round of betting (or last few rounds), you've left the  limit, since you have some amount of wealth and some small number of rounds of betting ahead of you, and it doesn't matter how you got there, so the arguments for Kelly betting don't apply. So I suspect that Kelly betting until near the end, when you start slightly adjusting away from Kelly betting based on some crude heuristics, and then doing an explicit expected value calculation for the last couple rounds, might be a good strategy to get close to optimal expected utility.

Incidentally, I think it's also possible to take a limit where Kelly betting gets you optimal utility in the end by making the favorability of the bets go to zero simultaneously with the number of rounds going to infinity, so that improving your strategy on a single bet no longer makes a difference.

I think that for all finite , the expected utility at timestep  from utility-maximizing bets is higher than that from Kelly bets. I think this is the case even if the difference converges to 0, which I'm not sure it does.

Why specifically higher? You must be making some assumptions on the utility function that you haven't mentioned.

Comment by AlexMennen on When and why should you use the Kelly criterion? · 2023-11-07T15:59:23.833Z · LW · GW

I do want to note though that this is different from "actually optimal"

By "near-optimal", I meant converges to optimal as the number of rounds of betting approaches infinity, provided initial conditions are adjusted in the limit such that whatever conditions I mentioned remain true in the limit. (e.g. if you want Kelly betting to get you a typical outcome of  in the end, then when taking the limit as the number  of bets goes to infinity, you better have starting money , where  is the geometric growth rate you get from bets, rather than having a fixed starting money while taking the limit ). This is different from actually optimal because in practice, you get some finite amount of betting opportunities, but I do mean something more precise than just that Kelly betting tends to get decent outcomes.

Comment by AlexMennen on When and why should you use the Kelly criterion? · 2023-11-07T04:38:07.427Z · LW · GW

The reason I brought this up, which may have seemed nitpicky, is that I think this undercuts your argument for sub-Kelly betting. When people say that variance is bad, they mean that because of diminishing marginal returns, lower variance is better when the mean stays the same. Geometric mean is already the expectation of a function that gets diminishing marginal returns, and when it's geometric mean that stays fixed, lower variance is better if your marginal returns diminish even more than that. Do they? Perhaps, but it's not obvious. And if your marginal returns diminish but less than for log, then higher variance is better. I don't think any of median, mode, or looking at which thing more often gets a higher value are the sorts of things that it makes sense to talk about trading off against lowering variance either. You really want mean for that.

Comment by AlexMennen on When and why should you use the Kelly criterion? · 2023-11-07T03:08:48.102Z · LW · GW

Correct. This utility function grows fast enough that it is possible for the expected utility after many bets to be dominated by negligible-probability favorable tail events, so you'd want to bet super-Kelly.

If you expect to end up with lots of money at the end, then you're right; marginal utility of money becomes negigible, so expected utility is greatly effected by neglible-probability unfavorable tail events, and you'd want to bet sub-Kelly. But if you start out with very little money, so that at the end of whatever large number of rounds of betting, you only expect to end up with  money in most cases if you bet Kelly, then I think the Kelly criterion should be close to optimal.

(The thing you actually wrote is the same as log utility, so I substituted what you may have meant). The Kelly criterion should optimize this, and more generally  for any , if the number of bets is large. At least if  is an integer, then, if  is normally distributed with mean  and standard deviation , then  is some polynomial in  and  that's homogeneous of degree . After a large number  of bets,  scales proportionally to  and  scales proportionally to , so the value of this polynomial approaches its  term, and maximizing it becomes equivalent to maximizing , which the Kelly criterion does. I'm pretty sure you get something similar when  is noninteger.

It depends how much money you could end up with compared to . If Kelly betting usually gets you more than  at the end, then you'll bet sub-Kelly to reduce tail risk. If it's literally impossible to exceed  even if you go all-in every time and always win, then this is linear, and you'll bet super-Kelly. But if Kelly betting will usually get you less than  but not by too many orders of magnitude at the end after a large number of rounds of betting, then I think it should be near-optimal.

If there's many rounds of betting, and Kelly betting will get you  as a typical outcome, then I think Kelly betting is near-optimal. But you might be right if .

Comment by AlexMennen on When and why should you use the Kelly criterion? · 2023-11-06T05:38:02.779Z · LW · GW

If you bet more than Kelly, you'll experience lower average returns and higher variance.

No. As they discovered in the dialog, average returns is maximized by going all-in on every bet with positive EV. It is typical returns that will be lower if you don't bet Kelly.

Comment by AlexMennen on When and why should you use the Kelly criterion? · 2023-11-06T01:42:27.706Z · LW · GW

The Kelly criterion can be thought of in terms of maximizing a utility function that depends on your wealth after many rounds of betting (under some mild assumptions about that utility function that rule out linear utility). See https://www.lesswrong.com/posts/NPzGfDi3zMJfM2SYe/why-bet-kelly

Comment by AlexMennen on A new intro to Quantum Physics, with the math fixed · 2023-10-31T18:11:51.341Z · LW · GW

For two, your specific claims about the likely confusion that Eliezer's presentation could induce in "laymen" is empirically falsified to some degree by the comments on the original post: in at least one case, a reader noticed the issue and managed to correct for it when they made up their own toy example, and the first comment to explicitly mention the missing unitarity constraint was left over 10 years ago.

Some readers figuring out what's going on is consistent with many of them being unnecessarily confused.

Comment by AlexMennen on How Many Bits Of Optimization Can One Bit Of Observation Unlock? · 2023-04-26T19:33:33.392Z · LW · GW

I don't think this one works. In order for the channel capacity to be finite, there must be some maximum number of bits N you can send. Even if you don't observe the type of the channel, you can communicate a number n from 0 to N by sending n 1s and N-n 0s. But then even if you do observe the type of the channel (say, it strips the 0s), the receiver will still just see some number of 1s that is from 0 to N, so you have actually gained zero channel capacity. There's no bonus for not making full use of the channel; in johnswentworth's formulation of the problem, there's no such thing as some messages being cheaper to transmit through the channel than others.

Comment by AlexMennen on Average probabilities, not log odds · 2023-04-17T03:01:10.165Z · LW · GW

We "just" need to update the three geometric averages on this background knowledge. Plausibly how this should be done in this case is to normalize them such that they add to one.

My problem with a forecast aggregation method that relies on renormalizing to meet some coherence constraints is that then the probabilities you get depend on what other questions get asked. It doesn't make sense for a forecast aggregation method to give probability 32.5% to A if the experts are only asked about A, but have that probability predictably increase if the experts are also asked about B and C. (Before you try thinking of a reason that the experts' disagreement about B and C is somehow evidence for A, note that no matter what each of the experts believe, if your forecasting method is mean log odds, but renormalized to make probabilities sum to 1 when you ask about all 3 outcomes, then the aggregated probability assigned to A can only go up when you also ask about B and C, never down. So any such defense would violate conservation of expected evidence.)

(In the case of the arithmetic mean, updating on the background information plausibly wouldn't change anything here, but that's not the case for other possible background information.)

Any linear constraints (which are the things you get from knowing that certain Boolean combinations of questions are contradictions or tautologies) that are satisfied by each predictor will also be satisfied by their arithmetic mean.

But it is anyway a more general question (than the question of whether the geometric mean of the odds is better or the arithmetic mean of the probabilities): how should we "average" two or more probability distributions (rather than just two probabilities), assuming they come from equally reliable sources?

That's part of my point. Arithmetic mean of probabilities gives you a way of averaging probability distributions, as well as individual probabilities. Geometric mean of log odds does not.

If we assume that the prior was indeed important here then this makes sense, but if we assume that the prior was irrelevant (that they would have arrived at 25% even if their prior was e.g. 10% rather than 50%), then this doesn't make sense. (Maybe they first assumed the probability of drawing a black ball from an urn was 50%, then they each independently created a large sample, and ~25% of the balls came out black. In this case the prior was mostly irrelevant.) We would need a more general description under which circumstances the prior is indeed important in your sense and justifies the multiplicative evidence aggregation you proposed.

In this example, the sources of evidence they're using are not independent; they can expect ahead of time that each of them will observe the same relative frequency of black balls from the urn, even while not knowing in advance what that relative frequency will be. The circumstances under which the multiplicative evidence aggregation method is appropriate are exactly the circumstances in which the evidence actually is independent.

But in the second case I don't see how a noisy process for a probability estimate would lead to being "forced to set odds that you'd have to take bets on either side of, even someone who knows nothing about the subject could exploit you on average".

They make their bet direction and size functions of the odds you offer them in such a way that they bet more when you offer better odds. If you give the correct odds, then the bet ends up resolving neutrally on average, but if you give incorrect odds, then which direction you are off in correlates with how big a bet they make in such a way that you lose on average either way.

Comment by AlexMennen on Concave Utility Question · 2023-04-15T05:31:54.497Z · LW · GW

Oh, derp. You're right.

Comment by AlexMennen on Concave Utility Question · 2023-04-15T04:49:49.553Z · LW · GW

I think the way I would rule out my counterexample is by strengthening A3 to if  and  then there is ...

Comment by AlexMennen on Concave Utility Question · 2023-04-15T01:10:10.024Z · LW · GW

Q2: No. Counterexample: Suppose there's one outcome  such that all lotteries are equally good, except for the lottery than puts probability 1 on , which is worse than the others.

Comment by AlexMennen on What Discovering Latent Knowledge Did and Did Not Find · 2023-03-17T19:11:39.217Z · LW · GW

I'm not sure why you don't like calling this "redundancy". A meaning of redundant is "able to be omitted without loss of meaning or function" (Lexico). So ablation redundancy is the normal kind of redundancy, where you can remove sth without losing the meaning. Here it's not redundant, you can remove a single direction and lose all the (linear) "meaning".

Suppose your datapoints are   (where the coordinates  and  are independent from the standard normal distribution), and the feature you're trying to measure is . A rank-1 linear probe will retain some information about the feature. Say your linear probe finds the  coordinate. This gives you information about ; your expected value for this feature is now , an improvement over its a priori expected value of . If you ablate along this direction, all you're left with is the  coordinate, which tells you exactly as much about the feature  as the  coordinate does, so this rank-1 ablation causes no loss in performance. But information is still lost when you lose the  coordinate, namely the contribution of  from the feature. The thing that you can still find after ablating away the  direction is not redundant with the the rank-1 linear probe in the  direction you started with, but just contributes the same amount towards the feature you're measuring.

The point is, the reason why CCS fails to remove linearly available information is not because the data "is too hard". Rather, it's because the feature is non-linear in a regular way, which makes CCS and Logistic Regression suck at finding the direction which contains all linearly available data (which exists in the context of "truth", just as it is in the context of gender and all the datasets on which RLACE has been tried).

Disagree. The reason CCS doesn't remove information is neither of those, but instead just that that's not what it's trained to do. It doesn't fail, but rather never makes any attempt. If you're trying to train a function such that  and , then  will achieve optimal loss just like  will.

Comment by AlexMennen on What Discovering Latent Knowledge Did and Did Not Find · 2023-03-15T20:13:55.208Z · LW · GW

What you're calling ablation redundancy is a measure of nonlinearity of the feature being measured, not any form of redundancy, and the view you quote doesn't make sense as stated, as nonlinearity, rather than redundancy, would be necessary for its conclusion. If you're trying to recover some feature , and there's any vector  and scalar  such that  for all data  (regardless of whether there are multiple such , which would happen if the data is contained in a proper affine subspace), then there is a direction such that projection along it makes it impossible for a linear probe to get any information about the value of . That direction is , where  is the covariance matrix of the data. This works because if , then the random variables  and  are uncorrelated (since ), and thus  is uncorrelated with .

If the data is normally distributed, then we can make this stronger. If there's a vector  and a function  such that  (for example, if you're using a linear probe to get a binary classifier, where it classifies things based on whether the value of a linear function is above some threshhold), then projecting along  removes all information about . This is because uncorrelated linear features of a multivariate normal distribution are independent, so if , then  is independent of , and thus also of . So the reason what you're calling high ablation redundancy is rare is that low ablation redundancy is a consequence of the existence of any linear probe that gets good performance and the data not being too wildly non-Gaussian.

Comment by AlexMennen on What Discovering Latent Knowledge Did and Did Not Find · 2023-03-15T00:47:35.707Z · LW · GW

Ablating along the difference of the means makes both CCS & Supervised learning fail, i.e. reduce their accuracy to random guessing. Therefore:

  • The fact that Recursive CCS finds many good direction is not due to some “intrinsic redundancy” of the data. There exist a single direction which contains all linearly available information.
  • The fact that Recursive CCS finds strictly more than one good direction means that CCS is not efficient at locating all information related to truth: it is not able to find a direction which contains as much information as the direction found by taking the difference of the means. Note: Logistic Regression seems to be about as leaky as CCS. See INLP which is like Recursive CCS, but with Logistic Regression.

I don't think that's a fair characterization of what you found. Suppose, for example, that you're given a vector in  whose th component is , where  is a random variable with high variance, and  are i.i.d. with mean  and tiny variance. There is a direction which contains all the information about  contained in the vector, namely the average of the coordinates. Subtracting out the mean of the coordinates from each coordinate will remove all information about . But the data is plenty redundant; there are  orthogonal directions each of which contain almost all of the available information about , so a probe trained to recover  that learns to just copy one of the coordinates will be pretty efficient at recovering . If the  have variance  (i.e. are just constants always equal to ), then there are  orthogonal directions each of which contain all information about , and a probe that copies one of them is perfectly efficient at extracting all information about .

If you can find multiple orthogonal linear probes that each get good performance at recovering some feature, then something like this must be happening.

Comment by AlexMennen on GPT-4 Predictions · 2023-02-18T16:43:48.260Z · LW · GW

My point wasn't that the equation didnt hold perfectly, but that the discrepancies are very suspicious. Two of the three discrepancies were off by exactly 1 order of magnitude, making me fairly confident that they are the result of a typo. (Not sure what's going on with the other discrepency).

Comment by AlexMennen on GPT-4 Predictions · 2023-02-18T04:21:43.342Z · LW · GW

In the table of parameters, compute, and tokens, compute/(parameters*tokens) is always 6, except in one case where it's 0.6, one case where it's 60, and one case where it's 2.75. Are you sure this is right?

Comment by AlexMennen on Event [Berkeley]: Alignment Collaborator Speed-Meeting · 2022-12-19T02:50:58.030Z · LW · GW

Done, thanks.

Comment by AlexMennen on Open technical problem: A Quinean proof of Löb's theorem, for an easier cartoon guide · 2022-11-24T23:19:34.374Z · LW · GW

It would kind of use assumption 3 inside step 1, but inside the syntax, rather than in the metalanguage. That is, step 1 involves checking that the number encoding "this proof" does in fact encode a proof of C. This can't be done if you never end up proving C.

One thing that might help make clear what's going on is that you can follow the same proof strategy, but replace "this proof" with "the usual proof of Lob's theorem", and get another valid proof of Lob's theorem, that goes like this: Suppose you can prove that []C->C, and let n be the number encoding a proof of C via the usual proof of Lob's theorem. Now we can prove C a different way like so:

  1. n encodes a proof of C.
  2. Therefore []C.
  3. By assumption, []C->C.
  4. Therefore C.

Step 1 can't be correctly made precise if it isn't true that n encodes a proof of C.

Comment by AlexMennen on Kelsey Piper's recent interview of SBF · 2022-11-17T16:50:55.013Z · LW · GW

The revelation that he spent maybe 10x as much on villas for his girlfriends as EA cause areas

Source?

Comment by AlexMennen on Kelsey Piper's recent interview of SBF · 2022-11-17T06:59:54.697Z · LW · GW

The idea that he was trying to distance himself from EA to protect EA doesn't hold together because he didn't actually distance himself from EA at all in that interview. He said ethics is fake, but it was clear from context that he meant ordinary ethics, not utilitarianism.

Comment by AlexMennen on Why bet Kelly? · 2022-11-16T01:36:40.829Z · LW · GW

"Having been handed this enormous prize, how do I maximize the probability that I max out on utility?" Hm, but that actually doesn't give back any specific criterion, since basically any strategy that never bets your whole stack will win.

That's not quite true. If you bet more than double Kelly, your wealth decreases. But yes, Kelly betting isn't unique in growing your wealth to infinity in the limit as number of bets increases.

If the number of bets is very large, but due to some combination of low starting wealth relative to the utility bound and slow growth rate, it is not possible to get close to maximum utility, then Kelly betting should be optimal.

Comment by AlexMennen on Why bet Kelly? · 2022-11-16T00:19:33.691Z · LW · GW

I basically endorse what kh said. I do think it's wrong to think you can fit enormous amounts of expected value or disvalue into arbitrarily tiny probabilities.

Comment by AlexMennen on Why bet Kelly? · 2022-11-15T23:31:02.190Z · LW · GW

It is true that in practice, there's a finite amount of credit you can get, and credit has a cost, limiting the practical applicability of a model with unlimited access to free credit, if the optimal strategy according to the model would end up likely making use of credit which you couldn't realistically get cheaply. None of this seems important to me. The easiest way to understand the optimal strategy when maximum bet sizes are much smaller than your wealth is that it maximizes expected wealth on each step, rather than that it maximizes expected log wealth on each step. This is especially true if you don't already understand why following the Kelly criterion is instrumentally useful, and I hadn't yet gotten to the section where I explained that, and in fact used the linear model in order to show that Kelly betting is optimal by showing that it's just the linear model on a log scale.

One could similarly object that since currency is discrete, you can't go below 1 unit of currency and continue to make bets, so you need to maintain a log-scale bankroll where you prevent your log wealth from going negative, and you should really be maximizing your expected log log wealth, which happens to give you the same results when your wealth is a large enough number of currency units that the discretization doesn't make a difference. Like, sure, I guess, but it's still useful to model currency as continuous, so I see no need to account for its discreteness in a model. Similarly, in situations where the limitations on funds available to place bets with don't end up affecting you, I don't think it needs to be explicitly included in the model.

Comment by AlexMennen on Why bet Kelly? · 2022-11-15T22:36:12.747Z · LW · GW

Access to credit. In the logarithmic model, you never make bets that could make your net worth zero or negative.

Comment by AlexMennen on Why bet Kelly? · 2022-11-15T19:58:18.066Z · LW · GW

Again, the max being a small portion of your net worth isn't the assumption behind the model; the assumption is just that you don't get constrained by lack of funds, so it is a different model. It's true that if the reason you don't get constrained by lack of funds is that the maximum bets are small relative to your net worth, then this is also consistent with maximizing log wealth on each step. But this isn't relevant to what I brought it up for, which was to use it as a step in explaining the reason for the Kelly criterion in the section after it.

Comment by AlexMennen on Why bet Kelly? · 2022-11-15T18:49:05.886Z · LW · GW

No. The point of the model where acting like your utility is linear is optimal wasn't that this is a more realistic model than the assumptions behind the Kelly criterion; it's just another simplified model, which is slightly easier to analyze, so I was using it as a step in showing why you should follow the Kelly criterion when it is your wealth that constrains the bet sizes you can make. It's also not true that the linear-utility model I described is still just maximizing log wealth; for instance, if the reason that you're never constrained by available funds is that you have access to credit, then your wealth could go negative, and then its log wouldn't even be defined.

Comment by AlexMennen on Kelly *is* (just) about logarithmic utility · 2022-11-12T19:36:32.334Z · LW · GW

Most of the arguments for Kelly betting that you address here seem like strawmen, except for (4), which can be rescued from your objection, and an interpretation of johnswentworth's version of (2), which you actually mention in footnote 3, but seem unfairly dismissive of.

The assumptions according to which your derived utility function is logarithmic is that expected utility doesn't get dominated by negligible-probability tail events. For instance, if you have a linear utility function and you act like it, you almost surely get 0 payout, but your expected payout is enormous because of the negligible-probability tail event in which you win every bet. Even if you do Kelly betting instead, the expected payout is going to be well outside the range of typical payouts, because of the negligible-probability tail event in which you win a statistically improbably number of bets. This won't happen if, for instance, you have a bounded utility function, for which typical payouts from Kelly betting will not get you infinitesimally close to the bounds. The class of myopic utility functions is infinite, yes, but in the grand scheme of things, compared to the space of possible utility functions, is very tiny, and I don't think it should be surprising that there are relatively mild assumptions that imply results that aren't true of most of the myopic utility functions.

In footnote 3, you note that optimizing for all quantiles simultaneously is not possible. Kelly betting comes extremely close to doing this. Your implied objection is, I assume, that the quantifier order is backwards from what would make this really airtight: When comparing Kelly betting to a different strategy, for every quantile, Kelly betting is superior after sufficiently many iterations, but there is no single sufficient number of iterations after which Kelly betting is superior for every quantile; if you have enough iterations such that Kelly betting is better for quantiles 1% through 99%, the alternative strategy could still be so much better at the 99.9% quantile that it outweighs all this. This is where the assumption that negligible-probability tail events don't dominate expected value calculations makes this difference not matter so much. I think that this is a pretty natural assumption, and thus that this really is almost as good.

Comment by AlexMennen on Without specific countermeasures, the easiest path to transformative AI likely leads to AI takeover · 2022-07-19T07:51:56.511Z · LW · GW

But in fact, I expect the honest policy to get significantly less reward than the training-game-playing policy, because humans have large blind spots and biases affecting how they deliver rewards.

The difference in reward between truthfulness and the optimal policy depends on how humans allocate rewards, and perhaps it could be possible to find a clever strategy for allocating rewards such that truthfulness gets close to optimal reward.

For instance, in the (unrealistic) scenario in which a human has a well-specified and well-calibrated probability distribution over the state of the world, so that the actual state of the world (known to the AI) is randomly selected from this distribution, the most naive way to allocate rewards would be to make the loss be the log of the probability the human assigns to the answers given by the AI (so it gets better performance by giving higher-probability answers). This would disincentivize answering questions honestly if the human is often wrong. A better way to allocate rewards would be to ask a large number of questions about the state of the world, and, for each simple-to-describe property that an assignment of answers to each of these questions could have, which is extremely unlikely according to the human's probability distribution (e.g. failing calibration tests), penalize assignments of answers to questions that satisfy this property. That way answering according to a random selection from the human's probability distribution (which we're modeling the actual state of the world as) will get high reward with high probability, while other simple-to-describe strategies for answering questions will likely have one of the penalized properties and get low reward.

Of course, this doesn't work in real life because the state of the world isn't randomly selected from human's beliefs. Human biases make it more difficult to make truthfulness get close to optimal reward, but not necessarily impossible. One possibility would be to only train on questions that the human evaluators are extremely confident of the correct answers to, in hopes that they can reliably reward the AI more for truthful answers than for untruthful ones. This has the drawback that there would be no training data for topics that humans are uncertain about, which might make it infeasible for the AI to learn about these topics. It sure seems hard to come up with a reward allocation strategy that allows questions on which the humans are uncertain in training but still makes truth-telling a not-extremely-far-from-optimal strategy, under realistic assumptions about how human beliefs relate to reality, but it doesn't seem obviously impossible.

That said, I'm still skeptical that AIs can be trained to tell to truth (as opposed to say things that are believed by humans) by rewarding what seems like truth-telling, because I don't share the intuition that truthfulness is a particularly natural strategy that will be easy for gradient descent to find. If it's trained on questions in natural language that weren't selected for being very precisely stated, then these questions will often involve fuzzy, complicated concepts that humans use because we find useful, even though they aren't especially natural. Figuring out how to correctly answer these questions would require learning things about how humans understand the world, which is also what you need in order to exploit human error to get higher reward than truthfulness would get.

Comment by AlexMennen on Deep Learning Systems Are Not Less Interpretable Than Logic/Probability/Etc · 2022-06-06T18:52:00.652Z · LW · GW

It sounds to me like, in the claim "deep learning is uninterpretable", the key word in "deep learning" that makes this claim true is "learning", and you're substituting the similar-sounding but less true claim "deep neural networks are uninterpretable" as something to argue against. You're right that deep neural networks can be interpretable if you hand-pick the semantic meanings of each neuron in advance and carefully design the weights of the network such that these intended semantic meanings are correct, but that's not what deep learning is. The other things you're comparing it to that are often called more interpretable than deep learning are in fact more interpretable than deep learning, not (as you rightly point out) because the underlying structures they work with is inherently more interpretable, but because they aren't machine learning of any kind.

Comment by AlexMennen on Against Time in Agent Models · 2022-05-15T02:48:15.632Z · LW · GW

This seems related in spirit to the fact that time is only partially ordered in physics as well. You could even use special relativity to make a model for concurrency ambiguity in parallel computing: each processor is a parallel worldline, detecting and sending signals at points in spacetime that are spacelike-separated from when the other processors are doing these things. The database follows some unknown worldline, continuously broadcasts its contents, and updates its contents when it receives instructions to do so. The set of possible ways that the processors and database end up interacting should match the parallel computation model. This makes me think that intuitions about time that were developed to be consistent with special relativity should be fine to also use for computation.

Comment by AlexMennen on Vanessa Kosoy's Shortform · 2022-05-06T17:44:45.564Z · LW · GW

Wikipedia claims that every sequence is Turing reducible to a random one, giving a positive answer to the non-resource-bounded version of any question of this form. There might be a resource-bounded version of this result as well, but I'm not sure.

Comment by AlexMennen on Is Fisherian Runaway Gradient Hacking? · 2022-04-10T18:26:11.085Z · LW · GW
  1. By "optimal", I mean in an evidential, rather than causal, sense. That is, the optimal value is that which signals greatest fitness to a mate, rather than the value that is most practically useful otherwise. I took Fisherian runaway to mean that there would be overcorrection, with selection for even more extreme traits than what signals greatest fitness, because of sexual selection by the next generation. So, in my model, the value of  that causally leads to greatest chance of survival could be , but high values for  are evidence for other traits that are causally associated with survivability, so  offers best evidence of survivability to potential mates, and Fisherian runaway leads to selection for . Perhaps I'm misinterpreting Fisherian runaway, and it's just saying that there will be selection for  in this case, instead of over-correcting and selecting for ? But then what's all this talk about later-generation sexual selection, if this doesn't change the equilibrium?
  2. Ah, so if we start out with an average , standard deviation , and optimal , then selecting for larger  has the same effect as selecting for  closer to , and that could end up being what potential mates do, driving  up over the generations, until it is common for individuals to have positive , but potential mates have learned to select for higher ? Sure, I guess that could happen, but there would then be selection pressure on potential mates to stop selecting for higher  at this point. This would also require a rapid environmental change that shifts the optimal value of ; if environmental changes affecting optimal phenotype aren't much faster than evolution, then optimal phenotypes shouldn't be so wildly off the distribution of actual phenotypes.
Comment by AlexMennen on Is Fisherian Runaway Gradient Hacking? · 2022-04-10T14:47:49.088Z · LW · GW

Fisherian runaway doesn't make any sense to me.

Suppose that each individual in a species of a given sex has some real-valued variable , which is observable by the other sex. Suppose that, absent considerations about sexual selection by potential mates for the next generation, the evolutionarily optimal value for  is 0. How could we end up with a positive feedback loop involving sexual selection for positive values of , creating a new evolutionary equilibrium with an optimal value  when taking into account sexual selection? First the other sex ends up with some smaller degree of selection for positive values of  (say selecting most strongly for ). If sexual selection by the next generation of potential mates were the only thing that mattered, then the optimal value of  to select for is , since that's what everyone else is selecting for. That's stability, not positive feedback. But sexual selection by the next generation of potential mates isn't the only thing that matters; by stipulation, different values of  have effects on evolutionary fitness other than through sexual selection, with values closer to  being better. So, when choosing a mate, one must balance the considerations of sexual selection by the next generation (for which  is optimal) and other considerations (for which  is optimal), leading to selection for mates with  being evolutionarily optimal. That's negative feedback. How do you get positive feedback?

Comment by AlexMennen on Save Humanity! Breed Sapient Octopuses! · 2022-04-06T00:36:26.610Z · LW · GW

I know this was tagged as humor, but taking it seriously anyway,

I'm skeptical that breeding octopuses for intelligence would yield much in the way of valuable insights for AI safety, since octopuses and humans have so much in common that AGI wouldn't. That said, it's hard to rule out that uplifting another species could reveal some valuable unknown unknowns about general intelligence, so I unironically think this is a good reason to try it.

Another, more likely to pay off, benefit to doing this would be as a testbed for genetically engineering humans for higher intelligence (which also might have benefits for AI safety under long-timelines assumptions). I also think it would just be really cool from a scientific perspective.

Comment by AlexMennen on The Halting Problem and the Impossible Photocopier · 2022-04-01T03:17:28.254Z · LW · GW

One example of a class of algorithms that can solve its own halting problem is the class of primitive recursive functions. There's a primitive recursive function  that takes as input a description of a primitive recursive function  and input  and outputs  if  halts, and  otherwise: this program is given by , because all primitive recursive functions halt on all inputs. In this case, it is  that does not exist.

I think  should exist, at least for classical bits (which as others have pointed out, is all that is needed), for any reasonably versatile model of computation. This is not so for , since primitive recursion is actually an incredibly powerful model of computation; any program that you should be able to get an output from before the heat death of the universe can be written with primitive recursion, and in some sense, primitive recursion is ridiculous overkill for that purpose.

Comment by AlexMennen on Ukraine Post #6: More Data and Peace Terms · 2022-03-26T20:05:17.707Z · LW · GW

If a group decides something unanimously, and has the power to do it, they can do it. That would take them outside the formal channels of the EU (or in another context of NATO) but I do not see any barrier to an agreement to stop importing Russian gas followed by everyone who agreed to it no longer importing Russian gas. Hungary would keep importing, but that does not seem like that big a problem.

If politicians can blame Hungary for their inaction, then this partially protects them from being blamed by voters for not doing anything. But it doesn't protect them at all from being blamed for high fuel prices if they stop importing it from Russia. So they have incentives not to find a solution to this problem.

Comment by AlexMennen on Rational and irrational infinite integers · 2022-03-26T02:56:56.194Z · LW · GW

If you have a 10-adic integer, and you want to reduce it to a 5-adic integer, then to know its last n digits in base 5, you just need to know what it is modulo . If you know what it is modulo , then you can reduce it module , so you only need to look at the last n digits in base 10 to find its last n digits in base 5. So a base-10 integer ending in ...93 becomes a base-5 integer ending in ...33, because 93 mod 25 is 18, which, expressed in base 5, is 33.

The Chinese remainder theorem tells us that we can go backwards: given a 5-adic integer and a 2-adic integer, there's exactly one 10-adic integer that reduces to each of them. Let's say we want the 10-adic integer that's 1 in base 5 and -1 in base 2. The last digit is the digit that's 1 mod 5 and 1 mod 2 (i.e. 1). The last 2 digits are the number from 0 to 99 that's 1 mod 25 and 3 mod 4 (i.e. 51). The last 3 digits are the number from 0 to 999 that's 1 mod 125 and 7 mod 8 (i.e. 751). And so on.

Comment by AlexMennen on Rational and irrational infinite integers · 2022-03-25T15:27:35.649Z · LW · GW

(My intuition tells me that if you choose a subset of the primes dividing the base, you can somehow obtain from that a value of  in a way that maps "none" to 1, and "all" to -1, and the remaining combinations to the irrational integers. No more specific ideas.)

That is correct! In base  (for distinct primes ), an integer is determined by an integer in each of the bases , essentially by the Chinese remainder theorem. In other words, . In prime base, 1 and -1 are the only two square roots of 1. In arbitrary base, a number squares to 1 iff the number it reduces to in each prime factor of the base also squares to 1, and there's 2 options for each of these factors.

Comment by AlexMennen on On expected utility, part 1: Skyscrapers and madmen · 2022-03-22T02:30:24.158Z · LW · GW

Yes, I think we're on the same page now.

Comment by AlexMennen on Ukraine Post #5: Bits of Information · 2022-03-22T02:17:53.257Z · LW · GW

the question is whether Ukraine will get defensive agreements in case of another Russian invasion. Since I do not believe such promises are worth much I expect this not to end up being a dealbreaker. ... Ukraine is demanding ‘security guarantees’ as part of the deal that I am thinking they should actively not want, because they are not worth the paper they are printed on and that failure then weakens credibility. As it is now, either invading Ukraine again would create a broader war or it wouldn’t.

I'm curious why you see so little value in promises of protection. Is it because you believed the common misconception that the US and UK already made and have now broken such promises in the Budapest memorandum?

Comment by AlexMennen on On expected utility, part 1: Skyscrapers and madmen · 2022-03-19T20:32:44.086Z · LW · GW

Ah, I think that is what I was talking about. By "actual utility", you mean the sum over the utility of the outcome of each decision problem you face, right? What I was getting at is that your utility function splitting as a sum like this is an assumption about your preferences, not just about the relationship between the various decision problems you face.

Comment by AlexMennen on On expected utility, part 1: Skyscrapers and madmen · 2022-03-19T03:56:41.335Z · LW · GW

Oh, are you talking about the kind of argument that starts from the assumption that your goal is to maximize a sum over time-steps of some function of what you get at that time-step? (This is, in fact, a strong assumption about the nature of the preferences involved, which representation theorems like VNM don't make.)

Comment by AlexMennen on Whence the determinant? · 2022-03-18T21:59:35.222Z · LW · GW

If we call this construction  then the construction I'm thinking of is . Note that  is locally -dimensional, so my construction is locally isomorphic to yours but globally twisted. It depends on  via more than just its local dimension. Also note that with this definition we will get that  is always isomorphic to 

Oh right, I was picturing  being free on connected components when I suggested that. Silly me.

If we pick a basis of  then it induces a bijection between  and . So we could define a map  to be 'alternating' if and only if the corresponding map  is alternating. The interesting thing I noticed about this definition is that it doesn't depend on which basis you pick for . So I have some hope that since this construction isn't basis dependent, I might be able to write down a basis-independent definition of it.

 is alternating if , right? So if we're willing to accept kludgy definitions of determinant in the process of defining , then we're all set, and if not, then we'll essentially need another way to define determinant for projective modules because that's equivalent to defining an alternating map?