Fractional progress estimates for AI timelines and implied resource requirements 2021-07-15T18:43:10.163Z
Intermittent Distillations #4: Semiconductors, Economics, Intelligence, and Technological Progress. 2021-07-08T22:14:23.374Z
Anthropic Effects in Estimating Evolution Difficulty 2021-07-05T04:02:18.242Z
An Intuitive Guide to Garrabrant Induction 2021-06-03T22:21:41.877Z
Rogue AGI Embodies Valuable Intellectual Property 2021-06-03T20:37:30.805Z
Intermittent Distillations #3 2021-05-15T07:13:24.438Z
Pre-Training + Fine-Tuning Favors Deception 2021-05-08T18:36:06.236Z
Less Realistic Tales of Doom 2021-05-06T23:01:59.910Z
Agents Over Cartesian World Models 2021-04-27T02:06:57.386Z
[Linkpost] Treacherous turns in the wild 2021-04-26T22:51:44.362Z
Intermittent Distillations #2 2021-04-14T06:47:16.356Z
Transparency Trichotomy 2021-03-28T20:26:34.817Z
Intermittent Distillations #1 2021-03-17T05:15:27.117Z
Strong Evidence is Common 2021-03-13T22:04:40.538Z
Open Problems with Myopia 2021-03-10T18:38:09.459Z
Towards a Mechanistic Understanding of Goal-Directedness 2021-03-09T20:17:25.948Z
Coincidences are Improbable 2021-02-24T09:14:11.918Z
Chain Breaking 2020-12-29T01:06:04.122Z
Defusing AGI Danger 2020-12-24T22:58:18.802Z
TAPs for Tutoring 2020-12-24T20:46:50.034Z
The First Sample Gives the Most Information 2020-12-24T20:39:04.936Z
Does SGD Produce Deceptive Alignment? 2020-11-06T23:48:09.667Z
What posts do you want written? 2020-10-19T03:00:26.341Z
The Solomonoff Prior is Malign 2020-10-14T01:33:58.440Z
What are objects that have made your life better? 2020-05-21T20:59:27.653Z
What are your greatest one-shot life improvements? 2020-05-16T16:53:40.608Z
Training Regime Day 25: Recursive Self-Improvement 2020-04-29T18:22:03.677Z
Training Regime Day 24: Resolve Cycles 2 2020-04-28T19:00:09.060Z
Training Regime Day 23: TAPs 2 2020-04-27T17:37:15.439Z
Training Regime Day 22: Murphyjitsu 2 2020-04-26T20:18:50.505Z
Training Regime Day 21: Executing Intentions 2020-04-25T22:16:04.761Z
Training Regime Day 20: OODA Loop 2020-04-24T18:11:30.506Z
Training Regime Day 19: Hamming Questions for Potted Plants 2020-04-23T16:00:10.354Z
Training Regime Day 18: Negative Visualization 2020-04-22T16:06:46.138Z
Training Regime Day 17: Deflinching and Lines of Retreat 2020-04-21T17:45:34.766Z
Training Regime Day 16: Hamming Questions 2020-04-20T14:51:31.310Z
Mark Xu's Shortform 2020-03-10T08:11:23.586Z
Training Regime Day 16: Hamming Questions 2020-03-01T18:46:32.335Z
Training Regime Day 15: CoZE 2020-02-29T17:13:42.685Z
Training Regime Day 14: Traffic Jams 2020-02-28T17:52:28.354Z
Training Regime Day 13: Resolve Cycles 2020-02-27T17:45:07.845Z
Training Regime Day 12: Focusing 2020-02-26T19:07:15.407Z
Training Regime Day 11: Socratic Ducking 2020-02-25T17:19:57.320Z
Training Regime Day 10: Systemization 2020-02-24T17:20:15.385Z
Training Regime Day 9: Double-Crux 2020-02-23T18:08:31.108Z
Training Regime Day 8: Noticing 2020-02-22T19:47:03.898Z
Training Regime Day 7: Goal Factoring 2020-02-21T17:55:29.848Z
Training Regime Day 6: Seeking Sense 2020-02-20T17:33:29.011Z
Training Regime Day 5: TAPs 2020-02-19T18:11:05.649Z
Training Regime Day 4: Murphyjitsu 2020-02-18T17:33:12.523Z


Comment by Mark Xu (mark-xu) on Intermittent Distillations #4: Semiconductors, Economics, Intelligence, and Technological Progress. · 2021-07-21T23:14:12.825Z · LW · GW

Yeah that seems like a reasonable example of a good that can't be automated.

I think I'm mostly interested in whether these sorts of goods that seem difficult to automate will be a pragmatic constraint on economic growth. It seems clear that they'll eventually be ultimate binding constraints as long as we don't get massive population growth, but it's a separate question about whether or not they'll start being constraints early enough to prevent rapid AI-driven economic growth.

Comment by Mark Xu (mark-xu) on The topic is not the content · 2021-07-09T20:24:16.949Z · LW · GW


You might consider cross-posting this to the EA forum.

Comment by Mark Xu (mark-xu) on Intermittent Distillations #4: Semiconductors, Economics, Intelligence, and Technological Progress. · 2021-07-09T19:00:07.871Z · LW · GW

Thanks! I will try, although they will likely stay very intermittent.

Comment by Mark Xu (mark-xu) on rohinmshah's Shortform · 2021-06-26T23:43:57.027Z · LW · GW

My house implemented such a tax.

Re 1, we ran into some of the issues Matthew brought up, but all other COVID policies are implicitly valuing risk at some dollar amount (possibly inconsistently), so the Pigouvian tax seemed like the best option available.

Comment by Mark Xu (mark-xu) on Precognition · 2021-06-15T16:47:39.970Z · LW · GW

I'd be interested to see the rest of this list, if you're willing to share.

Comment by Mark Xu (mark-xu) on Rogue AGI Embodies Valuable Intellectual Property · 2021-06-12T23:34:06.463Z · LW · GW

Yeah, I'm really not sure how the monopoly -> non-monopoly dynamics play out in practice. In theory, perfect competition should drive the cost to the cost of marginal production, which is very low for software. I briefly tried getting empirical data for this, but couldn't find it, plausibly since I didn't really know the right search terms.

Comment by Mark Xu (mark-xu) on An Intuitive Guide to Garrabrant Induction · 2021-06-06T23:02:09.636Z · LW · GW

both of those sections draw from section 7.2 of the original paper

Comment by Mark Xu (mark-xu) on An Intuitive Guide to Garrabrant Induction · 2021-06-05T00:32:58.427Z · LW · GW

Yes, and there will always exist such a trader.

Comment by Mark Xu (mark-xu) on An Intuitive Guide to Garrabrant Induction · 2021-06-03T22:49:45.489Z · LW · GW

Thanks! Should be fixed now.

Comment by Mark Xu (mark-xu) on How refined is your art of note-taking? · 2021-05-20T19:11:47.445Z · LW · GW

It’s based on bullet points, which I find helpful. It also lets me reference other notes I’ve taken.

I like the idea of question notes. Thanks for the tip!

Comment by Mark Xu (mark-xu) on How refined is your art of note-taking? · 2021-05-20T05:45:00.802Z · LW · GW

The particular technology stack I use for notes on reading is {Instapaper, PDF Expert on iPad} -> Readwise -> Roam Research -> Summarize it.

To answer your specific questions:

  1. If I plan on summarizing, I tend to only highlight important bits. I write down any connections I make with other concepts. Readwise reminds me of 15 highlights I've taken in the past per day, which I've been doing for about half a year. I'm not sure if it's helpful, but the time cost is low, so I continue.

  2. Sometimes if I want to know what I thought about specific posts. If it's just high-level concepts, I'll generally just skim the relevant material. If I find myself looking something up more than twice, I'll put it into anki.

  3. No, but I only really study technical things. I find it difficult to summarize/remember history, plausibly because I don't change the way I take notes.

  4. Roam Research seems pretty good. RemNote is similar and incorporates more spaced repetition. SuperMemo allows one to create flashcards as they read (readwise does something similar, but the functionality is worse, I think [I've never used SuperMemo, but plan to try it]).

  5. Attention, future reference, and comprehension are all goals. The primary goal seems to be forcing connections with other ideas and forcing myself to have an opinion about what I'm reading at all.

Comment by Mark Xu (mark-xu) on Pre-Training + Fine-Tuning Favors Deception · 2021-05-09T01:57:58.700Z · LW · GW

thanks, fixed

Comment by mark-xu on [deleted post] 2021-05-01T00:31:18.232Z

Can you be more specific?

Comment by Mark Xu (mark-xu) on AMA: Paul Christiano, alignment researcher · 2021-04-29T17:28:57.245Z · LW · GW, a game that Paul develops

Comment by Mark Xu (mark-xu) on AMA: Paul Christiano, alignment researcher · 2021-04-29T05:28:40.283Z · LW · GW

How would you teach someone how to get better at the engine game?

Comment by Mark Xu (mark-xu) on AMA: Paul Christiano, alignment researcher · 2021-04-29T02:58:25.341Z · LW · GW

You've written multiple outer alignment failure stories. However, you've also commented that these aren't your best predictions. If you condition on humanity going extinct because of AI, why did it happen?

Comment by Mark Xu (mark-xu) on [Linkpost] Treacherous turns in the wild · 2021-04-27T18:01:31.841Z · LW · GW

This is a cool example, thanks!

Comment by Mark Xu (mark-xu) on Opinions on Interpretable Machine Learning and 70 Summaries of Recent Papers · 2021-04-14T22:52:58.870Z · LW · GW

I'm curious what "put it in my SuperMemo" means. Quick googling only yielded SuperMemo as a language learning tool.

Comment by Mark Xu (mark-xu) on Transparency Trichotomy · 2021-03-28T22:12:11.478Z · LW · GW

I agree it's sort of the same problem under the hood, but I think knowing how you're going to go from "understanding understanding" to producing an understandable model controls what type of understanding you're looking for.

I also agree that this post makes ~0 progress on solving the "hard problem" of transparency, I just think it provides a potentially useful framing and creates a reference for me/others to link to in the future.

Comment by Mark Xu (mark-xu) on Strong Evidence is Common · 2021-03-15T03:13:22.453Z · LW · GW

Yeah, I agree 95% is a bit high.

Comment by Mark Xu (mark-xu) on Open Problems with Myopia · 2021-03-11T23:38:25.355Z · LW · GW

One way of looking at DDT is "keeping it dumb in various ways." I think another way of thinking about is just designing a different sort of agent, which is "dumb" according to us but not really dumb in an intrinsic sense. You can imagine this DDT agent looking at agents that do do acausal trade and thinking they're just sacrificing utility for no reason.

There is some slight awkwardness in that the decision problems agents in this universe actually encounter means that UDT agents will get higher utility than DDT agents.

I agree that the maximum a posterior world doesn't help that much, but I think there is some sense in which "having uncertainty" might be undesirable.

Comment by Mark Xu (mark-xu) on Open Problems with Myopia · 2021-03-11T20:08:15.505Z · LW · GW

has been changed to imitation, as suggested by Evan.

Comment by Mark Xu (mark-xu) on Open Problems with Myopia · 2021-03-10T19:55:39.105Z · LW · GW

Yeah, you're right that it's obviously unsafe. The words "in theory" were meant to gesture at that, but it could be much better worded. Changed to "A prototypical example is a time-limited myopic approval-maximizing agent. In theory, such an agent has some desirable safety properties because a human would only approve safe actions (although we still would consider it unsafe)."

Comment by Mark Xu (mark-xu) on Open Problems with Myopia · 2021-03-10T19:52:10.922Z · LW · GW

Yep - I switched the setup at some point and forgot to switch this sentence. Thanks.

Comment by Mark Xu (mark-xu) on Coincidences are Improbable · 2021-02-25T00:48:50.346Z · LW · GW

This is brilliant.

Comment by Mark Xu (mark-xu) on Coincidences are Improbable · 2021-02-24T19:43:19.545Z · LW · GW

I am using the word "causal" to mean d-connected, which means not d-seperated. I prefer the term "directly causal" to mean A->B or B->A.

In the case of non-effects, the improbable events are "taking Benadryl" and "not reacting after consuming an allergy"

Comment by Mark Xu (mark-xu) on DanielFilan's Shortform Feed · 2021-02-14T21:51:42.945Z · LW · GW

I agree market returns are equal in expectation, but you're exposing. yourself to more risk for the same expected returns in the "I pick stocks" world, so risk-adjusted returns will be lower.

Comment by Mark Xu (mark-xu) on Ways to be more agenty? · 2021-01-05T15:01:04.782Z · LW · GW

I sometimes roleplay as someone role playing as myself, then take the action that I would obviously want to take, e.g. "wow sleeping regularly gives my character +1 INT!" and "using anki every day makes me level up 1% faster!"

Comment by Mark Xu (mark-xu) on Collider bias as a cognitive blindspot? · 2020-12-31T16:10:03.317Z · LW · GW

If X->Z<-Y, then X and Y are independent unless you're conditioning on Z. A relevant TAP might thus be:

  • Trigger: I notice that X and Y seem statistically dependent
  • Action: Ask yourself "what am I conditioning on?". Follow up with "Are any of these factors causally downstream of both X and Y?" Alternatively, you could list salient things causally downstream of either X or Y and check the others.

This TAP unfortunately abstract because "things I'm currently conditioning on" isn't an easy thing to list, but it might help.

Comment by Mark Xu (mark-xu) on Chain Breaking · 2020-12-29T20:21:21.553Z · LW · GW

yep, thanks

Comment by Mark Xu (mark-xu) on Great minds might not think alike · 2020-12-29T16:49:54.845Z · LW · GW

Here are some possibilities:

  • great minds might not think alike
  • untranslated thinking sounds untrustworthy
  • disagreement as a lack of translation
Comment by Mark Xu (mark-xu) on Open & Welcome Thread - December 2020 · 2020-12-27T17:36:52.249Z · LW · GW

Transferring money is usually done via ACH bank transfer, which is usually accessed in the "deposit" tab of the "transfers" tab of your investment account.

I'm not sure how to be confident in the investment in general. One simple way is to double-check the ticker symbol, e.g. MSFT for Microsoft, actually corresponds to the company you want. For instance, ZOOM does not correspond to Zoom Technologies, rather ZM is the correct ticker.

Talking to a financial advisor might be helpful. I have been told r/personalfinance is a reasonable source for advice, although I've never checked it out thoroughly.

Comment by Mark Xu (mark-xu) on Defusing AGI Danger · 2020-12-26T18:53:43.160Z · LW · GW

My opposite intuition is suggested by the fact that if you're trying to guess correctly a series of random digits with 80% "1" and 20% "0", then you should always guess "1".

I don't quite know how to model cross-pollination and diminishing sort of returns. I think working on both for the information value is likely going to be very good. It seems hard to imagine a scenario where you're robustly confident that one project is 80% better taking diminishing returns into account without being able to create a 3rd project with the best features of both, but if you're in that scenario I think just spending all your efforts on the 80% project seems correct.

One example is deciding between 2 fundamentally different products your startup could be making. We also supposed that creating an MVP of either product that would provide information would take a really long time. In this situation, if you suspect one of them is 60% likely to be better than the other it would be less useful to spend your time in a 60/40 split rather than building the MVP of the one likely to be better and reevaluating after getting more information.

The version of your claim that I agree with is "In your current epistemic state, you should spend all your time pursuing the 80% project, but the 80% probably isn't that robust, working on a project has diminishing returns, and other projects will give more information value, globally the amount of time you expect to spend on the 80% project is about 80%."

Comment by Mark Xu (mark-xu) on Defusing AGI Danger · 2020-12-26T18:06:20.403Z · LW · GW

I absolutely agree that I'm not arguing for "safety by default".

I don't quite agree that you should split effort between strategies, i.e. it seems likely that if you think 80% disaster by default, you should dedicate 100% of your efforts to that world.

Comment by Mark Xu (mark-xu) on What trade should we make if we're all getting the new COVID strain? · 2020-12-25T22:54:51.186Z · LW · GW

Is what you mean by "naive instrument" SPY put options?

Comment by Mark Xu (mark-xu) on Operationalizing compatibility with strategy-stealing · 2020-12-25T17:14:16.884Z · LW · GW

Using the perspective from The ground of optimization means you can get rid of the action space and just say "given some prior and some utility function, what percentile of the distribution does this system tend to evolve towards?" (where the optimization power is again the log of this percentile)

We might then say that an optimizing system is compatible with strategy stealing if it's retargetable for a wide set of utility functions in a way that produces an optimizing system that has the same amount of optimization power.

An AI that is compatible with strategy stealing is one such way to of producing an optimizing system that is compatible with strategy stealing with a particularly easy form of retargeting, but the difficulty of retargeting provides another useful dimension along which optimizing systems vary, e.g. instead of the optimization power the AI can direct towards different goals, you have a space of the "size" of allowed retargeting and the optimization power applied toward the goal for all goals and retargeting sizes.

Comment by Mark Xu (mark-xu) on Defusing AGI Danger · 2020-12-25T04:06:18.905Z · LW · GW

Thanks! Also, oops - fixed.

Comment by Mark Xu (mark-xu) on The First Sample Gives the Most Information · 2020-12-24T23:41:26.058Z · LW · GW

The game play in Decrypto, Chameleon, and Spyfall are similar to the game you just suggested.

Comment by Mark Xu (mark-xu) on Is there an easy way to turn a LW sequence into an epub? · 2020-12-24T21:35:28.892Z · LW · GW

A button that changes a sequence into one really long post might also be sufficient when paired with other tools, e.g. instapaper.

Comment by Mark Xu (mark-xu) on 100 Tips for a Better Life · 2020-12-24T01:15:25.358Z · LW · GW

for linux, the put windows plugin works reasonably well for me:

there's also writing wmctrl scripts:

Comment by Mark Xu (mark-xu) on Training Regime Day 8: Noticing · 2020-12-23T21:50:03.246Z · LW · GW


Comment by Mark Xu (mark-xu) on Steelmanning Divination · 2020-12-23T17:46:49.236Z · LW · GW

It's been a while, but:

  • When my brain says "I'm tired stop working" it mostly means "I'm tired of working on this thing stop working on this thing". Switching tasks often allows me to maintain productivity.
  • Trying to think while on a walk doesn't work for me at all. Thinking without something to write with is impossible.
  • Social media should probably be avoided.
  • Writing things as a way to clarify thinking works much better than I expected.
Comment by Mark Xu (mark-xu) on Steelmanning Divination · 2020-12-18T16:00:58.380Z · LW · GW

This post made me try adding more randomness to my life for a week or so. I learned a small amount. I remain excited about automated tools that help do things like this, e.g. recent work from Ought.

Comment by Mark Xu (mark-xu) on Being the (Pareto) Best in the World · 2020-12-18T15:59:31.997Z · LW · GW

I tend to try to do things that I think are in my comparative advantage. This post hammered home the point that comparative advantage exists along multiple dimensions. For example, as a pseudo-student, I have almost no accumulated career capital, so I risk less by doing projects that might not pan out (under the assumption that career capital gets less useful over time). This fact can be combined other properties I have to more precisely determine comparative advantage.

This post also gives the useful intuition that being good at multiple things exponentially cuts down the number of people you're competing with. I use this heuristic a reasonable amount when trying to decide the best projects to be working on.

Comment by Mark Xu (mark-xu) on Understanding “Deep Double Descent” · 2020-12-18T15:39:36.570Z · LW · GW

This post gave a slightly better understanding of the dynamics happening inside SGD. I think deep double descent is strong evidence that something like a simplicity prior exists in SGG, which might have actively bad generalization properties, e.g. by incentivizing deceptive alignment. I remain cautiously optimistic that approaches like Learning the Prior can get circumnavigate this problem.

Comment by Mark Xu (mark-xu) on Cultural accumulation · 2020-12-07T22:28:31.073Z · LW · GW

Why do you believe the Manhattan project cost about $3.3T? Quick googling yields ~$20B in 2007 dollars.

Edit: further googling shows WWII costs about ~$4T, so maybe you confused the two numbers? I'm pretty surprised that the Manhattan project is only ~1% of the cost of WWII, so maybe something is going on.

Comment by Mark Xu (mark-xu) on A space of proposals for building safe advanced AI · 2020-11-21T17:17:51.676Z · LW · GW

I claim that if we call the combination of the judge plus one debater Amp(M), then we can think of the debate as M* being trained to beat Amp(M) by Amp(M)'s own standards.

This seems like a reasonable way to think of debate.

I think, in practice (if this even means anything), the power of debate is quite bounded by the power of the human, so some other technique is needed to make the human capable of supervising complex debates, e.g. imitative amplification.

Comment by Mark Xu (mark-xu) on A space of proposals for building safe advanced AI · 2020-11-20T16:15:11.654Z · LW · GW

Debate: train M* to win debates against Amp(M).

I think Debate is closer to "train M* to win debates against itself as judged by Amp(M)".

Comment by Mark Xu (mark-xu) on Mark Xu's Shortform · 2020-11-08T22:34:24.948Z · LW · GW

If you have DAI right now, minting on and swapping yTrump for nTrump on is an almost guaranteed 15% profit.

Comment by Mark Xu (mark-xu) on Does SGD Produce Deceptive Alignment? · 2020-11-07T16:40:33.647Z · LW · GW

Yep. Meant to say "if a model knew that it was in its last training episode and it wasn't going to be deployed." Should be fixed.