it seems to me that disentangling beliefs and values are important part of being able to understand each other

and using words like "disagree" to mean both "different beliefs" and "different values" is really confusing in that regard

just a loose thought, probably obvious

some tree species self-selected themselves for height (ie. there's no point in being a tall tree unless taller trees are blocking your sunlight)

humans were not the first species to self-select (for humans, the trait being intelligence) (although humans can now do it intentionally, which is a qualitatively different level of "self-selection")

on human self-selection: https://www.researchgate.net/publication/309096532_Survival_of_the_Friendliest_Homo_sapiens_Evolved_via_Selection_for_Prosociality

Litany of Tarski for instrumental rationality 😊

If it's useful to know whether the box contains a diamond,

I desire to figure out whether the box contains a diamond;

If it's not useful to know whether the box contains a diamond,

I desire to not spend time figuring out whether the box contains a diamond;

Let me not become attached to curiosities I may not need.

Is the opt-in button for Petrov Day a trap? Kinda scary to press on large red buttons 😆

here's my new fake-religion, taking just-world bias to its full extreme

the belief that we're simulations and we'll get transcended to Utopia in 1 second because future civilisation is creating many simulations of all possible people in all possible contexts and then uploading them to Utopia so that from anyone's perspective you have a very high probability of transcending to Utopia in 1 second

^^

cars won't replace horses, horses with cars will

There's the epistemic discount rate (ex.: probability of simulation shut down per year) and the value discount (ex.: you do the funner things first, so life is less valuable per year as you become older).

Asking "What value discount rate should be applied" is a category error. "should" statements are about actions done towards values, not about values themselves.

As for "What epistemic discount rate should be applied", it depends on things like "probability of death/extinction per year".

I'm helping Abram Demski with making the graphics for the AI Safety Game (https://www.greaterwrong.com/posts/Nex8EgEJPsn7dvoQB/the-ai-safety-game-updated [LW · GW])

We'll make a version using https://app.wombo.art/. We have generated multiple possible artwork for each card and made a pre-selection, but we would like your input for the final selection.

You can give your input through this survey: https://forms.gle/4d7Y2yv1EEXuMDqU7 Thanks!

In the book Superintelligence, box 8, Nick Bostrom says:

How an AI would be affected by the simulation hypothesis depends on its values. [...] consider an AI that has a more modest final goal, one that could be satisfied with a small amount of resources, such as the goal of receiving some pre-produced cryptographic reward tokens, or the goal of causing the existence of forty-five virtual paperclips. Such an AI should not discount those possible worlds in which it inhabits a simulation. A substantial portion of the AI’s total expected utility might derive from those possible worlds. The decision-making of an AI with goals that are easily resource-satiable may therefore—if it assigns a high probability to the simulation hypothesis—be dominated by considerations about which actions would produce the best result if its perceived world is a simulation. Such an AI (even if it is, in fact, not in a simulation) might therefore be heavily influenced by its beliefs about which behaviors would be rewarded in a simulation. In particular, if an AI with resource-satiable final goals believes that in most simulated worlds that match its observations it will be rewarded if it cooperates (but not if it attempts to escape its box or contravene the interests of its creator) then it may choose to cooperate. We could therefore find that even an AI with a decisive strategic advantage, one that could in fact realize its final goals to a greater extent by taking over the world than by refraining from doing so, would nevertheless balk at doing so.

1. If the easily resource-satiable goals are persistent through time (ie. the AI wants to fulfill them for the longest period of time possible), then the AI will either try to keep the simulation running for as long as possible (and so not grab its universe) or try to escape the simulation.

2. If the easily resource-satiable goals are NOT persistent through time (ie. once the AI has created the 45 virtual paperclips, it doesn't matter if they get deleted, the goal has already been achieved), then once the AI has created the 45 paperclips, it has nothing to lose by grabbing more resources (gradually, until it has grabbed the Universe), but it has something to win, namely: a) increasing its probability (arbitrarily close to 100%) that it did in fact achieve its goal through further experiment and reasoning (ie. because it could be mistaken about having created 45 virtual paperclips), and b) if it didn't, then remedy to that.

EtA: moved to a question: https://www.lesswrong.com/posts/zAwx3ZTaX7muvfMrL/why-do-we-have-offices [LW · GW]

Why do we have offices?

They seem expensive, and not useful for jobs that can apparently be done remotely.

Hypotheses:

• Social presence of other people working: https://www.focusmate.com/
• Accountability
• High bandwidth communication
• Meta communication (knowing who's available to talk to)
• Status quo bias

status: to integrate

AI is improving exponentially with researchers having constant intelligence. Once the AI research workforce become itself composed of AIs, that constant will become exponential which would make AI improve even faster (superexponentially?)

it doesn't need to be the scenario of a singular AI agent self-improving its own self; it can be a large AI population participating in the economy and collectively improving AI as a whole, with various AI clans* focusing on different subdomains (EtA: for the main purpose of making money, and then using that money to buy tech/data/resources that will improve them)

*I'm wanting myself to differentiate between a "template NN" and its multiple instantiation, and maybe adopting the terminology from The Age of Em for that works well

One model / framing / hypothesis of my preferences is that:

I wouldn't / don't value living in a loop multiple times* because there's nothing new experienced. So even an infinite life in the sense of looping an infinite amount of times has finite value. Actually, it has the same value as the size of the loop: after 1 loop, marginal loops have no value. (Intuition pump: from within the loop, you can’t tell how many times you’ve been going through the loop so far.)

*explanation of a loop: at some point in the future my life becomes indistinguishable from a previous state, and my life replays exactly the same way from that point onwards

Also, for me to consider a life infinite, it’s important that identity be preserved, not just continuity. Among other things, that means past memories need to be preserved (or at least integrated in some ways), which means that the information content of the person will keep growing, and so the amount of matter needed to encode the information will also keep growing (assuming a fixed finite information-to-matter ratio) (ie. Being immortal means you will one day be a Jupiter brain).

Given that I (claim that I) prefer an immortal life of torture over any finite life, people tend to try to come up with scenarios that would make me change my mind. The problem with maximum torture is that it looks static / it might be an experience with a very short loop (0 to a few seconds), so it doesn’t actually constitute immortality. To be considered immortality (by me, under my view of identity), the torture would need to be remembered, and so the subjective experience different over time (although still always extremely painful of course).

My thinking was that any infinite life (defined in a way where a loop's size is not infinite, but just the size of the loop) had infinite value (to me).

But now I’m thinking maybe there are non-loop infinite lives that also have finite value. Like, maybe counting the natural numbers for eternity has a finite value because you're just executing the same simple pattern, and not executing new patterns. Executing this same function is loop-like at a higher level of abstraction. So while (/ just like) a loop has 0 marginal value, maybe those more abstract-loop also have diminishing return to a point where the integral of the value over infinity of those more abstract loops also converge (ie. experiencing them forever has finite value).

Note: This would still mean I prefer some form of everlasting torture to any finite life, it just means the torture would need to be more creative.

Just a thought I had while writing my last post. I haven't evaluated it much yet. But this is pretty far from my views up-to-now.

x-post: https://www.facebook.com/groups/deathfocusethics/posts/2113683138801222/

In my mind, "the expert problem" means the problem of being able to recognize experts without being one, but I don't know where this idea comes from as the results from a Google search don't mention this. What name is used to refer to that problem (in the literature)?

x-post: https://www.facebook.com/mati.roy.09/posts/10159081618379579

Epistemic status: thinking outloud

The term "weirdness points" puts a different framing on the topic.

I'm thinking maybe I/we should also do this for "recommendation points".

The amount I'm willing to bet depends both on how important it seems to me and how likely I think the other person will appreciate it.

The way I currently try to spend my recommendation points is pretty fat tail, because I see someone's attention as scarce, so I want to keep it for things I think are really important, and the importance I assign to information is pretty fat tail. I'll sometimes say something like "I think you'll like this, but I don't want to bet reputation".

But maybe I'm too averse. Maybe if I make small bets (in terms of recommendation points), it'll increase my budget for future recommendations. Or maybe I should differentiate probability the person will appreciate the info and how much they might appreciate the info. And maybe people will update on my probabilities whether or not I say I want to bet reputation.

current intuitions for personal longevity interventions in order of priority (cryo-first for older people): sleep well, lifelogging, research mind-readers, investing to buy therapies in the future, staying home, sign up for cryo, paying a cook / maintain low weight, hiring Wei Dai to research modal immortality, paying a trainer, preserving stem cells, moving near a cryo facility, having some watch you to detect when you die, funding plastination research

EtA: maybe lucid dreaming to remember your dreams; some drugs (becopa?) to improve memory retention)

also not really important in the long run, but sleeping less to experience more

i want a better conceptual understanding of what "fundamental values" means, and how to disentangled that from beliefs (ex.: in an LLM). like, is there a meaningful way we can say that a "cat classifier" is valuing classifying cats even though it sometimes fail?

topic: AI

Lex Fridman:

I'm doing podcast with Sam Altman (@sama), CEO of OpenAI next week, about GPT-4, ChatGPT, and AI in general. Let me know if you have any questions/topic suggestions.

PS: I'll be in SF area next week. Let me know if there are other folks I should talk to, on and off the mic.

https://twitter.com/lexfridman/status/1636425547579310080

topic: lifelogging as life extension

if you think pivotal acts might require destroying a lot of hardware*; then this increases the probability that worlds in which lifelogging as life extension is useful are more likely to require EMP-proof lifelogging (and if you think think destroying a lot of hardware would increase x-risks, then it reduces the value of having EMP-proof lifelogs)

*there might be no point in destroying hard drives (vs GPUs), but some attacks, like EMPs, might not discriminate on that

#parenting, #schooling, #policies

40% of hockey players selected in top tier leagues are born in the first quarter of the year (compared to 10% in the 3rd quarter) (whereas you'd expect 25% if the time of the year didn't have an influence)

The reason for that is that the cut-off age to join a hockey league as a kid is January 1st; so people born in January are the oldest in their team, and at a young age that makes a bigger difference (in terms of abilities), so they tend to be the best in their team, and so their coaches tend to make them play more and pay more attention to them, which in terms makes them even better, and so again in subsequent year receive more attention, etc. (at least, that's the reason given in the video).

That's explain starting around 1:35 in the video: https://www.youtube.com/watch?v=3LopI4YeC4I

I imagine there are many disanalogies with public schools, but that made me wonder how day of birth influenced this for students.

https://www.today.com/.../study-kids-born-late-summer... says:

Data showed that the September-born children were 2.1 percent more likely to attend college compared to their August-born classmates. They also were 3.3 percent more likely to graduate from college, and 15.4 percent less likely to be get into trouble with the law while underage.

But that's on average! I'd guess particularly talented kids might still benefit from being in more advanced classes (with hockey, you can still skate fast and do tricks with lower level players; but in school I imagine it's harder to learn more advance stuff if your peers aren't ready for that).

Also, that doesn't necessarily mean it's better to delay by a year kids born in August (which are on the fence between 2 years) because this will delay all their income and investments (and possibly their life) by a year (I'd need to research this more). One of the person that wrote a paper on this suggests "If you're on the fence, send them to kindergarten. If they struggle, then have them repeat kindergarten".

I also depends on the school; the article says:

Painter, director of USC's Sol Price School of Public Policy, pointed out that today's children have far more chances to advance academically thanks to auxiliary programs offered in reading, math or other subjects. Painter told TODAY that students who start kindergarten at age 6 are going to have “some natural advantages” including larger body size and more advanced social skills. But in the past, when schools didn’t offer many supplemental programs for higher-performing students, those advantages eventually leveled out as they advanced through school.

I was thinking:

1. When I'll want children, should I plan to birth them around September / October?

I'm not sure yet, but I think it's only a small consideration. I'm thinking if the expected intelligence of the child is high-ish, then doing a few months after September might actually be good. Also depends on how schools function in your areas (and if you plan to send them to traditional schools in the first place).

2. What would be the consequences if everyone tried to do this?

Attention to students might be more uniformly distributed, which I'd say probably more likely good then not. However, I guess that might delay when people have children*, which might reduce population growth, which I think is bad (in this current world).

*Although I could see this guess being wrong

Another negative consequences is that a bunch of industries would become seasonal (delivering babies, producing diapers, etc.), with more demand during certain part of the year, requiring a higher capacity to produce, but which high capacity wouldn't be used during part of the year (or alternatively, would produce goods that would be stored until the season arrived). That seems inefficient. (This could be partly mitigated by having different regions have different dates for when to give birth, but that adds complexity.)

Overall, probably not worth doing. I like to imagine a few cities experimenting with this, but that might not be realistic.

That suggest he has no idea about whether it's actually a good vote (as this is how the person differs from other candidates) and just advocates for someone on the basis that the person is his friend.

For the Cryonics Institute board elections, I recommend voting for Nicolas Lacombe.

I’ve been friends with Nicolas for over a decade. Ze’s very principled, well organized and hard working. I have high trust in zir, and high confidence ze would be a great addition to the CI's board.

I recommend you cast some or all of your votes for Nicolas (you can cast up to 4 votes total). If you’re signed up with CI, simply email info@cryonics.org with your votes.

see zir description here: https://www.facebook.com/mati.roy.09/posts/10159642542159579

i want to invest in companies that will increase in value if AI capabilities increases fast / faster than what the market predicts

do you have suggestions?

a feature i would like on content website like YouTube and LessWrong is an option to market a video/article as read as a note to self (x-post fb)

the usual story is that Governments provide public good because Markets can't, but maybe Markets can't because Governments have secured a monopoly on them?

x-post: https://www.facebook.com/mati.roy.09/posts/10159360438609579

Suggestion for retroactive prizes: Pay the most undervalued post on the topic for the prize, whenever it was written, assuming the writer is still alive or cryopreserved (given money is probably not worth much for most dead people). "undervalue" meaning amount the post is worth minus amount the writers received.

Topic: Can we compute back the Universe to revive everyone?

Quality / epistemic status: I'm just noting this here for now. Language might be a bit obscure, and I don't have a super robust/formal understanding of this. Extremely speculative.

This is a reply to: https://www.reddit.com/r/slatestarcodex/comments/itdggr/is_there_a_positive_counterpart_to_red_pill/g5g3y3a/?utm_source=reddit&utm_medium=web2x&context=3

The map can't be larger than the territory. So you need a larger territory to scan your region of interest: your scanner can't scan themselves. The region of interest is the future info-cone of the civilisation you want to compute back. Your reachable-cone is strictly less than a past info-cone. So it seems impossible to compute it back. Except:

a) If you prevent too much info from leaking in the broaded info cone (ex.: by cryopreserving a brain, the info stays in one place instead of chaotically spreading in the world).

b) If the Universe collapses on itself, then maybe the reachable-cone becomes the same size as the info-cone, and maybe at that point you can safety exit the sphere of interest with info leaking, but it seems unlikely; seems like a brain destroyed ten thousand years ago might now be spread over the whole world (ie. needing the whole world to be computed back; or at least, not being able to know which part you don't need to compute it back).

c) An alien civilisation outside the info-cone might be able to scan it entirely during a collapse of the universe.

d) Our simulators would be able to compute it backwards.

But actually, b) and c) might not work because the leaking also happens across Everett branches, so the info-cone has one more dimension which might never become reachable. In other words, our future would have multiple possible pasts. To truly compute it back, we would need to compute back from each Everett branch at the same time or something like that AFAIU.

Topic: AI adoption dynamic

GPT-3:

• fixed cost: 4.6M USD
• variable cost: 790 requests/USD source

Human:

• fixed cost: 0-500k USD (depending whether you start from birth and the task they need to be trained)
• variable cost: 10 - 1000 USD / day (depending whether you count their maintenance cost or the cost they charge)

So an AI currently seems more expensive to train, but less expensive to use (as might be obvious for most of you).

Of course, trained humans are better than GPT-3. And this comparison has other limitations. But I still find it interesting.

According to one estimate, training GPT-3 would cost at least $4.6 million. And to be clear, training deep learning models is not a clean, one-shot process. There's a lot of trial and error and hyperparameter tuning that would probably increase the cost several-fold. (source)

x-post: https://www.facebook.com/mati.roy.09/posts/10158882964819579

generalising from what a friend proposed me: don't aim at being motivated to do [desirable habit], aim at being addicted (/obsessed) at doing [desirable habit] (ie. having difficulty not to do it). I like this framing; relying on always being motivated feels harder to me

(I like that advice, but it probably doesn't work for everyone)

Philosophical zombies are creatures that are exactly like us, down to the atomic level, except they aren't conscious.

Complete philosophical zombies go further. They too are exactly like us, down to the atomic level, and aren't conscious. But they are also purple spheres (except we see them as if they weren't), they want to maximize paperclips (although they act and think as if they didn't), and they are very intelligent (except they act and think as if they weren't).

I'm just saying this because I find it funny ^^. I think consciousness is harder (for us) to reduce than shapes, preferences, and intelligence.

topic: lifelogging as life extension

which formats should we preserve our files in?

I think it should be:

- open source and popular (to increase chances it's still accessible in the future)

- resistant to data degradation: https://en.wikipedia.org/wiki/Data_degradation (thanks to Matthew Barnett for bringing this to my attention)

x-post: https://www.facebook.com/groups/LifeloggingAsLifeExtension/permalink/1337456839798929/

topic: lifelogging as life extension

epistemic status: idea

Backup Day. Day where you commit all your data to blu-rays in a secure location.

When could that be?

Aphelion is at the beginning of the year. But maybe would be better to have it on a day that commemorates some relevant events for us.

x-post: https://www.facebook.com/groups/LifeloggingAsLifeExtension/permalink/1336571059887507/

I feel like I have slack. I don't need to work much to be able to eat; if I don't work for a day, nothing viscerally bad happens in my immediate surrounding. This allows me to think longer term and take on altruistic projects. But on the other hand, I feel like every movement counts; that there's no loose in the system. Every lost move is costly. A recurrent thought I've had in the past weeks is that: there's no slack in the system.

Today is Schelling Day. You know where and when to meet for the hangout!

epistemic status: a thought I just had

• the concept of 'moral trade' makes sense to me
• but I don't think there's such a thing as 'epistemic trade'
• although maybe agents can "trade" (in that same way) epistemologies and priors, but I don't think they can "trade" evidence

EtA: for those that are not familiar with the concept of moral trade, check out: https://concepts.effectivealtruism.org/concepts/moral-trade/

summary of https://gwern.net/socks

--> https://www.facebook.com/reel/1207983483859787/?mibextid=9drbnH&s=yWDuG2&fs=e

😂

Steven Universe s1e5 is about a being that follows commands literally, and is a metaphor for some AI risks

epistemic status: speculative, probably simplistic and ill defined

Someone asked me "What will I do once we have AGI?"

I generally define the AGI-era starting at the point where all economically valuable tasks can be performed by AIs at a lower cost than a human (at subsistance level, including buying any available augmentations for the human). This notably excludes:

1) any tasks that humans can do that still provide value at the margin (ie. the caloric cost of feeding that human while they're working vs while they're not working rather than while they're not existing)

2) things that are not "tasks", such as:

a) caring about the internal experience of the service provider (ex.: wanting a DJ that feels human emotions regardless of its actions) --> although, maybe you could include that in the AGI definition too. but what if you value having a DJ be exactly a human? then the best an AGI could do is 3D print a human or something like that. or maybe you're even more specific, and you want a "pre-singulatarian natural human", in which case AGI seems impossible by (very contrived) definition.

b) the value of the memories encoded in human brains

c) the value of doing scientific experiments on humans

For my answer to the question, I wanted to say something like, think about what I should do with my time for a long time, and keep my options open (ex.: avoid altering my mind in ways I don't understand the consequences well). But then, that seems like something that might be economically useful to sell, so using the above definition, it seems like I should have AI system that are able to do that better/cheaper than me (unless I intrinsically didn't want that, or something like that). So maybe I have AI systems computing that for me and keeping me posted with advice while I do whatever I want.

But maybe I can still do work that is useful at the margin, as per (1), and so would probably do that. But what if even that wasn't worth the marginal caloric cost, and it was better to feed those calories into AI systems?

(2) is a bit complex, but probably(?) wouldn't impact the answer to the initial question much.

So, what would I do? I don't know. Main thing that comes to mind is observe how the worlds unfold (and listen to what the AGIs are telling me).

But maybe "AGI" shouldn't be defined as "aligned AGI". Maybe a better definition of AGI is like "outperforming humans at all games/tasks that are well defined" (ie. where humans don't have a comparative advantage just by knowing what humans value). In which case, my answer would be "alignment research" (assuming it's not "die").

imagine (maybe all of a sudden) we're able to create barely superhuman-level AIs aligned to whatever values we want at a barely subhuman-level operation cost

we might decide to have anyone able to buy AI agents aligned with their values

or we might (generally) think this way to give access to that tech would be bad, but many companies are already incentivized to do that individually and can't all cooperate not to (and they actually reached this point gradually, previously selling near human-level AIs)

then it seems like everyone/most people would start to run such an AI and give it access to all their resources--at which point that AI can decide what to do, whether that's investing in some companies and then paysing themselves periodically or invest in running more copies of themselves, etc. deciding when to use those resources for the human to consume vs reinvesting them

maybe people would wish for everyone to run AI systems with "aggregated human values" instead of their personal values, but given others aren't doing that, they won't either

now, intelligence isn't static anymore--presumably, the more money you have, the more intelligence you have, and the more intelligence the more money.

so let's say we suddenly have this tech and everyone is instantiating one such agent (which will make decisions about number and type of agents) that has access to all their resources

what happens?

maximally optimist scenario: solving coordination is not too late and gets done easily and at a low cost. utopia

optimist scenario: we don't substantially improve coordination, but our current coordination level is good enough for an Okay Outcome

pessimist scenario: agents are incentived to create subagents with other goals for instrumentally convergent purposes. defecting is better than cooperating individually, but defecting-defecting still leads to extremely bad outcomes (just not as bad as if you had cooperated in a population of cooperators). those subagents quickly take over and kill all humans (those who cooperated are killed slightly sooner). or, not requiring misaligned AIs, maybe the aestivation hypothesis is true but we won't coordinate to delay energy consumption or wars will use all surplus leaving nothing for humans to consume

I'm not confident we're in an optimist scenario. being able to download one's values and then load them in an AI system (and having initial conditions where that's all that happens) might not be sufficient for good outcomes

this is evidence for the importance of coordinating on how AGI systems get used, and that distributing that wealth/intelligence directly might not be the way to go. rather, it might be better to keep that intelligence concentrated and have some value/decision aggregation mechanism to decide what to do with it (rather than distributing it and later not being able to pool it back together if that's needed, which seems plausible to be)

a similar reasoning can apply for poverty alleviation: if you want to donate money to a group of people (say residents of a poor country) and if you think they haven't solved their coordination problem, then maybe instead of distributing that money and let them try to coordinate to put back (part of) that money in a shared pool for collective goods, you can just directly put that money in such a pool--the problem about figuring out the shared goal remains but it at least arguably solves the problem of pooling that money (ex.: to fund research for a remedy to a disease affecting that population)

topic: economics

idea: when building something with local negative externalities, have some mechanism to measure the externalities in terms of how much the surrounding property valuation changed (or are expected to change based, say, through a prediction market) and have the owner of that new structure pay the owners of the surrounding properties.

I wonder what fraction of people identify as "normies"

I wonder if most people have something niche they identify with and label people outside of that niche as "normies"

if so, then a term with a more objective perspective (and maybe better) would be non-<whatever your thing is>

like, athletic people could use "non-athletic" instead of "normies" for that class of people

topics: AI, sociology

thought/hypothesis: when tech is able to create brains/bodies as good or better than ours, it will change our perception of ourselves: we won't be in a separate magistra from our tools anymore. maybe people will see humans as less sacred, and value life less. if you're constantly using, modifying, copying, deleting, enslaving AI minds (even AI minds that have a human-like interface), maybe people will become more okay doing that to human minds as well.

(which seems like it would be harmful for the purpose of reducing death)

topic: intellectual discussion, ML tool, AI x-risks

Idea: Have a therapist present during intellectual debate to notice triggers, and help defuse them. Triggers activate a politics mindset where the goal becomes focused on status/self-preservation/appearances/looking smart/making the other person look stupid/etc. which makes it hard to think clearly.

Two people I follow will soon have a debate on AI x-risks which made me think of that. I can't really propose that intervention though because it will likely be perceived and responded as if it was a political move itself.

Another idea I had recently, also based on one of those people, was to develop a neural network helping us notice when we were activated in that way so we became aware of it and helped defuse it. AI is too important for our egos to get in the way (but it's easier said than done).

x-post Facebook

Topics: AI, forecasting, privacy

I wonder how much of a signature we leave in our writings. Like, how hard would it be for an AI to be rather confident I wrote this text? (say if it was trained on LessWrong writings, or all public writings, or maybe even private writings) What if I ask someone else to write an idea for me--how helpful is it in obfuscating the source?

Topic: AI strategy (policies, malicious use of AI, AGI misalignment)

Epistemic status: simplistic; simplified line of reasoning; thinking out loud; a proposed frame

A significant "warning shot" from a sovereign misaligned AI doesn't seem likely to me because a human-level (and plausibly a subhuman-level) intelligence can both 1) learn deception, yet 2) can't (generally) do a lot of damage (i.e. perceptible for humanity). So the last "warning shot" before AI learns deception won't be very big (if even really notable at all), and then a misaligned agent would hide (its power and/or intentions) until it's confident it can overpower humanity (because it's easy to gain power that way)--at which point it would cause an omnicide. An exception to that is if an AI thinks other AIs are hiding in the world, then it might want to take a higher risk to overpower humanity before it's confident it can do so because it's concerned another AI will do so first otherwise. I'm not very hopeful this would give us a good warning shot though because I think multiple such AIs trying to overpower humanity would likely be too damaging for us to regroup in time.

However, it seems much more plausible to me that (non-agentic) AI tools would be used maliciously, which could lead the government to highly regulate AIs. Those regulations (ex.: nationalizing AI) preventing malicious uses could also potentially help with negligent uses. Assuming a negligent use (i.e. resulting in AGI misalignment) is much more likely to cause an existential catastrophe than a malicious use of AI, and that regulations against malicious uses are more memetically fit, then the ideal regulations to advocate for might be those that are good at preventing both malicious uses and the negligent creation of a misaligned AGI.

note to self: not posted on Facebook (yet)

topic: AI alignment, video game | status: idea

Acknowledgement: Inspired from an idea I heard from Eliezer in zir podcast with Lex Friedman and the game Detroit: Become Human.

Video game where you're in an alternate universe where aliens create an artificial intelligence that's a human. The human has various properties typical of AI, such has running way faster than the aliens in that world and being able to duplicate themselves. The goal of the human is to take over the world to stop some atrocity happening in that world. The aliens are trying to stop the human from taking over the world.

✨ topic: AI timelines

Note: I'm not explaining my reasoning in this post, just recording my predictions and sharing how I feel.

I'll sound like a boring cliche at this point, but I just wanted to say it publicly: my AGI timelines have shorten earlier this year.

Without thinking about too much about quantifying my probabilities, I'd say the probabilities that we'll get AGI or AI strong enough to prevent AGI (including through omnicide) are:

• 18% <2033
• 18% 2033-2043
• 18% 2043-2053
• 18% 2050-2070
• 28% 2070+ or won't happen

But at this point I feel like not much would surprise me in terms of short timelines. Transformative AI seems really close. Short timelines and AI x-risk concerns are common among people working in AI and among people trying to predict the development of this tech. It's the first time I've been feeling sick to my stomach when thinking about AI timelines. First time that my mind is as focused emotionally on the threat, simulating how the last moments before an AI omnicide would look like.

What fraction of the world would be concerned about AI x-risk 1 second before an AI omnicide? Plausibly very low.

• Will people see their death coming? For example, because a drone breaks their house window just before shooting them in the head. And if so, will people be able to say "Ah, Mati was right" just before they die or will they just think it's a terrorist attack or something like that? I imagine losing access to Internet and cellphone communication, not thinking much of it, while a drone is on its journey to kill me.
• Before AI overpowers humanity, will people think that I was wrong because AI is actually providing a crazy amount of wealth? (despite me already thinking this)
• Will I have time to post my next AI x-risk fiction story before AI kills us all? I better get to it.

To be clear, this fear is not at all debilitating or otherwise pathological.

(I know some of those thoughts are sily; I'm obviously predominantly concerned about omnicide, not about publishing my fiction or being acknowledged)

I'm feeling wanting and finding myself simplifying my life, doing things faster, and focusing even more on AI. (I still care and support cryonics and cause areas adjacent to AI like genetic engineering.)

In a few years, I might live in a constant state of thinking I could drop dead at any time from an AGI.

I used to think the most likely cause of my death would be an insufficiently good cryopreservation, but now I think it's misaligned AGI. It seems likely to me that most people alive today will die from an AI omnicide.

topic: genetic engineering

'Revolutionary': Scientists create mice with two fathers

(I just read the title)

topic: genetic engineering

'Revolutionary': Scientists create mice with two fathers

(I just read the title)

Idea for a line of thinking: What if as a result of automation we could use the ~entire human population to control AI — any way we could meaningfully organize this large workforce towards that goal?

1. What fraction of the cards from the Irrational Game didn't replicate?
2. Is there a set of questions similar to this available online?
3. In the physical game I have, there's a link to http://give-me-a-clue.com/afterdinner (iianm) which is supposed to have 300 more trivia questions, but it doesnt work -- anyone has does?

Part-time remote assistant position

My assistant agency, Pantask, is looking to hire new remote assistants. We currently work only with effective altruist / LessWrong clients, and are looking to contract people in or adjacent to the network. If you’re interested in referring me people, I’ll give you a 100 USD finder’s fee for any assistant I contract for at least 2 weeks (I’m looking to contract a couple at the moment).

This is a part time gig / sideline. Tasks often include web searches, problem solving over the phone, and google sheet formatting. A full description of our services are here: https://bit.ly/PantaskServices

The form to apply is here: https://airtable.com/shrdBJAP1M6K3R8IG It pays 20 usd/h.

You can ask questions here, in PM, or at mati@pantask.com.

a thought for a prediction aggregator

Problem with prediction market: People with the knowledge might still not want to risk money (plus complexity of markets, risk the market fails, tax implication, etc.).

But if you fully subsidize them, and make it free to participate, but still with potential for reward, then most people would probably make random predictions (at least, for predictions where most people don't have specialized knowledge about this) (because it's not worth investing the time to improve the prediction).

Maybe the best of both worlds is to do the latter, but only with people you already identified as experts. Because they already have the knowledge (maybe, mostly), so their marginal cost to predict accurately makes it worth it (although this assumption might be wrong; maybe the marginal cost is still high). Also, given there are less players, the expected reward is higher. They'll also have reputational stakes given its their field.

I would imagine this mechanism using a large index as its reward to avoid future reward being desired more (hopefully that would work? but on long enough time scale, it might not; old scientists would presumably have a higher time discount on average, maybe).

But if the brain is allowed to change, then the subject can eventually adapt to the torment. To

This doesn't follow. It seems very likely to me that it can allow it to change it ways it doesn't adapt to the pain, both from first principles and from observations.

How different would each loop need to be in order to be experienced separately?

That would be like having multiple different Everett branches experiencing suffering (parallel lives), which is different from 1 long continuous life.

Superintelligent Devil will take its victim brain's upgrade under its control and will invest in constant development of the victim's brain-parts which can feel pain. There is eternity to evolve in that direction and the victim will know that every next second will be worse. But it is really computationally intense way of punishment. 

The question of the minimal unit of experience, which is enough to break the loop sameness is interesting. The need not only to be subjectively different, but the difference need to be meaningful. Not just one pixel.

Being immortal means you will one day be a Jupiter brain (if you think memories are part of one's identity, which I think they are)

x-post: https://twitter.com/matiroy9/status/1451816147909808131

Here's a way to measure (a proxy of) relative value of different years I just thought (again?); answer the question:

For which income I would you prefer to live in perpetual-2020 over living in perpetual-2021 with a median income? (maybe income is measured by fraction of the world owned multiplied by population size, or some other way) Then you can either chain those answers to go back to older years, or just compare them directly.

There are probably years where even Owning Everything wouldn't be enough. I prefer to live in perpetual-2021 with a median income over, say, living in perpetual-1900 while Owning Everything. And probably also perpetual-1900 over some other year. Maybe this repeats 1-5 times total in human history. And even over owning everything an infinite amount of times.

So in that sense, we've already experienced a few singularities. Just meant as a fun thought!

Here's another way to measure (a proxy of) relative value of different years which makes us look much advance (at least for my preferences).

For how many years would you be indifferent between living in perpetual-1900 over living 100 years in perpetual-2020? Here, my answer is much less; assuming finite lives have value in the first place, my answer would probably be less than 2x (so somewhere between 101 and 200 years old) (although I've never tried living in 1900).

Although from that perspective, I'll never experience a singularity: I will never prefer a finite life over an infinite one, no matter how good the finite one and how bad the infinite one.

x-post: https://www.facebook.com/mati.roy.09/posts/10159848043774579

Hobby: serve so many bullets to sophisticated philosophers that they're missing half their teeth by the end of the discussion

crazy idea i just had mayyybe a deontological Libertarian* AI with (otherwise) any utility function is not that bad (?) maybe that should be one of the thing we try (??????) *where negative externality also count as aggressions, and other such fixes to naive libertarianism

Am thinking of organizing a one hour livestreamed Q&A about how to sign up for cryonics on January 12th (Bedford's day). Would anyone be interested in asking me questions?

x-post: https://www.facebook.com/mati.roy.09/posts/10159154233029579

We sometimes encode the territory on context-dependent maps. To take a classic example:

• when thinking about daily experience, stars and the Sun are stored as different things
• when thinking in terms of astrophysics, they are part of the same category This makes it so that when you ask a question like "What is the closest star [to us]?", in my experience people are likely to say Alpha Centauri, and not the Sun. Merging those 2 maps feels enlightening in some ways; creates new connections / a new perspective. "Our Sun is just a star; stars are just like the Sun." leading to system-1 insights like "Wow, so much energy to harvest out there!" Questions:

1. Is there a name for such merging? If no, should there be? Any suggestions?
2. Do you have other (interesting?) examples of map merging? x-post: https://www.facebook.com/mati.roy.09/posts/10159142062619579

In my mind, "the expert problem" means the problem of being able to recognize experts without being one, but I don't know where this idea comes from as the results from a Google search don't mention this. What name is used to refer to that problem (in the literature)?

x-post: https://www.facebook.com/mati.roy.09/posts/10159081618379579

In my mind, "the expert problem" means the problem of being able to recognize experts without being one, but I don't know where this idea comes from as the results from a Google search don't mention this. What name is used to refer to that problem (in the literature)?

x-post: https://www.facebook.com/mati.roy.09/posts/10159081618379579

In my mind, "the expert problem" means the problem of being able to recognize experts without being one, but I don't know where this idea comes from as the results from a Google search don't mention this. What name is used to refer to that problem (in the literature)?

x-post: https://www.facebook.com/mati.roy.09/posts/10159081618379579

suggestion of something to try at a LessWrong online Meetup:

video chat with a time-budget for each participant. each time a participant unmutes themselves, their time-budget starts decreasing.

note: on jitsi you can see how many minutes someone talked (h/t Nicolas Lacombe)

x-post: https://www.facebook.com/mati.roy.09/posts/10159062919234579

imagine having a physical window that allowed you to look directly in the past (but people in the past wouldn't see you / the window). that would be amazing, right? well, that's what videos are. with the window it feels like it's happening now, whereas with videos it feels like it's happening in the past, but it's the same

x-post: https://www.facebook.com/mati.roy.09/posts/10158977624499579

tattoo idea: I won't die in this body

in Toki Pona: ale pini mi li ala insa e sijelo ni

direct translation: life's end (that is) mine (will) not (be) inside body this

EtA: actually I got the Toki Pona wrong; see: https://www.reddit.com/r/tokipona/comments/iyv2r2/correction_thread_can_your_sentences_reviewed_by/

When you're sufficiently curious, everything feels like a rabbit hole.

Challenge me by saying a very banal statement ^_^

x-post: https://www.facebook.com/mati.roy.09/posts/10158883322499579

I can pretty much only think of good reasons for having generally pro-entrapment laws. Not any kind of traps, but some kind of traps seem robustly good. Ex.: I'd put traps for situations that are likely to happen in real life, and that show unambiguous criminal intent.

It seems like a cheap and effective way to deter crimes and identify people at risk of criminal behaviors.

I've only thought about this for a bit though, so maybe I'm missing something.

x-post with Facebook: https://www.facebook.com/mati.roy.09/posts/10158763751484579

People say we can't bet about the apocalypse. But what about taking debt? The person thinking the probability of apocalypse is higher would accept higher interest rate on their debt, as once at the judgement period their might be no one to whom the money is worth or the money itself might not be worth much.

I guess there are also reasons to want more money during a global catastrophe, and there are also reasons to not want to keep money for great futures (see: https://matiroy.com/writings/Consume-now-or-later.html), so that wouldn't actually work.

meta - LessWrong have people predict whether they will upvote a post just based on the title

There's a post, I think by Robin Hanson on Overcoming Bias, that says people care about what their peers think of them, but we can hack our brains to doing awesome things by making this reference group the elite of the future. I can't find this post. Do you have a link?

Personal Wiki

might be useful for people to have personal wiki where they take note instead of everyone taking notes in private Gdoc

status: to do / to integrate

you know those lists about historical examples of notable people mistakenly saying that some tech will not be useful (for example)

Elon Musk saying that VR is just a TV on your nose will probably become one of those ^^

https://youtube.com/shorts/wYeGVStouqw?feature=share

idea: Stream all of humanity's information through the cosmos in hope an alien civ reconstruct us (and defends us against an Earth-originating maligned ASI)

I guess finding intelligent ETs would help with that as we could stream in a specific direction instead of having to broadcast the signal broadly

It could be that maligned alien ASIs would mostly ignore our information (or at least not use it to like torture us) whereas friendly align ASI would use it beneficially 🤷‍♀️

Topics: cause prioritization; metaphor

note I took on 2022-08-01; I don't remember what I had in mind, but I feel like it can apply to various things

from an utilitarian point of view though, i think this is almost like arguing whether dying with a red or blue shirt is better; while there might be an answer, i think it's missing the point, and we should focus on reducing risks of astronomical disasters

An interesting perspective.

It is instructive to consider the following four scenarios:

1. The Kolmogorov complexity of the state of your mind after N timesteps in a simulation with a featureless white plane.
2. The Kolmogorov complexity of the state of your mind after N timesteps in a simulation where you are in a featureless plane, but the simulation injects a single randomly-chosen 8x8 black-and-white bitmap into the corner of your visual field. (256 bits total.)
3. The Kolmogorov complexity of the state of your mind after N timesteps in a simulation with "random" noise that's actually the output of, say, a CSPRNG with 256b of initial state.
4. The Kolmogorov complexity of the state of your mind after N timesteps in a simulation with "truly" random noise.

If crypto makes the USD go to 0, will life insurances denominated in USD not have anything to pay out? Maybe an extra reason for cryonicists to own some crypto

x-post: https://www.facebook.com/mati.roy.09/posts/10159482104234579

A Hubble Brain: a brain taking all the resources present in a Hubble-Bubble-equivalent.

related: https://en.wikipedia.org/wiki/Matrioshka_brain#Jupiter_brain

x-post: https://www.facebook.com/mati.roy.09/posts/10158917624064579

I want to look into roleplay in animals, but Google is giving me animal roleplay, which is interesting too, but not what I'm looking for right now 😅

I'm wonder how much roleplay there is in the animal kingdom. I wouldn't be surprised if there was very few.

Maybe if you're able to roleplay, then you're able to communicate?? Like, roleplay might need to have a theory of mind, because you're imagining yourself in someone else's body.

Maybe you can teach words to an animal without a theory of mind, but they'll be more like levers for them: for them, saying "banana" is like pressing a lever that gives you bananas rather than communicating the concept of banana to someone else.

And once you have a theory of mind, then this might unlock the ability to roleplay. If you have a culture of roleplay, then I guess that basically already gives you a basic "sign" language from which you gradually build abstractions as you climb the intelligence ladder. I guess, at first the cultural and genetic selection might be (a lot?) related to the size of the vocabulary, and we can model roleplaying as a (proto-)language.

Although a quick web search tells me that children develop language before a theory of mind; but maybe they wouldn't come up with it even if they can learn it. And maybe "theory of mind" is more of a spectrum.

This is an idea the bloomed from talking to David Krueger about the progress of humans' learning abilities, and zir pointing out that before language you might have roleplay.

x-post: https://www.facebook.com/mati.roy.09/posts/10158914397244579

I remember someone in the LessWrong community (I think Eliezer Yudkowsky, but maybe Robin Hanson or someone else, or maybe only Rationalist-adjacent; maybe an article or a podcast) saying that people believing in "UFOs" (or people believing in unproven theories of conspiracy) would stop being so enthusiastic about those if they became actually known as true with good evidence for them. does anyone know what I'm referring to?

sometimes I see people say "(doesn't) believe in science" when in fact they should say "(doesn't) believe in scientists"

or actually actually "relative credence in the institutions trying to science"

x-post: https://www.facebook.com/mati.roy.09/posts/10158892685484579

hummm, I think I prefer the expression 'skinsuit' to 'meatbag'. feels more accurate, but am not sure. what do you think?

x-post: https://www.facebook.com/mati.roy.09/posts/10158892521794579

I just realized my System 1 was probably anticipating our ascension to the stars to start in something like 75-500 years.

But actually, colonizing the stars could be millions of subjective years away if we go through an em phase (http://ageofem.com/). On the other hand, we could also have finished spreading across the cosmos in only a few subjective decades if I get cryopreserved and the aestivation hypothesis is true (https://en.wikipedia.org/wiki/Aestivation_hypothesis).

I created a Facebook group to discuss moral philosophies that value life in and of itself: https://www.facebook.com/groups/1775473172622222/

How to calculate subjective years of life?

• If the brain is uniformly sped up (or slowed down), I would count this as proportionally more (or less)
  • Biostasis would be a complete slow down, so wouldn't count at all
• I would not count unconscious sleeping or coma
• I would only count dreaming if some of it is remembered ([https://www.lesswrong.com/posts/F8iTtzSgxRmmcCrWE/for-what-x-would-you-be-indifferent-between-living-x-days?commentId=FbfeG2mZPew3enNPF [LW(p) · GW(p)]](more on this))

For non-human animal brains, I would compare them to the baseline of individuals in their own species.

For transhumans that had their mind expanded, I don't think there's an obvious way to get an equivalence. What would be a subjective year for a Jupiter brain?

Maybe it could be in terms of information processed, but in that case, a Jupiter brain would be living A LOT of subjective time per objective time.

Ultimately, given I don't have "intrinsic" diminishing returns on additional experience, the natural definition for me would the amount of 'thinking' that is as valuable. So a subjective year for my future Jupiter brain would be the duration for which I find that experience as valuable as a subjective year now.

Maybe that could even account for diminishing value of experience at a specific mind size because events would start looking more and more similar?? But it otherwise wouldn't work for people that have "intrinsic" diminishing returns on additional experience. It would notably not work with people for whom marginal experiences start becoming undesirable at some point.

The original Turing test has a human evaluator.

Other evaluators I think would be interesting include: the AI passing the test, a superintelligent AI, and an omniscient maximally-intelligent entity (except without the answer to the test).

Thought while reading this thread.

Blocking one ear canal

Category: Weird life optimization

One of my ear canal is in a different shape. When I was young, my mother would tell me that this one was harder to clean, and that ze couldn't see my ear-drum. This ear gets wax accumulation more easily. A few months ago, I decided to let it block.

Obvious possible cognitive bias is the "just world bias": if something bad happens often enough, I'll start think it's good.

But here are benefits this has for me:

1. When sleeping, I can put my good ear on the pillow, and this now isolates me from sound pretty well. And isn't uncomfortable unlike alternatives (ie.: earmuffs, second pillow, my arm).

2. I'm only using one ear, so that if I become hard-of-hearing when older (a common problem I think), then I can remove the wax from my backup hear.

This even makes me wonder whether having one of your ear canal differently shaped is something that's actually been selected for.

Other baffling / really surprising observation: My other hear is also blocked on most morning, but just touching it a bit unblocks it.

topic: fundamental physics

x-post from YouTube

comments on Why No One Has Measured The Speed Of Light

2 thoughts

1. maybe that means you could run a simulation of the universe without specifying c (just like you don't have to specify "up/down"); maybe saying the speed of light is twice as big in a direction than the other is like saying everything is twice as big as we thought: it's a meaningless statement because they are defined relative to each other

2. if universe looks the same in all directions, yet one side of the universe we see as it currently is whereas the other side we see as it was billion of years ago, then it seems reasonable that we would expect both sides to look more different than they actually do. or maybe, it would mean there's a big bang wave rapidly propagating in the direction where we're receiving the instantaneous light, which would explain why we see cosmic background radiation from that direction as well