Posts

Review: Dr Stone 2024-09-29T10:35:53.175Z
Looking for intuitions to extend bargaining notions 2024-08-24T05:00:13.995Z
Moving away from physical continuity 2024-07-12T05:05:01.231Z
Inner Optimization Mechanisms in Neural Nets 2024-05-12T17:52:52.803Z
User-inclination-guessing algorithms: registering a goal 2024-03-20T15:55:01.314Z
ProgramCrafter's Shortform 2023-07-21T05:26:03.188Z
LLM misalignment can probably be found without manual prompt engineering 2023-07-08T14:35:44.119Z
Does object permanence of simulacrum affect LLMs' reasoning? 2023-04-19T16:28:22.094Z
The frozen neutrality 2023-04-01T12:58:40.873Z
Proposal on AI evaluation: false-proving 2023-03-31T12:12:15.636Z
How AI could workaround goals if rated by people 2023-03-19T15:51:04.743Z

Comments

Comment by ProgramCrafter (programcrafter) on Open Thread Fall 2024 · 2024-12-19T17:17:06.235Z · LW · GW

Does your family have the same opinions as your social circle?

Quite similar, in fact - at least where they care to do so! I do listen for perspective, but I still can't put society's revealed opinion into span of those who I know better!

Getting to know your neighbors is another way to expose yourself to people who often think differently.

A good idea! I'll have to take it a bit more general, because I'm an university student with dormitory and already know many people around; though, eating at some local cafe with diverse customers should work.

Being an Uber driver would be too taxing on my time, but I'm sure there is another idea instantiation which would work!

 

(And while we're in an Open Thread, I'd like to thank LessWrong for featuring that one can make informed decisions on pretty much any topic! I've chosen uni aligned with me and I'm not disappointed with it so far.)

Comment by ProgramCrafter (programcrafter) on Review: Dr Stone · 2024-12-18T20:18:38.788Z · LW · GW

Tsukasa just plain lost and the logical ending is for him to be stoned or killed

except that
1) there is sometimes too much of such hostility in the real world,
2) some people can legitimately be redeemed - that is, they change thinking strategies and approximations to their values when they see what has been created (re: Einstein's reaction to atomic bomb creation),
3) I don't think anyone depicted in the anime would have valued fairness/consistency/other base for punishing Tsukasa - and with death penalty, no less - over compassion.

Authors might have held the position (shout-out to @the gears to ascension!) "[we] want literally every human to get to go to space often and come back to a clean and cozy world. This currently seems unlikely. Let's change that." Even if they haven't: there is such vibe, and I somewhat endorse it!

Comment by ProgramCrafter (programcrafter) on Review: Breaking Free with Dr. Stone · 2024-12-18T11:11:19.782Z · LW · GW

Nice to see more fans!)) I have also written a review with more focus on rationality demonstration within the anime: https://www.lesswrong.com/posts/yqXoFx7jsSjbWgvBq/review-dr-stone.

Comment by ProgramCrafter (programcrafter) on Open Thread Fall 2024 · 2024-12-17T20:25:14.231Z · LW · GW

How can I get an empathic handle on my region/country/world's society (average/median/some sample of its people, to be clearer)?

I seem to have got into a very specific social circle, being a constant LW reader and so on. That often makes me think "well, there is question X, a good answer is A1 and it is also shared within the circle, but many people overall will no doubt think A2 and I don't even know who or why". I can read survey/poll results but not understand why would people even want to ban something like surrogate motherhood or things like that.

I've heard one gets to know people when works with them. If so, I'd like to hear suggestions for some [temporary] professions which could aid me here?

Comment by ProgramCrafter (programcrafter) on Zombies! Substance Dualist Zombies? · 2024-12-12T19:45:06.745Z · LW · GW

We can suggest a Weak Zombie Argument: It is logically possible to have a universe where all qualia of red and green are inverted in the minds of its inhabitants, while all physical things remain the same.

I'd say that universe you describe IS one we're living in (note I'm not using "equal to" or "same as", I'm rejecting that those are two different worlds). It doesn't really matter for anything happening in the universe whether strawberry color is labelled "red", "black", "colF00" or "qualia_fa615f8f", or if that particular quale might never happen, as long as causal network is isomorphic to the original one.

Comment by ProgramCrafter (programcrafter) on Understanding Shapley Values with Venn Diagrams · 2024-12-12T00:15:14.744Z · LW · GW

Shapley values are the ONLY way to guarantee: <Efficiency, Symmetry, Linearity, Null player properties>

Well it doesn't end at that: it turns out Shapley values for more than 2 players are not nicely behaved and instead violate Maximin Dominance, as demonstrated in https://www.lesswrong.com/posts/vJ7ggyjuP4u2yHNcP/threat-resistant-bargaining-megapost-introducing-the-rose#ROSE_Value__N_Player_Case__.

The article I link showed how this is fixed:

Shapley values are about adding everyone one-by-one to a team in a random order and everyone gets their marginal value they contributed to the team.

And that's kinda like giving everyone a random initiative ordering and giving everyone the surplus they can extract in the resulting initiative game.

If we're doing that, then maybe a player, regardless of their position, can ensure they get their maximin value? Maybe this sort of Random-Order Surplus Extraction can work. ROSE.

Comment by ProgramCrafter (programcrafter) on 2024 Unofficial LessWrong Census/Survey · 2024-12-10T11:28:14.134Z · LW · GW

I've completed the survey, skipping just a few questions I thought too ambiguous!

(Note: I've waited a bit to intentionally mask submit time, since otherwise I'd be identifiable by brag order :-) )

Comment by ProgramCrafter (programcrafter) on Expevolu, a laissez-faire approach to country creation · 2024-12-05T23:06:10.695Z · LW · GW

I think CBs won't work as described at least because they need to be allocated to children/new citizens somehow? Given that territory of adopting country is non-increasing while its population increases over time, the state would need to buy out part of CB and grant those pieces, and that would presumably create CB borders drift.

Overall, I downvoted this post because I felt it only describes "that's the idea of expevolu, and its benefits", not realistically exploring what would happen upon adoption, and what drawbacks are inherited from previously suggested ideas.

Comment by ProgramCrafter (programcrafter) on Sorry for the downtime, looks like we got DDosd · 2024-12-02T15:19:53.155Z · LW · GW

It's a solution! However it comes with its own downsides. For instance, Codeforces users ranted on Cloudflare usage for a while, with following things (mapped to LessWrong) highlighted:

  • The purpose of an API is defeated: even the API endpoints on the same domain are restricted, which prevents users from requesting posts via GraphQL. In particular, ReviewBot will be down (or be hosted in LW internal infrastructure).
  • In China, Cloudflare is a big speed bump.
  • Cloudflare-protected sites are reported to randomly lag a lot.
    > I had been assuming that this is a server problem, but from talking to some people it seems like this is an issue with differential treatment of who is accessing CF.
    Lack of interaction smoothness might be really noticeable for new users, comparing to current state.
Comment by ProgramCrafter (programcrafter) on Passages I Highlighted in The Letters of J.R.R.Tolkien · 2024-11-25T23:24:25.071Z · LW · GW

Thank you for highlighting philosophical position!

I take it that Tolkien [early on] thought human thinking is often malign to their desires and corrupts the latter when given enough power; and power without cognition improvement often comes with machines and other tools for doing something new; so people should be truer to their desires and prefer to use things without much relying on them.

This position is clearly in contrast with uplifting, which suggests giving everyone more power to lead fulfilling lives - as much power as possible - and backing the supply with full force of humanity. I would actually try uplifting, however little the stable solution zone would be, because I feel that to be cool!

Comment by ProgramCrafter (programcrafter) on Two flavors of computational functionalism · 2024-11-25T23:01:29.782Z · LW · GW

Well it probably is computing insofar it's the wind bringing in actual bits of information, not you while searching for a specific pattern instantiation. The test: consider whether if original grains were moved around to form another prime number, would the wind still process them a similar way and yield correct answer?

Comment by ProgramCrafter (programcrafter) on Eli's shortform feed · 2024-11-12T03:43:59.924Z · LW · GW

I've started writing a small research paper on this, using mathematical framework, and understood that I had long conflated Shapley values with ROSE values. Here's what I found, having corrected that error.

ROSE bargaining satisfies Efficiency, Pareto Optimality, Symmetry*, Maximin Dominance and Linearity - a bunch of important desiderata. Shapley values, on other hand, don't satisfy Maximin Dominance so someone might unilaterally reject cooperation; I'll explore ROSE equilibrium below.

  1. Subjects: people and services for finding partners.
  2. By Proposition 8.2, ROSE value remains same if moves transferring money within game are discarded. Thus, we can assume no money transfers.
  3. By Proposition 11.3, ROSE value for dating service is equal or greater than its maximin.
  4. By Proposition 12.2, ROSE value for dating service is equal or less than its maximum attainable value.
  5. There's generally one move for a person to maximize their utility: use the dating service with highest probability of success (or expected relationship quality) available.
  6. There are generally two moves for a service: to launch or not to launch. First involves some intrinsic motivation and feeling of goodness minus running costs, the second option has value of zero exactly.
  7. For a large service, running costs (including moderation) exceed much realistic motivation. Therefore, maximum and maximin values for it are both zero.
  8. From (7), (3) and (4), ROSE value for large dating service is zero.
  9. Therefore, total money transfers to a large dating service equal its total costs.

So, why yes or why no?


By the way, Shapley values suggest paying a significant sum! Given value of a relationship of $10K (can be scaled), and four options for finding partners ( -- self-search,  -- friend's help,  -- dating sites,  -- the specialized project suggested up the comments), the Shapley-fair price per success would be respectively $550, $650 and $4400.

P.S. I'm explicitly not open to discussing what price I'd be cheerful to pay to service which would help to build relationships. In this thread, I'm more interested in whether there are new decision theory developments which would find maximin-satisfying equilibria closer to Shapley one.

Comment by ProgramCrafter (programcrafter) on Eli's shortform feed · 2024-11-11T00:46:38.546Z · LW · GW

at a $100k valuation of a marriage

I don't think one can coherently value marriage 20 times as much as than a saved life ($5k as GiveWell says)? Indeed there is more emotional attachment to a person who's your partner (i.e. who you are emotionally attached to) than to a random human in the world, but surely not that much?

And if a marriage is valued at $10k, then the credit assignment 1%/10% would make the allocation $100/$1000 - and it seems that people really want to round the former towards zero

Comment by ProgramCrafter (programcrafter) on Open Thread Fall 2024 · 2024-11-09T22:32:48.751Z · LW · GW

When rereading [0 and 1 Are Not Probabilities], I thought: can we ever specify our amount of information in infinite domains, perhaps with something resembling hyperreals?

  1. An uniformly random rational number from  is taken. There's an infinite number of options meaning that prior probabilities are all zero (), so we need infinite amount of evidence to single out any number.
    (It's worth noting that we have codes that can encode any specific rational number with a finite word - for instance, first apply bijection of rationals to natural numbers, then use Fibonacci coding; but in expectation we need to receive infinite bits to know an arbitrary number).

    Since  symbol doesn't have nice properties with regards to addition and subtraction, we might define a symbol  which means "we need some information to single out one natural number out of their full set". Then, the uniform prior over  would have form  (prefix and suffix standing for values outside  segment) while a communication "the number is " would carry  bits of evidence on average, making the posterior .
  2. The previous approach suffers from a problem, though. What if two uniformly random rationals  are taken, forming a square on coordinate grid?
    If we've been communicated  information about , we clearly have learned nothing about  and thus cannot pinpoint the specific point, requiring  more bits.

    However, there's bijection between  and , so we can assign a unique natural number to any point in the square, and therefore can communicate it in  bits in expectation, without any coefficient .

When I tried exploring some more, I've validated that greater uncertainty (, communication of one real number) makes smaller ones () negligible, and that evidence for a natural number can presumably be squeezed into communication for a real value. That also makes the direction look unpromising.

 

However, there can be a continuation still: are there books/articles on how information is quantified given a distribution function?

Comment by ProgramCrafter (programcrafter) on quila's Shortform · 2024-11-08T20:17:25.726Z · LW · GW

Never say 'nothing' :-)

  1. the world might be in such state that attempts to do good bring it into some failure instead, and doing the opposite is prevented by society
    (AI rise and blame-credit which rationality movement takes for it, perhaps?)
  2. what if, for some numerical scale, the world would give you option "with 50%, double goodness score; otherwise, lose almost everything"? Maximizing EV on this is very dangerous...
Comment by ProgramCrafter (programcrafter) on 2024 Unofficial LW Community Census, Request for Comments · 2024-11-02T18:08:05.295Z · LW · GW

I guess I'm looking for questions of this family:

  1. Do you sometimes tell things that are not literally true but help the person you're talking to in understanding?
  2. On average, do you believe statements by members of rationalist community significantly more (+1.0 bit of evidence or more) than exact same words from non-rationalists?
  3. What is the biggest barrier you face when trying to communicate rational ideas to others? [a) Emotional resistance b) Lack of shared vocabulary c) Time constraints d) Preexisting strong beliefs e) Complexity of ideas f) People disengaging randomly]

Also,

  • Have you ever intervened on someone's behalf where the person was failing and would prefer to succeed?
  • How many people can a [brainstorming] conversation hold on average, so that everyone is active?
Comment by ProgramCrafter (programcrafter) on What can we learn from insecure domains? · 2024-11-02T15:33:36.998Z · LW · GW

I feel like an important question is: how far does this generalize? We can estimate the IQ gap between the dumbest person who successfully uses the internet (probably in the 80's) and the smartest malware author (got to be at least 150+).  Is that the limit somehow, or does this knack extend across even more orders of magnitude?

If imagine a world where 100 IQ humans are using an internet that contains malware written by 1000 IQ AGI, do humans just "avoid the bad parts"?

For reactive threats, the upper bound is probably at most "people capable of introspection who can detect they are not sure some action will be to net benefit, and therefore refuse to take it". For active threatening factors, that's an arms race (>=40% this race is not to infinity - basically, if more-cooperating DT strategies are any good).

Maybe the subject is researched more in biology? Example topic: eating unknown food (berries, nuts) in forest, and balance of lifetime adaptation vs evolutionary adaptation (which involves generations passing).

Comment by ProgramCrafter (programcrafter) on 2024 Unofficial LW Community Census, Request for Comments · 2024-11-02T00:18:25.705Z · LW · GW

It would be nice to see at least three questions which would demonstrate how person extracts evidence from others' words, how much time and emotions could they spend if they needed to communicate a point precisely, etc.

I'll have to sleep on that, actually. Will return tomorrow, presumably with more concrete ideas)

Comment by ProgramCrafter (programcrafter) on What can we learn from insecure domains? · 2024-11-02T00:13:25.711Z · LW · GW

99.9% of all cryptocurrency projects are complete scams (conservative estimate).

On first skim, I agree with the estimate as stated and would post a limit order for either side. I'd also like to note that "crypto in general is terrible" instead of "all crypto is terrible", as there have been applications developed that do not allow you to lose all funds without explicit acknowledgement.

Similarly, Cyber Security is terrible.  Basically every computer on the internet is infected with multiple types of malware.

It is presumably terrible (or, 30%, result of availability bias), and I've observed bugs happen because functionality upgrade did not consider its interaction with all other code. However, I disagree that every computer is infected; probably you meant that it is under constant stream of attack attempts?

 

The insecure domains mainly work because people have charted known paths, and shown that if you follow those paths your loss probability is non-null but small. As a matter of IT, it would be really nice to have systems which don't logically fail at all, but that requires good education and pressure-resistance skills for software developers.

Comment by ProgramCrafter (programcrafter) on What TMS is like · 2024-11-01T23:58:53.807Z · LW · GW

I think TMS doesn't rewrite anything, instead activating neural circuits in another pattern? Then, new pattern is not depressed, brain can notice that (on either conscious or subconscious level) and make appropriate changes to neural connections.

Basically, I believe that whatever resulting patterns (including "other parts of you changed into something non-native and alien") you dis-endorse, are "committed" with significantly lower probability.

Comment by ProgramCrafter (programcrafter) on 2024 Unofficial LW Community Census, Request for Comments · 2024-11-01T23:51:54.822Z · LW · GW

P(Bitcoin) What is the probability that the price of one bitcoin will be higher on July 1st, 2025, than it was on December 1st, 2024? ($???)

Probably best to include "what price of one bitcoin do you expect on July 1st, 2025, given that it was $??? on December 1st, 2024?" as well.
You could also include P(weak EMH) - instead of P(GPT-5 Release) if there's not enough space.

Overall, the questions seemed insufficiently checking social skills to me, instead preferring testing large, "impactful" beliefs.

Comment by ProgramCrafter (programcrafter) on What TMS is like · 2024-10-31T19:59:52.738Z · LW · GW

How do they figure out what waveforms to use and at what frequencies?

Presumably Fourier transform or its developments can be used. (After all, light speed is negligible at the scale we're considering.)

Comment by ProgramCrafter (programcrafter) on Review: “The Case Against Reality” · 2024-10-29T20:22:46.328Z · LW · GW

In the underlying computer which executes the actions which are represented by this interface, there is nothing that resembles a pointer

Course' there are: probably 64 bits in memory (for more degree of detail: 64 places of persistent electric charge with two stable states), which change iff pointer moves, and each bit restricts the places pointer can appear at. That resemblance exists certainly; I also agree there's no resemblance like "small pointer-like thing/charge pattern in RAM module".

In other words, one has to taboo "resemblance" but it's not clear if that can be done.

Comment by ProgramCrafter (programcrafter) on What are some good ways to form opinions on controversial subjects in the current and upcoming era? · 2024-10-29T19:52:05.824Z · LW · GW

I interpreted your algorithm for listing computables to be something like "enumerate the Turing machines that output '.' then 0s and 1s and list what they print", without worrying about the fact that some computables repeat.

I'm pretty sure my argument did not mention how computables are listed at all, rather proving that for any specific listing the inverse-diagonal is computable as well.

If you have any surjection: N→S⊂(N→{0,1}) and diagonalize against it, you know the result is not in S. This fact doesn't depend on the actual nature of the surjection N→S, just that S is the image. Here S is the computables.

Yes. However, it's the specific choice of set "computables" which creates the contradiction: I agree with "inverse-diagonal for rationals is an irrational number" and like.

Once again: for any "user-provided" computable table of computable digit sequences, I can, in finite time, get value for any specific position in table; therefore, each digit of inverse sequence is computable; therefore, I conclude that the inverse-diagonal sequence is itself computable (if I'm not mistaken in definitions).

Comment by ProgramCrafter (programcrafter) on What are some good ways to form opinions on controversial subjects in the current and upcoming era? · 2024-10-29T16:37:43.572Z · LW · GW
  1. Retrieve I-th computable from table: computable by assumption.
  2. Get K-th digit from a given computable: computable by definition.
  3. Get I-th digit of I-th computable: computable as composition of (1) and (2).
  4. Invert given digit: trivially computable.
  5. Get an inverted I-th digit of I-th computable: computable as composition of (3) and (4).

 

I've actually written formal refutation of any bijection between  and  in Idris 2.

Demonstration

total stands for functions which are defined everywhere on their stated domain (as opposed to partial), and computations for which are proven to always halt (if there's no such guarantee, function is covering). %default total means that all functions below must be total.

Void type means that a contradiction was obtained from given arguments.

%default total


data Bijection : Type -> Type -> Type where
  ||| A bijection between two types.
  ||| @ a the first type
  ||| @ b the second type
  ||| @ fwd mapping from first type to the second
  ||| @ bck mapping from second type to the first
  ||| @ fb_producer proof that fwd . bck = id
  ||| @ bf_producer proof that bck . fwd = id
  DoMap : {0 a,b : Type} -> (fwd: a -> b) -> (bck: b -> a)
          -> (fb_producer: (av : a) -> (bck $ fwd av) = av)
          -> (bf_producer: (bv : b) -> (fwd $ bck bv) = bv)
          -> Bijection a b

map_fneq : {0 a,b : Type} -> {f : a->b} -> {g : a->b} -> f = g -> (v:a) -> f v = g v
map_fneq (Refl {x = f}) v = Refl

bool_inv : {v : Bool} -> (v = not v) -> Void
bool_inv {v = True}  prf = uninhabited prf
bool_inv {v = False} prf = uninhabited prf

diagonal : (Bijection Nat (Nat -> Bool)) -> Void
diagonal (DoMap f g _ bf_prf) = bool_inv conflict where
  H : Nat -> Bool
  H i = not $ f i i
  
  h_hi_by_idx : f (g H) (g H) = H (g H)
  h_hi_by_idx = map_fneq (bf_prf H) (g H)
  
  h_hi_by_def : H (g H) = (not $ f (g H) (g H))
  h_hi_by_def = Refl
  
  conflict : f (g H) (g H) = (not $ f (g H) (g H))
  conflict = trans h_hi_by_idx h_hi_by_def
Comment by ProgramCrafter (programcrafter) on What are some good ways to form opinions on controversial subjects in the current and upcoming era? · 2024-10-29T11:54:18.166Z · LW · GW

If you put the computable numbers in the table, the inverted diagonal will be an uncomputable number.

In fact,
diag-a1. if you put the computable numbers in the table, and
diag-a2. pretend that the assignment is computable also, and
diag-a3. (optionally) pretend that procedure for looking up the row of specific number is computable also,

then
diag-l1. the inverted diagonal is computable by explicit construction,
diag-l2. the inverted diagonal does not belong to the table,

thus
diag-t1. we have the contradiction (based on diag-l1 and diag-l2),
diag-t2. (if diag-a3 holds) the contradicting place is computable as well.

I agree with your view but that specific example was wrong!

Comment by ProgramCrafter (programcrafter) on A Semiotic Critique of the Orthogonality Thesis · 2024-10-27T14:46:08.459Z · LW · GW

A goal is, fundamentally, an idea. As the final step in a plan, you can write it out as a symbolic representation of the “world state” you are trying to achieve, although it could represent other things as well. In a planning computer agent, this will probably terminate in a bunch of 1s and 0s stored in its memory. 

In order for this symbolic representation to be meaningful, it must be comparable and distinct from other symbolic representations. World state A in the agent's plan could be contrasted from world state B, C and D. This is a very fundamental fact about how information and meaning work, if World State A was indistinguishable from all the others, there would be no reason for the agent to act, because its goal would have been “accomplished”.

This has a logic error. There need not be one best world state, and a world state need not be distinguishable from all others - merely from some of them. (In fact, utility function yielding a real value compresses the world into a characteristic of things we care about in such a way.)

Also, with unbounded computations, utility optimizer could tell supremum (best outcome) for any set of world states you'd provide it; without that, it will have less granularity, work on set of close states (for instance, "easily coming to human mind") or employ other optimization techniques.

I believe this underlies much of the disagreement, because then more knowledge or more intelligence might change only the relations of "final goal" sign but not its meaning (re: isomorphism).


Your series of posts also assume that signs have a fixed order. This is false. For instance, different fields of mathematics treat real number as either first order signs (atomic objects) or higher-order ones, defined as relations on rational numbers.

Or, for an easier example to work on: equality could be a second-order sign "object A is same as object B", or it may be defined using third order expression "for any property P, A and B either both have the property or both not have it". It is no coincidence that those definitions are identical; you cannot assume that if something is expressible using higher order signs, is not also expressible in lower order.

And this might undermine the rest of argument.


Engaging with the perspective of orthogonality thesis itself: rejecting it means that a change in intelligence will lead, in expectation, to change in final goals. Could you name the expected direction of such a change, like "more intelligent agents will act with less kindness"?

Comment by ProgramCrafter (programcrafter) on Is the Power Grid Sustainable? · 2024-10-27T14:10:04.772Z · LW · GW

Before power grid dissolves, it has also to hit factories (and business in general). I don't think resulting price increments are predictable - they might as well start some crisis in economy. (And there might be money-unrelated outcomes, like if restaurants start ignoring some safety standards trying to save on electricity or like, which could be disastrous with habit of eating out.)

Comment by ProgramCrafter (programcrafter) on Change My Mind: Thirders in "Sleeping Beauty" are Just Doing Epistemology Wrong · 2024-10-23T19:25:12.733Z · LW · GW

She certainly gets a reward for following experimental protocol, but beyond that... I concur there's the problem, and I have the same issue with standard formulation asking for probability.

In particular, pushing problem out to morality "what should Sleeping Beauty answer so that she doesn't feel as if she's lying" doesn't solve anything either; rather, it feels like asking question "is continuum hypothesis true?" providing only options 'true' and 'false', while it's actually independent of ZFC axioms (claims of it or of its negation produce different models, neither proven to self-contradict).

P.S. One more analogue: there's a field, and some people (experimenters) are asking whether it rained recently with clear intent to walk through if it didn't; you know it didn't rain but there are mines all over the field.
I argue you should mention the mines first ("that probability - which by the way will be 1/2 - can be found out, conforms to epistemology, but isn't directly usable anywhere") before saying if there was rain.

Comment by ProgramCrafter (programcrafter) on Change My Mind: Thirders in "Sleeping Beauty" are Just Doing Epistemology Wrong · 2024-10-22T17:32:43.246Z · LW · GW

What exactly do you mean by "different tools need to be used"? Can you give me an example?

I mean that Beauty should maintain full model of experiment, and use decision theory as well as probability theory (if latter is even useful, which it admittedly seems to be). If she didn't keep track of full setup but only "a fair coin was flipped, so the odds are 1:1", she would predictably lose when betting on the coin outcome.

 

Also, I've minted another "paradox" version. I can predict you'll take issue with one of formulations in it, but what do you think about it?

A fair coin is flipped, hidden from you.

On Heads, you're waken up on Monday, asked "what credence do you have that coin landed Heads?"; on Tuesday, you're let go.

If coin landed Tails, you're waken up on Monday and still asked "what credence do you have that coin landed Heads?"; then, with no memory erasure, you're waken up on Tuesday, and experimenter says to you: "Name the credence that coin landed Heads, but you must name the exact same number as yesterday". Afterwards, you're let go.

If you don't follow experiment protocol, you lose/lose out on some reward.

Comment by ProgramCrafter (programcrafter) on What's a good book for a technically-minded 11-year old? · 2024-10-20T12:42:22.074Z · LW · GW

Have you tried asking the kid? Perhaps they already have perspective matching the real world on some topic.

Comment by ProgramCrafter (programcrafter) on is there a big dictionary somewhere with all your jargon and acronyms and whatnot? · 2024-10-17T15:18:26.708Z · LW · GW

There is, in the left panel under "Concepts" link. https://www.lesswrong.com/tags/all.

I'd guess you haven't seen "Rationality: A-Z" (also called "The Sequences") from that site panel as well. It builds up many of local concepts from certain starting points over course of many essays, gradually increasing level of material. For a newcomer, it is rather common to be linked to some essay there upon some marker statements (for instance, "that hypothesis is improbable, but its alternatives are no better so I keep 'my' one!")

Comment by ProgramCrafter (programcrafter) on Change My Mind: Thirders in "Sleeping Beauty" are Just Doing Epistemology Wrong · 2024-10-17T15:00:51.995Z · LW · GW

Upon rereading your posts, I retract disagreement on "mutually exclusive outcomes". Instead...

Initially probability function is defined over iterations of probability experiment. You define a different function over all space and time, which you still call "probability". It surely has properties that you like, but it's a different function! Please use another name, this is already taken. Or add a disclaimer. Preferably do both. You know how easy it is to confuse people with such things! Definetely, do not start participating in the conversations about probability while talking about your function.

An obvious way to do so is put a hazard sign on "probability" and just not use it, not putting resources into figuring out what "probability" SB should name, isn't it? For instance, suppose Sleeping Beauty claims "my credence for Tails is "; any specific objection would be based on what you expected to hear.

(And now I realize a possible point why you're arguing to keep "probability" term for such scenarios well-defined; so that people in ~anthropic settings can tell you their probability estimates and you, being observer, could update on that information.)

As for why I believe probability theory to be useful in life despite the fact that sometimes different tools need to be used: I believe disappearing as a Boltzmann brain or simulated person is balanced out by appearing the same way, dissolving into different quantum branches is balanced out by branches reassembling, and likewise for most processes.

Comment by ProgramCrafter (programcrafter) on Why Academia is Mostly Not Truth-Seeking · 2024-10-16T21:44:06.719Z · LW · GW

For the record, there is still some valid, good-faith scientific research out there, but most of the academic “research” produced in more recent decades is either fabricated, dishonest, plagiarized, redundant, outdated, and/or useless.

This claim could be restated as: most of the academic "research" is either false because authors did not intend to be honest (or conformed to unrelated biases), or false because authors did not have more accurate data (this kind of research becomes superseded over time, when more belief depth is acquired). This might be true.

However, I don't know what redundant articles do in that list; I suppose you're claiming more articles stating the same point do not provide more evidence to it, but replication and more experiments in good faith are always good.

To say that “most” academic research is “fake” also implies that we can quantify how much of it is fake or not. I can’t precisely estimate, quantify, and judge every academic paper that gets published out there, so I don’t claim to know exactly how much of the current research being published is reliable. It probably also varies by fields, and it’s possible for papers to include a mixture of true and fake data, reasoning, and conclusions.

And thereon the essay goes to saying "most" without any description what "research" is taken as a sample set. Hastings' comment, on the other hand, suggested some alternatives:

Uniform measure is not going to be exciting – you’re going to get almost entirely undergraduate assignments and Third World paper mills. If your weighted sampler is “papers linked in articles about how academia is woke” you’re going to find a high %fake. If your weighed measure is “papers read during work hours by employees at F500 companies” you’ll find a lower, nonzero %fake.

 

However, we can generally say that most recent research in humanities (or human-centric sciences) is fake, redundant, or useless; most research in earth-centric sciences is true, fake, or questionable; and most research in STEM fields is true, fake, outdated, or redundant.

Major nitpicking here. If most true claims of human-centric studies are replicated, then each of the corresponding papers are redundant (as it'd have a duplicate); therefore, almost all research would be "fake, redundant, or useless". Moreover, for STEM fields "true, fake, outdated, or redundant" seems to describe universal set - that is, that statement is of no substance. I'd suggest clarifying what claims you had in mind, if you are not using them for emphasis only.

The best rule is not to assume that because an academic paper says X, that X is true.

The best rule known to us - i.e. Bayesian reasoning - mandates that we simply treat "research" as stream of characters, and assess probabilities of each stream being shown to you if X were true and if X were false. That is intractable; after some fallback, you get at "correct for authors' biases, and assume that paper's claims represent average of what happens". I have the impression LessWrong does pretty much that.

Academic research can be fake in different ways. It can simply be false. It can be emotionally manipulative propaganda masquerading as knowledge. It can be irrelevant or meaningless.

Specific claim being true or false necessarily screens off being "emotionally manipulative propaganda". A weaker point that would stand, though: "often papers are emotionally manipulative, even when the claims presented in them are inappicable to most real situations or meaningless outside of academia".

I believe the further parts of Sections 1 and 2 are not of much interest for LessWrong, except that they attempt establishing common knowledge that academic "research" is commonly fake. Section 3, with specific suggestions, could be positively received when posted separately.

Comment by ProgramCrafter (programcrafter) on Why I’m not a Bayesian · 2024-10-08T18:43:30.296Z · LW · GW

Understood. Does that formulation include most useful sentences?

For instance, "there exists a sentence which is more true than this one" must be excluded as equivalent to "this statement's truth value is strictly less than 1", but the extent of such exclusion is not clear to me at first skim.

Comment by ProgramCrafter (programcrafter) on What is it like to be psychologically healthy? Podcast ft. DaystarEld · 2024-10-07T21:59:04.971Z · LW · GW

I think this is implicitly making the target too narrow for people that care about getting there and might consider this a reference point.

Does a narrow, hardly achievable target actually have a negative effect? It would be interesting to see some research, in particular as it'd imply sequence "Challenging the Difficult" makes things worse for part of people.

Comment by ProgramCrafter (programcrafter) on Why I’m not a Bayesian · 2024-10-07T21:49:13.011Z · LW · GW

As Richard noted, meaning is context-dependent. When I say "is there water in the fridge?" I am not merely referring to h2o; I am referring to something like a container of relatively pure water in easily drinkable form.

Then why not consider structure as follows?

  1. you are searching for "something like a container of relatively pure water in easily drinkable form" - or, rather, "[your subconscious-native code] of water-like thing + for drinking",
  2. you emit sequence of tokens (sounds/characters) "is there water in the fridge?", approximating previous idea (discarding your intent to drink it as it might be inferred from context, omitting that you can drink something close to water),
  3. conversation partner hears "is there water in the fridge?", converted into thought "you asked 'is there water in the fridge?'",
  4. and interprets words as "you need something like a container of relatively pure water in easily drinkable form" - or, rather, "[their subconscious-native code] for another person, a water-like thing + for drinking".

That messes up with "meanings of sentences" but is necessary to rationally process filtered evidence.

Each statement that the clever arguer makes is valid evidence—how could you not update your probabilities? Has it ceased to be true that, in such-and-such a proportion of Everett branches or Tegmark duplicates in which box B has a blue stamp, box B contains a diamond? According to Jaynes, a Bayesian must always condition on all known evidence, on pain of paradox. But then the clever arguer can make you believe anything they choose, if there is a sufficient variety of signs to selectively report.

Comment by ProgramCrafter (programcrafter) on Compelling Villains and Coherent Values · 2024-10-07T21:14:51.969Z · LW · GW

Also, it is interesting to consider whether there are deep reasons that coherent belief seems to be anti-correlated with coherent morals.

I believe a major part of reason concentrates on "seems"; perhaps many people are [implicitly] [almost] rational (or not faced with practical problems challenging one's thinking very often), but don't give the vibe of having very coherent beliefs.

n=1. Once, I retold a few articles of Sequences (epistemic part) to my friend, who answered like "well, I have not read anything like that, but it maps to my understanding in finding out what happens around". And indeed, there seemed to be null difference between him and the techniques. I did not try testing for differences, as it would be too time-consuming and generally not to the best for me.

Comment by ProgramCrafter (programcrafter) on An argument that consequentialism is incomplete · 2024-10-07T20:43:06.999Z · LW · GW

To use a physics analogy, utility often isn't a potential function over state of affairs, and for many depends on path taken.

However, state of affairs is but a projection; state of world also includes mind states, and you might be indifferent between any quantum paths to worlds involving same mind state (including memory, beliefs) for you. (As a matter of values, I am not indifferent between paths either; rather, I endorse some integrated-utility up to an unspecified pinning point in future.)

Comment by ProgramCrafter (programcrafter) on Why I’m not a Bayesian · 2024-10-07T17:58:45.826Z · LW · GW

Unfortunately, although the new 1/2 truth value can resolve some paradoxes, it introduces new paradoxes. "This sentence is either false or 1/2" cannot be consistently assigned any of the three truth values.

Under some plausible assumptions, Lukaziewicz shows that we can resolve all such paradoxes by taking our truth values from the interval [0,1]...

Well, a straightforward continuation of paradox would be "This sentence has truth value in "; is it excluded by "plausible assumptions" or overlooked?

Comment by ProgramCrafter (programcrafter) on Why I’m not a Bayesian · 2024-10-07T15:35:09.557Z · LW · GW

I don't seem to understand how you use the word "thing" here; if it can refer to a physical object, then what computations can a wooden crate do, for instance? If none, then it doesn't get characterized different to a cup, and that seems strange..

Comment by ProgramCrafter (programcrafter) on What Hayek Taught Us About Nature · 2024-10-04T22:26:07.535Z · LW · GW

What if there was a great popular movement to see concrete evidence of project activities and impacts?

It would work, but it is not simple to get one; you'd need a culture of people who can actually distinguish between true and fabricated evidence, and be generally epistemically rational, and the questions must not form disconnected beliefs network. Net of falsehoods might collapse eventually; but with adversaries, net of truth could be warped as well.

Comment by ProgramCrafter (programcrafter) on Review: Dr Stone · 2024-10-03T17:16:06.389Z · LW · GW

There is also an alternative realistic scenario where Tsukasa [temporarily?] wins. That happens if upon killing Senku, Tsukasa buries or destroys his body himself.

Though, to get a scientist not trying to undermine the empire, Tsukasa would need to play Newcomb-like Omega (conditioning reviving the scientist on him cooperating, as any other option qualifies as a ignorable threat). Not getting any science advances would put the tribe on the edge of loss (due to illnesses, Petrification Kingdom, inability to revive people fast enough, inability to anticipate outcomes of any specific revival order, etc).

Comment by ProgramCrafter (programcrafter) on Open Thread Summer 2024 · 2024-09-30T16:41:56.866Z · LW · GW

I'm at this point pretty confident that under the Copenhagen interpretation, whenever an intergalactic photon hits earth, the wave-function collapse takes place on a semi-spherical wave-front many millions of lightyears in diameter. I'm still trying to wrap my head around what the interpretation of this event is in many-worlds.

Under MWI, before the photon (a peak in EM field) could hit Earth, there were a lot of worlds differing by EM field values ("electromagnetic tensor") - and, thus, with different photon directions, position, etc. Each of those worlds led to a variety of worlds; some, where light hit Earth, became somewhat different from those where light avoided it; so, integrated probability "photon is still on the way" decreases, while P(photon has been observed) increases. Whenever some probability mass of EM disturbances arrives, it is smoothly transformed, with no instant effects far away.

Comment by ProgramCrafter (programcrafter) on Where is the Learn Everything System? · 2024-09-28T22:10:15.664Z · LW · GW

Well, a Learn Everything System (LES) run by an AI tutor that adapts the content to fully engage you in a game-based educational experience run as a rich simulation of all human knowledge seems to me to be the ideal form of learning - for probably almost everyone.

I have another architecture in mind, actually (but I think specific choice is only a means towards the end).

The tutor must be able to infer how the student arrives at their conclusion for a few demo problems, that being "student's problem-solving structure[1]". Structure is then corrected towards valid for the problem (it might be helpful to demonstrate examples when structures yield different answers or take different time to complete). Though, that may run into an issue, making people less skeptic:

I recall reading, though I can't remember where, that physicists in some country were more likely to become extreme religious fanatics. This confused me, until the author suggested that physics students are presented with a received truth that is actually correct, from which they learn the habit of trusting authority.

It may be dangerous to present people with a giant mass of authoritative knowledge, especially if it is actually true. It may damage their skepticism.

  1. ^

    I say "structure" instead of "algorithm" because it often fails to solve the given problem, is very much non-deterministic, and also its intermediate nodes (lemmas, conjectures, estimations what path is closer to a cleanly-looking solution) are useful

Comment by ProgramCrafter (programcrafter) on Searching for Impossibility Results or No-Go Theorems for provable safety. · 2024-09-27T23:55:14.285Z · LW · GW

I think a major part of problem is that we haven't yet formalized what we need to prove - in other words, existing theorems' conclusions might be disconnected from actual world.

Comment by ProgramCrafter (programcrafter) on What is Randomness? · 2024-09-27T23:14:32.224Z · LW · GW

How do probabilities arise in the Everett interpretation of quantum mechanics?

After a series of quantum experiments, there will be a version of the observer who sees a sequence of very unlikely outcomes. How to make sense of that?

Well, in MWI there is the space of worlds, each associated with a certain complex number ("amplitude"). Some worlds can be uniform hydrogen over all the universe, some contain humans, a certain subspace contains me (I mean, collection of particles moving in a way introspectively-identical to me) writing a LessWrong comment, etc.

It so happens that the larger magnitude of said complex number is, the more often we see such world; IIRC, that inequality allows to prove that likelihood of seeing any world is proportional to squared modulo of amplitude, which is Born's rule.

The worlds space is presumably infinite-dimensional, and also expands over time (though not at exponential rate as is widely said, because "branches" must be merging all the time as well as splitting). That means that probability distribution assigns a very low likelihood to pretty much any world... but why do we get any outcomes then?

I'm not attempting to answer question why we experience things in the first place (preferring instead to seal it even for myself), but as for why we continue to do so conditional on experiencing something before: because of the "conditional on". Conditional probability is the non-unitary operation over our observations of phase space, retaining some parts while zeroing all others, which are "incompatible with our observations"; also, as its formula is , it can amplify likelihoods for small values of . That doesn't totally fix the issue, but I believe the right thing to do in improbable worlds is to continue updating on evidence and choosing best actions as usual.
(To demonstrate the issue with small probabilities is not fixed, let's divide likelihood of any single world by ; here's a 256-bit random string: e697c6dfb32cf132805d38cf85a60c832247449749293054704ad56209d2440e).

Comment by ProgramCrafter (programcrafter) on Doing Nothing Utility Function · 2024-09-26T22:39:31.156Z · LW · GW

Your idea seems to break when AI is being unpaused: as it has not done any beneficial actions, utility would suddenly go down from "simulated" to "normal", meaning that AI will likely resist waking it up.

Also, it assumes there is a separate module for making predictions, which cannot be manipulated by the agent. This assumption is not very probable in my view.

Comment by ProgramCrafter (programcrafter) on ProgramCrafter's Shortform · 2024-09-06T21:37:22.948Z · LW · GW

A sufficient condition for a good [possibly AI-gen] plan

Current posts mainly focus on necessary properties of plans they'd like to see/be executed. I suggest a sufficient condition:

Plan is good, should be acted upon, etc at least when it is endorsed in advance, endorsed in retrospect and endorsed in counterfactual.

  1. Endorsed in advance: everyone relevant hears the plan and possible outcomes in advance, evaluates acceptability and accepts the plan.

  2. Endorsed in retrospect: everyone relevant looks upon intended outcomes, checks what happened actually, evaluates plan and has no regret.

  3. Endorsed in counterfactual: given choice in a set of plans, person would evaluate the specific plan as acceptable - somewhat satisfying them, not inducing much desire to switch.

Choice according to these criteria is still hard, but it should be a bit less mysterious.

Comment by ProgramCrafter (programcrafter) on What are the effective utilitarian pros and cons of having children (in rich countries)? · 2024-09-02T16:58:56.796Z · LW · GW

Halving the global population has the same effect on climate as doubling the size of the Earth's atmosphere

assuming economics (CO2 emission) scales linearly with population.

(alt idea) I think a large contributor to greenhouse gases is transport to remote areas, so solving problems with housing prices in local areas could somewhat concentrate people and help with ecology.