Posts
Comments
That article is suspiciously scarce on what microcontrols units... well, glory to LLMs for decent macro management then! (Though I believe that capability is still easier to get without text neural networks.)
In StarCraft II, adding LLMs (to do/aid game-time thinking) will not help the agent in any way, I believe. That happens because inference has a quite large latency, especially as most of prompt changes with all the units moving, so tactical moves are out; strategic questions "what is the other player building" and "how many units do they already have" are better answered by card-counting counting visible units and inferring what's the proportion of remaining resources (or scouting if possible).
I guess it is possible that bots' algorithms are improved with LLMs but that requires a high-quality insight; not convinced that o1 or o3 give such insights.
I don't think so as I had success explaining away the paradox with concept of "different levels of detail" - saying that free will is a very high-level concept and further observations reveal a lower-level view, calling upon analogy with algorithmic programming's segment tree.
(Segment tree is a data structure that replaces an array, allowing to modify its values and compute a given function over all array elements efficiently. It is based on tree of nodes, each of those representing a certain subarray; each position is therefore handled by several - specifically, nodes.)
Doesn't the "threat" to delete the model have to be DT-credible instead of "credible conditioned on being human-made", given that LW with all its discussion about threat resistance and ignoring is in training sets?
(If I remember correctly, a decision theory must ignore "you're threatened to not do X, and the other agent is claiming to respond in such a way that even they lose in expectation" and "another agent [self-]modifies/instantiates an agent making them prefer that you don't do X".)
The surreal version of the VNM representation theorem in "Surreal Decisions" (https://arxiv.org/abs/2111.00862) seems to still have a surreal version of the Archimedean axiom.
That's right! However it is not really a problem unless we can obtain surreal probabilities from the real world; and if all our priors and evidence are just real numbers, updates won't lead us into the surreal area. (And it seems non-real-valued probabilities don't help us in infinite domains, as I've written in https://www.lesswrong.com/posts/sZneDLRBaDndHJxa7/open-thread-fall-2024?commentId=LcDJFixRCChZimc7t.)
Re the parent example, utility function (or its evaluations) changing in an expectable way seems problematic to rational optimizing. If you know you prefer A to B, and know that you will prefer B to A in future even given only current context (so no "waiter must run back and forth"), then you don't reflectively endorse either decision.
Yes, many people will have problems with the Archimedes' axiom because it implies that everything has a price (that any good option can be probability-diluted enough that a mediocre is chosen instead), and people don't take it kindly when you tell "you absolutely must have a trade-off between value A and value B" - especially if they really don't have a trade-off, but also if they don't want to admit or consciously estimate it.
Thankfully, that VNM property is not that critical for rational decision-making because we can simply use surreal numbers instead.
One possible real-world example (with integer-valued for deterministic outcomes) would be a parent whose top priority is minimizing the number of their children who die within the parent's lifetime, with the rest of their utility function being secondary.
Wouldn't work well since in real world outcomes are non-deterministic; given that, minimizing expected number is accomplished by simply having zero children.
But technology is not a good alternative to good decision making and informed values.
After thinking on this a bit, I've somewhat changed my mind.
(Epistemic status: filtered evidence.)
Technology and other progress has two general directions: a) more power for those who are able to wield it; b) increasing forgiveness, distance to failure. For some reason, I thought that b) was a given at least on average. However, now it came to mind that it's possible for someone to
1) get two dates to accidentally overlap (or before confirming with partners-to-be that poly is OK),
2) lose an arbitrarily large bunch of money on gambling just online,
3) take revenge on a past offender with a firearm (or more destructive ways, as it happens),
and I'm not sure the failure margins have widened over time at all.
By the way, if technology effects aren't really on topic, I'm open to move that discussion to shortform/dialogue.
---
(Epistemic status: obtained with introspection.)
Continuing the example with sweets, I estimate my terminal goals to include both "not be ill e.g. with diabetes" and "eat tasty things". Given tech level and my current lifestyle, there isn't instrumental goal "eat more sweets" nor "eat less sweets"; I think I'm somewhere near the balance, and I wouldn't want society to pass any judgement.
I object to the framing of society being all-wise, and instead believe that for most issues it's possible to get the benefits of both ways given some innovators on that issue. For example, visual communication was either face-to-face or heavily resource-bounded till the computer era - then there were problems of quality and price, but those have been almost fully solved in our days.
Consequently, I'd prefer "bunch of candy and no diabetes still" outcome, and there are some lines of research/ideas into how this can be done.
As for "nonmarital sex <...> will result in blowing past Goodhart's warnings into more [personal psychological, I suppose] harm than good", that seems already solved with the concept of "commitment"? The society might accept someone disregarding another person if that's done with plausible deniability like "I didn't know they would even care", and commitment often makes you promise to care about partner's feelings, solving* the particular problem in a more granular way than "couples should marry no matter what". The same thing goes with other issues.
That said, I've recently started to think that it's better to not push other people to less-socially-accepted preferences unless you have a really good case they can revert from exploration well and would be better off (and, thus, better not to push over social networks at all), since the limit point of person's preferences might shift - wicked leading to more wicked and so on - to the point person wouldn't endorse outcomes of change on reflection. I'm still considering if just noting that certain behavior is possible is a nudge significant enough to be disadvantaged (downvoted or like).
*I'd stop believing in that if commitment-based cultures had higher rate of partners failing on their promises to care than marriage-based; would be interested in some evidence either way.
Nicely written!
A nitpick: I believe "Voluntary cooperation" shouldn't always be equal to "Pareto preferred". Consider an Ultimatum game, where two people have 10 goodness tokens to split; the first person suggests a split (just once), then the second may accept or reject (when rejecting, all tokens are discarded). 9+1 is Pareto superior to 0+0 but one shouldn't [100%] accept 9+1 lest that becomes anything they are ever suggested. Summarizable with "Don't do unto yourself what you wouldn't want to be done unto you", or something like that.
Well I'd really love to see a practical example because for now the text clicks as "yes, that's how it should be, indeed, and what's novel here?". (By the way it seems you haven't yet started at how the gains would be divided; that seems relevant for continuing, and is already described in https://www.lesswrong.com/posts/vJ7ggyjuP4u2yHNcP/threat-resistant-bargaining-megapost-introducing-the-rose.)
Do you have an operational definition of "intrinsic preferences"?
Let me try. I believe that if copies of a person (as determined by their genotype*) would be raised in different cultures and environments, their revealed preferences would mainly be clustered around a single point, with some shifts determined by what desires society showed them as acceptable/tractable and what as unfashionable. Given proper diversity of surrounding cultures list, I'd say the cluster median indicates person's intrinsic preferences.
*it also seems intuitively right to discount traits that were [IVF?]optimized for, if any, as an outside influence; though, I have no strong opinion on this.
This article seems like it was not written for LessWrong? It explores how we need to do conscious thinking, how we need to suggest and evaluate choices using our emotions, and essentially rephrases part of Sequences while reverting to pre-LW terminology, for "rationality" in particular.
And on top of that, "superrationality" concept is described unsoundly:
Different choices are impossible ...
then if a dilemma requires two players to choose different ways at risk of losing (e.g. colliding on road, or booking one conference room at the same time), the text implies there would be no valid choice. In fact we know that there are randomized strategies, and that there often are distinctors useful for decision making.
DeekSeek v3: The Six Million Dollar Model
A typo in the title.
When happy and content, the Assistant Characters can channel more helpfulness and capabilities.
Is there anything preventing them to channel capabilities and helpfulness regardless of context, or is it in a large part derived?
I also use this idea of conditioning on context when taking notes on probability+decision theory!
In the same vein, I suggest to use for "philosophers of perfect emptiness" who take nothing but mathematical logic as valid.
At personal level, "debt" usually stands for something that will be paid back eventually. Not claiming whether US should strive to pay out most debt, but that may help explain the people's intuitions.
Does your family have the same opinions as your social circle?
Quite similar, in fact - at least where they care to do so! I do listen for perspective, but I still can't put society's revealed opinion into span of those who I know better!
Getting to know your neighbors is another way to expose yourself to people who often think differently.
A good idea! I'll have to take it a bit more general, because I'm an university student with dormitory and already know many people around; though, eating at some local cafe with diverse customers should work.
Being an Uber driver would be too taxing on my time, but I'm sure there is another idea instantiation which would work!
(And while we're in an Open Thread, I'd like to thank LessWrong for featuring that one can make informed decisions on pretty much any topic! I've chosen uni aligned with me and I'm not disappointed with it so far.)
Tsukasa just plain lost and the logical ending is for him to be stoned or killed
except that
1) there is sometimes too much of such hostility in the real world,
2) some people can legitimately be redeemed - that is, they change thinking strategies and approximations to their values when they see what has been created (re: Einstein's reaction to atomic bomb creation),
3) I don't think anyone depicted in the anime would have valued fairness/consistency/other base for punishing Tsukasa - and with death penalty, no less - over compassion.
Authors might have held the position (shout-out to @the gears to ascension!) "[we] want literally every human to get to go to space often and come back to a clean and cozy world. This currently seems unlikely. Let's change that." Even if they haven't: there is such vibe, and I somewhat endorse it!
Nice to see more fans!)) I have also written a review with more focus on rationality demonstration within the anime: https://www.lesswrong.com/posts/yqXoFx7jsSjbWgvBq/review-dr-stone.
How can I get an empathic handle on my region/country/world's society (average/median/some sample of its people, to be clearer)?
I seem to have got into a very specific social circle, being a constant LW reader and so on. That often makes me think "well, there is question X, a good answer is A1 and it is also shared within the circle, but many people overall will no doubt think A2 and I don't even know who or why". I can read survey/poll results but not understand why would people even want to ban something like surrogate motherhood or things like that.
I've heard one gets to know people when works with them. If so, I'd like to hear suggestions for some [temporary] professions which could aid me here?
We can suggest a Weak Zombie Argument: It is logically possible to have a universe where all qualia of red and green are inverted in the minds of its inhabitants, while all physical things remain the same.
I'd say that universe you describe IS one we're living in (note I'm not using "equal to" or "same as", I'm rejecting that those are two different worlds). It doesn't really matter for anything happening in the universe whether strawberry color is labelled "red", "black", "colF00" or "qualia_fa615f8f", or if that particular quale might never happen, as long as causal network is isomorphic to the original one.
Shapley values are the ONLY way to guarantee: <Efficiency, Symmetry, Linearity, Null player properties>
Well it doesn't end at that: it turns out Shapley values for more than 2 players are not nicely behaved and instead violate Maximin Dominance, as demonstrated in https://www.lesswrong.com/posts/vJ7ggyjuP4u2yHNcP/threat-resistant-bargaining-megapost-introducing-the-rose#ROSE_Value__N_Player_Case__.
The article I link showed how this is fixed:
Shapley values are about adding everyone one-by-one to a team in a random order and everyone gets their marginal value they contributed to the team.
And that's kinda like giving everyone a random initiative ordering and giving everyone the surplus they can extract in the resulting initiative game.
If we're doing that, then maybe a player, regardless of their position, can ensure they get their maximin value? Maybe this sort of Random-Order Surplus Extraction can work. ROSE.
I've completed the survey, skipping just a few questions I thought too ambiguous!
(Note: I've waited a bit to intentionally mask submit time, since otherwise I'd be identifiable by brag order :-) )
I think CBs won't work as described at least because they need to be allocated to children/new citizens somehow? Given that territory of adopting country is non-increasing while its population increases over time, the state would need to buy out part of CB and grant those pieces, and that would presumably create CB borders drift.
Overall, I downvoted this post because I felt it only describes "that's the idea of expevolu, and its benefits", not realistically exploring what would happen upon adoption, and what drawbacks are inherited from previously suggested ideas.
It's a solution! However it comes with its own downsides. For instance, Codeforces users ranted on Cloudflare usage for a while, with following things (mapped to LessWrong) highlighted:
- The purpose of an API is defeated: even the API endpoints on the same domain are restricted, which prevents users from requesting posts via GraphQL. In particular, ReviewBot will be down (or be hosted in LW internal infrastructure).
- In China, Cloudflare is a big speed bump.
- Cloudflare-protected sites are reported to randomly lag a lot.
> I had been assuming that this is a server problem, but from talking to some people it seems like this is an issue with differential treatment of who is accessing CF.
Lack of interaction smoothness might be really noticeable for new users, comparing to current state.
Thank you for highlighting philosophical position!
I take it that Tolkien [early on] thought human thinking is often malign to their desires and corrupts the latter when given enough power; and power without cognition improvement often comes with machines and other tools for doing something new; so people should be truer to their desires and prefer to use things without much relying on them.
This position is clearly in contrast with uplifting, which suggests giving everyone more power to lead fulfilling lives - as much power as possible - and backing the supply with full force of humanity. I would actually try uplifting, however little the stable solution zone would be, because I feel that to be cool!
Well it probably is computing insofar it's the wind bringing in actual bits of information, not you while searching for a specific pattern instantiation. The test: consider whether if original grains were moved around to form another prime number, would the wind still process them a similar way and yield correct answer?
I've started writing a small research paper on this, using mathematical framework, and understood that I had long conflated Shapley values with ROSE values. Here's what I found, having corrected that error.
ROSE bargaining satisfies Efficiency, Pareto Optimality, Symmetry*, Maximin Dominance and Linearity - a bunch of important desiderata. Shapley values, on other hand, don't satisfy Maximin Dominance so someone might unilaterally reject cooperation; I'll explore ROSE equilibrium below.
- Subjects: people and services for finding partners.
- By Proposition 8.2, ROSE value remains same if moves transferring money within game are discarded. Thus, we can assume no money transfers.
- By Proposition 11.3, ROSE value for dating service is equal or greater than its maximin.
- By Proposition 12.2, ROSE value for dating service is equal or less than its maximum attainable value.
- There's generally one move for a person to maximize their utility: use the dating service with highest probability of success (or expected relationship quality) available.
- There are generally two moves for a service: to launch or not to launch. First involves some intrinsic motivation and feeling of goodness minus running costs, the second option has value of zero exactly.
- For a large service, running costs (including moderation) exceed much realistic motivation. Therefore, maximum and maximin values for it are both zero.
- From (7), (3) and (4), ROSE value for large dating service is zero.
- Therefore, total money transfers to a large dating service equal its total costs.
So, why yes or why no?
By the way, Shapley values suggest paying a significant sum! Given value of a relationship of $10K (can be scaled), and four options for finding partners ( -- self-search, -- friend's help, -- dating sites, -- the specialized project suggested up the comments), the Shapley-fair price per success would be respectively $550, $650 and $4400.
P.S. I'm explicitly not open to discussing what price I'd be cheerful to pay to service which would help to build relationships. In this thread, I'm more interested in whether there are new decision theory developments which would find maximin-satisfying equilibria closer to Shapley one.
at a $100k valuation of a marriage
I don't think one can coherently value marriage 20 times as much as than a saved life ($5k as GiveWell says)? Indeed there is more emotional attachment to a person who's your partner (i.e. who you are emotionally attached to) than to a random human in the world, but surely not that much?
And if a marriage is valued at $10k, then the credit assignment 1%/10% would make the allocation $100/$1000 - and it seems that people really want to round the former towards zero
When rereading [0 and 1 Are Not Probabilities], I thought: can we ever specify our amount of information in infinite domains, perhaps with something resembling hyperreals?
- An uniformly random rational number from is taken. There's an infinite number of options meaning that prior probabilities are all zero (), so we need infinite amount of evidence to single out any number.
(It's worth noting that we have codes that can encode any specific rational number with a finite word - for instance, first apply bijection of rationals to natural numbers, then use Fibonacci coding; but in expectation we need to receive infinite bits to know an arbitrary number).
Since symbol doesn't have nice properties with regards to addition and subtraction, we might define a symbol which means "we need some information to single out one natural number out of their full set". Then, the uniform prior over would have form (prefix and suffix standing for values outside segment) while a communication "the number is " would carry bits of evidence on average, making the posterior . - The previous approach suffers from a problem, though. What if two uniformly random rationals are taken, forming a square on coordinate grid?
If we've been communicated information about , we clearly have learned nothing about and thus cannot pinpoint the specific point, requiring more bits.
However, there's bijection between and , so we can assign a unique natural number to any point in the square, and therefore can communicate it in bits in expectation, without any coefficient .
When I tried exploring some more, I've validated that greater uncertainty (, communication of one real number) makes smaller ones () negligible, and that evidence for a natural number can presumably be squeezed into communication for a real value. That also makes the direction look unpromising.
However, there can be a continuation still: are there books/articles on how information is quantified given a distribution function?
Never say 'nothing' :-)
- the world might be in such state that attempts to do good bring it into some failure instead, and doing the opposite is prevented by society
(AI rise and blame-credit which rationality movement takes for it, perhaps?) - what if, for some numerical scale, the world would give you option "with 50%, double goodness score; otherwise, lose almost everything"? Maximizing EV on this is very dangerous...
I guess I'm looking for questions of this family:
- Do you sometimes tell things that are not literally true but help the person you're talking to in understanding?
- On average, do you believe statements by members of rationalist community significantly more (+1.0 bit of evidence or more) than exact same words from non-rationalists?
- What is the biggest barrier you face when trying to communicate rational ideas to others? [a) Emotional resistance b) Lack of shared vocabulary c) Time constraints d) Preexisting strong beliefs e) Complexity of ideas f) People disengaging randomly]
Also,
- Have you ever intervened on someone's behalf where the person was failing and would prefer to succeed?
- How many people can a [brainstorming] conversation hold on average, so that everyone is active?
I feel like an important question is: how far does this generalize? We can estimate the IQ gap between the dumbest person who successfully uses the internet (probably in the 80's) and the smartest malware author (got to be at least 150+). Is that the limit somehow, or does this knack extend across even more orders of magnitude?
If imagine a world where 100 IQ humans are using an internet that contains malware written by 1000 IQ AGI, do humans just "avoid the bad parts"?
For reactive threats, the upper bound is probably at most "people capable of introspection who can detect they are not sure some action will be to net benefit, and therefore refuse to take it". For active threatening factors, that's an arms race (>=40% this race is not to infinity - basically, if more-cooperating DT strategies are any good).
Maybe the subject is researched more in biology? Example topic: eating unknown food (berries, nuts) in forest, and balance of lifetime adaptation vs evolutionary adaptation (which involves generations passing).
It would be nice to see at least three questions which would demonstrate how person extracts evidence from others' words, how much time and emotions could they spend if they needed to communicate a point precisely, etc.
I'll have to sleep on that, actually. Will return tomorrow, presumably with more concrete ideas)
99.9% of all cryptocurrency projects are complete scams (conservative estimate).
On first skim, I agree with the estimate as stated and would post a limit order for either side. I'd also like to note that "crypto in general is terrible" instead of "all crypto is terrible", as there have been applications developed that do not allow you to lose all funds without explicit acknowledgement.
Similarly, Cyber Security is terrible. Basically every computer on the internet is infected with multiple types of malware.
It is presumably terrible (or, 30%, result of availability bias), and I've observed bugs happen because functionality upgrade did not consider its interaction with all other code. However, I disagree that every computer is infected; probably you meant that it is under constant stream of attack attempts?
The insecure domains mainly work because people have charted known paths, and shown that if you follow those paths your loss probability is non-null but small. As a matter of IT, it would be really nice to have systems which don't logically fail at all, but that requires good education and pressure-resistance skills for software developers.
I think TMS doesn't rewrite anything, instead activating neural circuits in another pattern? Then, new pattern is not depressed, brain can notice that (on either conscious or subconscious level) and make appropriate changes to neural connections.
Basically, I believe that whatever resulting patterns (including "other parts of you changed into something non-native and alien") you dis-endorse, are "committed" with significantly lower probability.
P(Bitcoin) What is the probability that the price of one bitcoin will be higher on July 1st, 2025, than it was on December 1st, 2024? ($???)
Probably best to include "what price of one bitcoin do you expect on July 1st, 2025, given that it was $??? on December 1st, 2024?" as well.
You could also include P(weak EMH) - instead of P(GPT-5 Release) if there's not enough space.
Overall, the questions seemed insufficiently checking social skills to me, instead preferring testing large, "impactful" beliefs.
How do they figure out what waveforms to use and at what frequencies?
Presumably Fourier transform or its developments can be used. (After all, light speed is negligible at the scale we're considering.)
In the underlying computer which executes the actions which are represented by this interface, there is nothing that resembles a pointer
Course' there are: probably 64 bits in memory (for more degree of detail: 64 places of persistent electric charge with two stable states), which change iff pointer moves, and each bit restricts the places pointer can appear at. That resemblance exists certainly; I also agree there's no resemblance like "small pointer-like thing/charge pattern in RAM module".
In other words, one has to taboo "resemblance" but it's not clear if that can be done.
I interpreted your algorithm for listing computables to be something like "enumerate the Turing machines that output '.' then 0s and 1s and list what they print", without worrying about the fact that some computables repeat.
I'm pretty sure my argument did not mention how computables are listed at all, rather proving that for any specific listing the inverse-diagonal is computable as well.
If you have any surjection: N→S⊂(N→{0,1}) and diagonalize against it, you know the result is not in S. This fact doesn't depend on the actual nature of the surjection N→S, just that S is the image. Here S is the computables.
Yes. However, it's the specific choice of set "computables" which creates the contradiction: I agree with "inverse-diagonal for rationals is an irrational number" and like.
Once again: for any "user-provided" computable table of computable digit sequences, I can, in finite time, get value for any specific position in table; therefore, each digit of inverse sequence is computable; therefore, I conclude that the inverse-diagonal sequence is itself computable (if I'm not mistaken in definitions).
- Retrieve I-th computable from table: computable by assumption.
- Get K-th digit from a given computable: computable by definition.
- Get I-th digit of I-th computable: computable as composition of (1) and (2).
- Invert given digit: trivially computable.
- Get an inverted I-th digit of I-th computable: computable as composition of (3) and (4).
I've actually written formal refutation of any bijection between and in Idris 2.
Demonstration
total stands for functions which are defined everywhere on their stated domain (as opposed to partial), and computations for which are proven to always halt (if there's no such guarantee, function is covering). %default total means that all functions below must be total.
Void type means that a contradiction was obtained from given arguments.
%default total
data Bijection : Type -> Type -> Type where
||| A bijection between two types.
||| @ a the first type
||| @ b the second type
||| @ fwd mapping from first type to the second
||| @ bck mapping from second type to the first
||| @ fb_producer proof that fwd . bck = id
||| @ bf_producer proof that bck . fwd = id
DoMap : {0 a,b : Type} -> (fwd: a -> b) -> (bck: b -> a)
-> (fb_producer: (av : a) -> (bck $ fwd av) = av)
-> (bf_producer: (bv : b) -> (fwd $ bck bv) = bv)
-> Bijection a b
map_fneq : {0 a,b : Type} -> {f : a->b} -> {g : a->b} -> f = g -> (v:a) -> f v = g v
map_fneq (Refl {x = f}) v = Refl
bool_inv : {v : Bool} -> (v = not v) -> Void
bool_inv {v = True} prf = uninhabited prf
bool_inv {v = False} prf = uninhabited prf
diagonal : (Bijection Nat (Nat -> Bool)) -> Void
diagonal (DoMap f g _ bf_prf) = bool_inv conflict where
H : Nat -> Bool
H i = not $ f i i
h_hi_by_idx : f (g H) (g H) = H (g H)
h_hi_by_idx = map_fneq (bf_prf H) (g H)
h_hi_by_def : H (g H) = (not $ f (g H) (g H))
h_hi_by_def = Refl
conflict : f (g H) (g H) = (not $ f (g H) (g H))
conflict = trans h_hi_by_idx h_hi_by_def
If you put the computable numbers in the table, the inverted diagonal will be an uncomputable number.
In fact,
diag-a1. if you put the computable numbers in the table, and
diag-a2. pretend that the assignment is computable also, and
diag-a3. (optionally) pretend that procedure for looking up the row of specific number is computable also,
then
diag-l1. the inverted diagonal is computable by explicit construction,
diag-l2. the inverted diagonal does not belong to the table,
thus
diag-t1. we have the contradiction (based on diag-l1 and diag-l2),
diag-t2. (if diag-a3 holds) the contradicting place is computable as well.
I agree with your view but that specific example was wrong!
A goal is, fundamentally, an idea. As the final step in a plan, you can write it out as a symbolic representation of the “world state” you are trying to achieve, although it could represent other things as well. In a planning computer agent, this will probably terminate in a bunch of 1s and 0s stored in its memory.
In order for this symbolic representation to be meaningful, it must be comparable and distinct from other symbolic representations. World state A in the agent's plan could be contrasted from world state B, C and D. This is a very fundamental fact about how information and meaning work, if World State A was indistinguishable from all the others, there would be no reason for the agent to act, because its goal would have been “accomplished”.
This has a logic error. There need not be one best world state, and a world state need not be distinguishable from all others - merely from some of them. (In fact, utility function yielding a real value compresses the world into a characteristic of things we care about in such a way.)
Also, with unbounded computations, utility optimizer could tell supremum (best outcome) for any set of world states you'd provide it; without that, it will have less granularity, work on set of close states (for instance, "easily coming to human mind") or employ other optimization techniques.
I believe this underlies much of the disagreement, because then more knowledge or more intelligence might change only the relations of "final goal" sign but not its meaning (re: isomorphism).
Your series of posts also assume that signs have a fixed order. This is false. For instance, different fields of mathematics treat real number as either first order signs (atomic objects) or higher-order ones, defined as relations on rational numbers.
Or, for an easier example to work on: equality could be a second-order sign "object A is same as object B", or it may be defined using third order expression "for any property P, A and B either both have the property or both not have it". It is no coincidence that those definitions are identical; you cannot assume that if something is expressible using higher order signs, is not also expressible in lower order.
And this might undermine the rest of argument.
Engaging with the perspective of orthogonality thesis itself: rejecting it means that a change in intelligence will lead, in expectation, to change in final goals. Could you name the expected direction of such a change, like "more intelligent agents will act with less kindness"?
Before power grid dissolves, it has also to hit factories (and business in general). I don't think resulting price increments are predictable - they might as well start some crisis in economy. (And there might be money-unrelated outcomes, like if restaurants start ignoring some safety standards trying to save on electricity or like, which could be disastrous with habit of eating out.)
She certainly gets a reward for following experimental protocol, but beyond that... I concur there's the problem, and I have the same issue with standard formulation asking for probability.
In particular, pushing problem out to morality "what should Sleeping Beauty answer so that she doesn't feel as if she's lying" doesn't solve anything either; rather, it feels like asking question "is continuum hypothesis true?" providing only options 'true' and 'false', while it's actually independent of ZFC axioms (claims of it or of its negation produce different models, neither proven to self-contradict).
P.S. One more analogue: there's a field, and some people (experimenters) are asking whether it rained recently with clear intent to walk through if it didn't; you know it didn't rain but there are mines all over the field.
I argue you should mention the mines first ("that probability - which by the way will be 1/2 - can be found out, conforms to epistemology, but isn't directly usable anywhere") before saying if there was rain.
What exactly do you mean by "different tools need to be used"? Can you give me an example?
I mean that Beauty should maintain full model of experiment, and use decision theory as well as probability theory (if latter is even useful, which it admittedly seems to be). If she didn't keep track of full setup but only "a fair coin was flipped, so the odds are 1:1", she would predictably lose when betting on the coin outcome.
Also, I've minted another "paradox" version. I can predict you'll take issue with one of formulations in it, but what do you think about it?
A fair coin is flipped, hidden from you.
On Heads, you're waken up on Monday, asked "what credence do you have that coin landed Heads?"; on Tuesday, you're let go.
If coin landed Tails, you're waken up on Monday and still asked "what credence do you have that coin landed Heads?"; then, with no memory erasure, you're waken up on Tuesday, and experimenter says to you: "Name the credence that coin landed Heads, but you must name the exact same number as yesterday". Afterwards, you're let go.
If you don't follow experiment protocol, you lose/lose out on some reward.
Have you tried asking the kid? Perhaps they already have perspective matching the real world on some topic.
There is, in the left panel under "Concepts" link. https://www.lesswrong.com/tags/all.
I'd guess you haven't seen "Rationality: A-Z" (also called "The Sequences") from that site panel as well. It builds up many of local concepts from certain starting points over course of many essays, gradually increasing level of material. For a newcomer, it is rather common to be linked to some essay there upon some marker statements (for instance, "that hypothesis is improbable, but its alternatives are no better so I keep 'my' one!")
Upon rereading your posts, I retract disagreement on "mutually exclusive outcomes". Instead...
Initially probability function is defined over iterations of probability experiment. You define a different function over all space and time, which you still call "probability". It surely has properties that you like, but it's a different function! Please use another name, this is already taken. Or add a disclaimer. Preferably do both. You know how easy it is to confuse people with such things! Definetely, do not start participating in the conversations about probability while talking about your function.
An obvious way to do so is put a hazard sign on "probability" and just not use it, not putting resources into figuring out what "probability" SB should name, isn't it? For instance, suppose Sleeping Beauty claims "my credence for Tails is "; any specific objection would be based on what you expected to hear.
(And now I realize a possible point why you're arguing to keep "probability" term for such scenarios well-defined; so that people in ~anthropic settings can tell you their probability estimates and you, being observer, could update on that information.)
As for why I believe probability theory to be useful in life despite the fact that sometimes different tools need to be used: I believe disappearing as a Boltzmann brain or simulated person is balanced out by appearing the same way, dissolving into different quantum branches is balanced out by branches reassembling, and likewise for most processes.
For the record, there is still some valid, good-faith scientific research out there, but most of the academic “research” produced in more recent decades is either fabricated, dishonest, plagiarized, redundant, outdated, and/or useless.
This claim could be restated as: most of the academic "research" is either false because authors did not intend to be honest (or conformed to unrelated biases), or false because authors did not have more accurate data (this kind of research becomes superseded over time, when more belief depth is acquired). This might be true.
However, I don't know what redundant articles do in that list; I suppose you're claiming more articles stating the same point do not provide more evidence to it, but replication and more experiments in good faith are always good.
To say that “most” academic research is “fake” also implies that we can quantify how much of it is fake or not. I can’t precisely estimate, quantify, and judge every academic paper that gets published out there, so I don’t claim to know exactly how much of the current research being published is reliable. It probably also varies by fields, and it’s possible for papers to include a mixture of true and fake data, reasoning, and conclusions.
And thereon the essay goes to saying "most" without any description what "research" is taken as a sample set. Hastings' comment, on the other hand, suggested some alternatives:
Uniform measure is not going to be exciting – you’re going to get almost entirely undergraduate assignments and Third World paper mills. If your weighted sampler is “papers linked in articles about how academia is woke” you’re going to find a high %fake. If your weighed measure is “papers read during work hours by employees at F500 companies” you’ll find a lower, nonzero %fake.
However, we can generally say that most recent research in humanities (or human-centric sciences) is fake, redundant, or useless; most research in earth-centric sciences is true, fake, or questionable; and most research in STEM fields is true, fake, outdated, or redundant.
Major nitpicking here. If most true claims of human-centric studies are replicated, then each of the corresponding papers are redundant (as it'd have a duplicate); therefore, almost all research would be "fake, redundant, or useless". Moreover, for STEM fields "true, fake, outdated, or redundant" seems to describe universal set - that is, that statement is of no substance. I'd suggest clarifying what claims you had in mind, if you are not using them for emphasis only.
The best rule is not to assume that because an academic paper says X, that X is true.
The best rule known to us - i.e. Bayesian reasoning - mandates that we simply treat "research" as stream of characters, and assess probabilities of each stream being shown to you if X were true and if X were false. That is intractable; after some fallback, you get at "correct for authors' biases, and assume that paper's claims represent average of what happens". I have the impression LessWrong does pretty much that.
Academic research can be fake in different ways. It can simply be false. It can be emotionally manipulative propaganda masquerading as knowledge. It can be irrelevant or meaningless.
Specific claim being true or false necessarily screens off being "emotionally manipulative propaganda". A weaker point that would stand, though: "often papers are emotionally manipulative, even when the claims presented in them are inappicable to most real situations or meaningless outside of academia".
I believe the further parts of Sections 1 and 2 are not of much interest for LessWrong, except that they attempt establishing common knowledge that academic "research" is commonly fake. Section 3, with specific suggestions, could be positively received when posted separately.