Posts
Comments
I do feel like it would be good to start with a more optimistic prior on new posts. Over the last year, the mean post karma was a little over 13, and the median was 5.
This seems unlikely to satisfy linearity, as A/B + C/D is not equal to (A+C)/(B+D)
I don't feel particularly uncertain. This EA Forum comment and its parents inform my view quite a bit.
Maybe sometimes a team will die in the dungeon?
<details>blah blah</details>
So I did some super dumb modelling.
I was like: let's assume that there aren't interaction effects between the encounters either in the difficulty along a path or in the tendency to co-occur. And let's assume position doesn't matter. Let's also assume that the adventurers choose the minimally difficult path, only moving across room edges.
To estimate the value of an encounter, let's look at how the dungeons where it occurs in one of the two unavoidable locations (1 and 9) differ on average from the overall average.
Assuming ChatGPT did all the implementation correctly, this predictor never overestimates the score by much. Though it frequently, and sometimes egregiously, underestimates the score.
Anyway, using this model and this pathing assumption, we have DBN/OWH/NOC
We skip the goblins and put our fairly rubbish trap in the middle to stop adventurers picking and choosing which parts of the outside paths they take. The optimal path for the adventurers is DONOC, which has a predicted score of 30.29, which ChatGPT tells me is ~95th percentile.
I'd love to come at this with saner modelling (especially of adventurer behaviour), but I somewhat doubt I will.
I'm guessing encounter 4 (rather than encounter 6) follows encounter 3?
You can simulate a future by short-selling the underlying security and buying a bond with the revenue. You can simulate short-selling the same future by borrowing money (selling a bond) and using the money to buy the underlying security.
I think these are backwards. At the end of your simulated future, you end up with one less of the stock, but you have k extra cash. At the end of your simulated short sell, you end up with one extra of the stock and k less cash.
A neat stylised fact, if it's true. It would be cool to see people checking it in more domains.
I appreciate that Ege included all of examples, theory, and predictions of the theory. I think there's lots of room for criticism of this model, which it would be cool to see tried. In particular, as far as I understand the formalism, it doesn't seem like it is obviously discussing the costs of the investments, as opposed to their returns.
But I still like this as a rule of thumb (open to revision).
I still think this post is cool. Ultimately, I don't think the evidence presented here bares that strongly on the underlying question: "can humans get AIs to do their alignment homework?". But I think it bares on it at all, and was conducted quickly and competently.
I would like to live in a world where lots of people gather lots of weak pieces of evidence on important questions.
Yep, if the first vote takes the score to ≤ 0, then the post will be dropped off the latest list. This is somewhat ameliorated by:
(a) a fair number of people browsing https://lesswrong.com/allPosts
(b) https://greaterwrong.com having chronological sort by default
(c) posts appearing in recent discussion in order that they're posted (though I do wonder if we filter out negative karma posts from recent discussion)
I often play around with different karma / sorting mechanisms, and I do think it would be nice to have a more Bayesian approach that started with a stronger prior. My guess is the effect you're talking about isn't a big issue in practice, though probably worth a bit of my time to sample some negative karma posts.
I had a quick look in the database, and you do have some tag filters set, which could cause the behaviour you describe
- Because it's a number and a vector, you're unlikely to see anyone (other than programmers) trying to use i as a variable.
I think it's quite common to use i as index variable (for example, in a sum)
(edit: whoops, I see several people have mentioned this)
In this case sitting down with someone doing similar tasks but getting more use out of LMs would likely help.
I would contribute to a bounty for y'all to do this. I would like to know whether the slow progress is prompting-induced or not.
Click on the gear icon next to the feed selector
A quick question re: your list: do you have any tag filters set?
I think "unacceptable reputational costs" here basically means the conjunction of "Dustin doesn't like the work" and "incurs reputational costs for Dustin". Because of the first conjunct, I don't think this suggestion would help Lightcone, sadly.
The "latest" tab works via the hacker news algorithm. Ruby has a footnote about it here. I think we set the "starting age" to 2 hours, and the power for the decay rate to 1.15.
mod note: this post used to say "LessWrong doesn't seem to support the <details> element, otherwise I would put this code block in it".
We do now support it, so I've edited the post to put the code block in such an element
Robin Hanson is one of the intellectual fathers of LessWrong, and I'm very glad there's a curated, organised list of some of his main themes.
He's the first thinker I remember reading and thinking "what? that's completely wrong", who went on to have a big influence on my thought. Apparently I'm not an isolated case (paragraph 3 page 94).
Thanks, Arjun and Richard.
37bvhXnjRz4hipURrq2EMAXN2w6xproa9T
I've updated the post with it.
FTX did successfully retrieve the $1M from the title company! We didn't have any control over those funds, so I don't think we were involved apart from pointing FTX in the right direction.
Habryka means we would have to pick one number per Stripe link (eg one like for $5/month, 1 for $100/month, etc)
Are you checking the box for “Save my info for 1-click checkout with Link”? That’s the only way I’ve figured out get Stripe to ask for my phone number. If so, you can safely uncheck that
(Also, I don’t know if it’s important you, but I don’t think we would see your phone number if you gave it to Stripe)
What do you mean by A?
Habryka is slightly sloppily referring to using Janus' 'base model jailbreak' for Claude 3.5 Sonnet
as I understand it, the majority of this money will go towards supporting Lighthaven
I think if you take Habryka's numbers at face value, a hair under half of the money this year will go to Lighthaven (35% of core staff salaries@1.4M = 0.49M. 1M for a deferred interest payment. And then the claim that otherwise Lighthaven is breaking even). And in future years, well less than half.
I worry that the future of LW will be endangered by the financial burden of Lighthaven
I think this is a reasonable worry, but I again want to note that Habryka is projecting a neutral or positive cashflow from Lighthaven to the org.
That said, I can think of a couple of reasons for financial pessimism[1]. Having Lighthaven is riskier. It involves a bunch of hard-to-avoid costs. So, if Lighthaven has a bad year, that does indeed endanger the project as a whole.
Another reason to be worried: Lightcone might stop trying to make Lighthaven break even. Lightcone is currently fairly focused on using Lighthaven in revenue-producing ways. My guess is that we'll always try and structure stuff at Lighthaven such that it pays its own way (for example, when we ran LessOnline we sold tickets[2]). But maybe not! Maybe Lightcone will pivot Lighthaven to a loss-making plan, because it foresees greater altruistic benefit (and expects to be able to fundraise to cover it).
So the bundling of the two projects still leaks some risk.
Of course, you might also think Lighthaven makes LessWrong more financially robust, if on the mainline it ends up producing a modest profit that can be used to subsidise LessWrong.
- ^
Other than just doubting Habryka's projections, which also might make sense.
- ^
My understanding of the numbers is that we lost money once you take into account staff time, but broke even if you don't. And it seems the people most involved with running it are hopeful about cutting a bunch of costs in future.
I worry that cos this hasn't received a reply in a bit, people might think it's not in the spirit of the post. I'm even more worried people might think that critical comments aren't in the spirit of the post.
Both critical comments and high-effort-demanding questions are in the spirit of the post, IMO! But the latter might take awhile to get a response
The EIN is 92-0861538
My impression matches your initial one, to be clear. Like my point estimate of the median is like 85%, my confidence only extends to >50%
Lightcone is also heterogeneous, but I think it's accurate that the median view at Lightcone is >50% on misaligned takeover
Maybe remove decimal numbers entirely throughout the graphs? This is what it looked like for me, and led to the error. And this image is way zoomed in compared to what I see naturally on my screen.
Good idea. Done.
Yes, I think you're right. I was confused by Shoshannah's numbers last night, but it was late and I didn't manage to summon enough sapience to realise something was wrong and offer a correction. Thanks for doing that!
as applied to them
A slight nitpick: I think this treats their like of the activity applied to them as a scalar, but I think it's also plausibly a function of how you, the applier, go about it. Like maybe they are very in to this activity as applied to them, but not in the way you do it.
If I had to pick a favourite, I'd probably go for Fire and AIs, but The GPT is also great: very terrifying sublime
Perhaps say some words on why they might want to?
Curated! A short and sweet post, with a useful rule of thumb for those who live and die by the Fermi. There are some more good ones in the comments, too!
I don't think this distinction is robust enough to rely on as much of a defensive property. I think it's probably not that hard to think "I probably would have tried something in direction X, or direction Y", and then gather lots of bits about how well the clusters X and Y work.
I'm curious about Claude's coauthorship of this post. How did it help?
The crucial point is: when the untrusted model gets to step 6, it sees R in the history instead of A, so it knows its attempt was replaced.
Why does it know its attempt was replaced? Because it can guess it wouldn't have produced R? If it can tell that, how come it can't guess it specifically would have produced A?
I thought this title meant the post would be making a case from conservative (i.e. minimal) assumptions.
Maybe change the title to "making a politically conservative case for alignment" or something?
I wonder what the lifetime spend on dating apps is. I expect that for most people who ever pay it's >$100
I think the credit assignment is legit hard, rather than just being a case of bad norms. Do you disagree?
I would guess they tried it because they hoped it would be competitive with their other product, and sunset it because that didn't happen with the amount of energy they wanted to allocate to the bet. There may also have been an element of updating more about how much focus their core product needed.
I only skimmed the retrospective now, but it seems mostly to be detailing problems that stymied their ability to find traction.
It's possible no one tried literally "recreate OkC", but I think dating startups are very oversubscribed by founders, relative to interest from VCs [1] [2] [3] (and I think VCs are mostly correct that they won't make money [4] [5]).
(Edit: I want to note that those are things I found after a bit of googling to see if my sense of the consensus was borne out; they are meant in the spirit of "several samples of weak evidence")
I don't particularly believe you that OkC solves dating for a significant fraction of people. IIRC, a previous time we talked about this, @romeostevensit suggested you had not sufficiently internalised the OkCupid blog findings about how much people prioritised physical attraction.
You mention manifold.love, but also mention it's in maintenance mode – I think because the type of business you want people to build does not in fact work.
I think it's fine to lament our lack of good mechanisms for public good provision, and claim our society is failing at that. But I think you're trying to draw an update that's something like "tech startups should be doing an unbiased search through viable valuable business, but they're clearly not", or maybe, "tech startups are supposed to be able to solve a large fraction of our problems, but if they can't solve this, then that's not true", and I don't think either of these conclusions seem that licensed from the dating data point.
Yes, though I'm not confident.
I saw this poll and thought to myself "gosh, politics, religion and cultural opinions sure are areas where I actively try to be non-heroic, as they aren't where I wish to spend my energy".
They load it in as a web font (i.e. you load Calibri from their server when you load that search page). We don't do that on LessWrong
Yeah, that's a google Easter Egg. You can also try "Comic Sans" or "Trebuchet MS".
One sad thing about older versions of Gill Sans: Il1 all look the same. Nova at least distinguishes the 1.
IMO, we should probably move towards system fonts, though I would like to choose something that preserves character a little more.