Vote on worthwhile OpenAI topics to discuss 2023-11-21T00:03:03.898Z
New LessWrong feature: Dialogue Matching 2023-11-16T21:27:16.763Z
Does davidad's uploading moonshot work? 2023-11-03T02:21:51.720Z
Holly Elmore and Rob Miles dialogue on AI Safety Advocacy 2023-10-20T21:04:32.645Z
How to partition teams to move fast? Debating "low-dimensional cuts" 2023-10-13T21:43:53.067Z
Thomas Kwa's MIRI research experience 2023-10-02T16:42:37.886Z
Feedback-loops, Deliberate Practice, and Transfer Learning 2023-09-07T01:57:33.066Z
A Golden Age of Building? Excerpts and lessons from Empire State, Pentagon, Skunk Works and SpaceX 2023-09-01T04:03:41.067Z
Consider applying to a 2-week alignment project with former GitHub CEO 2023-04-04T06:20:49.532Z
How I buy things when Lightcone wants them fast 2022-09-26T05:02:09.003Z
How my team at Lightcone sometimes gets stuff done 2022-09-19T05:47:06.787Z
($1000 bounty) How effective are marginal vaccine doses against the covid delta variant? 2021-07-22T01:26:26.117Z
What other peptide vaccines might it be useful to make? 2021-03-03T06:25:40.130Z
Credence polls for 26 claims from the 2019 Review 2021-01-09T07:13:24.166Z
Weekend Review Bash: Guided review writing, Forecasting and co-working, in EU and US times 2021-01-08T21:04:12.332Z
Thread for making 2019 Review accountability commitments 2020-12-18T05:07:25.533Z
Which sources do you trust the most on nutrition advice for exercise? 2020-12-16T03:22:40.088Z
The LessWrong 2018 Book is Available for Pre-order 2020-12-01T08:00:00.000Z
Why is there a "clogged drainpipe" effect in idea generation? 2020-11-20T19:08:08.461Z
Final Babble Challenge (for now): 100 ways to light a candle 2020-11-12T23:17:07.790Z
Babble Challenge: 50 thoughts on stable, cooperative institutions 2020-11-05T06:38:38.997Z
Babble challenge: 50 consequences of intelligent ant colonies 2020-10-29T07:21:33.379Z
Babble challenge: 50 ways of solving a problem in your life 2020-10-22T04:49:42.661Z
What are some beautiful, rationalist artworks? 2020-10-17T06:32:43.142Z
Babble challenge: 50 ways of hiding Einstein's pen for fifty years 2020-10-15T07:23:48.541Z
Babble challenge: 50 ways to escape a locked room 2020-10-08T05:13:06.985Z
Babble challenge: 50 ways of sending something to the moon 2020-10-01T04:20:24.016Z
Sunday August 16, 12pm (PDT) — talks by Ozzie Gooen, habryka, Ben Pace 2020-08-14T18:32:35.378Z
Sunday August 9, 1pm (PDT) — talks by elityre, jacobjacob, Ruby 2020-08-06T22:50:21.550Z
Sunday August 2, 12pm (PDT) — talks by jimrandomh, johnswenthworth, Daniel Filan, Jacobian 2020-07-30T23:55:44.712Z
$1000 bounty for OpenAI to show whether GPT3 was "deliberately" pretending to be stupider than it is 2020-07-21T18:42:44.704Z
Lessons on AI Takeover from the conquistadors 2020-07-17T22:35:32.265Z
Meta-preferences are weird 2020-07-16T23:03:40.226Z
Sunday July 19, 1pm (PDT) — talks by Raemon, ricraz, mr-hire, Jameson Quinn 2020-07-16T20:04:37.974Z
Mazes and Duality 2020-07-14T19:54:42.479Z
Sunday July 12 — talks by Scott Garrabrant, Alexflint, alexei, Stuart_Armstrong 2020-07-08T00:27:57.876Z
Public Positions and Private Guts 2020-06-26T23:00:52.838Z
Missing dog reasoning 2020-06-26T21:30:00.491Z
Sunday June 28 – talks by johnswentworth, Daniel kokotajlo, Charlie Steiner, TurnTrout 2020-06-26T19:13:23.754Z - A Petition 2020-06-25T05:44:50.050Z
Sunday June 21st – talks by Abram Demski, alkjash, orthonormal, eukaryote, Vaniver 2020-06-18T20:10:38.978Z
FHI paper on COVID-19 government countermeasures 2020-06-04T21:06:51.287Z
[Job ad] Lead an ambitious COVID-19 forecasting project [Deadline extended: June 10th] 2020-05-27T16:38:04.084Z
Crisis and opportunity during coronavirus 2020-03-12T20:20:55.703Z
[Link] Beyond the hill: thoughts on ontologies for thinking, essay-completeness and forecasting 2020-02-02T12:39:06.563Z
[Part 1] Amplifying generalist research via forecasting – Models of impact and challenges 2019-12-19T15:50:33.412Z
[Part 2] Amplifying generalist research via forecasting – results from a preliminary exploration 2019-12-19T15:49:45.901Z
Running Effective Structured Forecasting Sessions 2019-09-06T21:30:25.829Z
How to write good AI forecasting questions + Question Database (Forecasting infrastructure, part 3) 2019-09-03T14:50:59.288Z
AI Forecasting Resolution Council (Forecasting infrastructure, part 2) 2019-08-29T17:35:26.962Z


Comment by jacobjacob on Dialogue on the Claim: "OpenAI's Firing of Sam Altman (And Shortly-Subsequent Events) On Net Reduced Existential Risk From AGI" · 2023-11-21T19:44:32.066Z · LW · GW

In the poll most people (31) disagreed with the claim John is defending here, but I'm tagging the additional few (3) who agreed with it @Charlie Steiner @Oliver Sourbut @Thane Ruthenis 

Interested to hear your guys' reasons, in addition to John's above! 

Comment by jacobjacob on Dialogue on the Claim: "OpenAI's Firing of Sam Altman (And Shortly-Subsequent Events) On Net Reduced Existential Risk From AGI" · 2023-11-21T19:03:28.406Z · LW · GW

One of my takeaways of how the negotiations went is that it seems sama is extremely concerned with securing access to lots of compute, and that the person who ultimately got their way was the person who sat on the compute.

The "sama running Microsoft" idea seems a bit magical to me. Surely the realpolitik update here should be: power lies in the hands of those with legal voting power, and those controlling the compute. Sama has neither of those things at Microsoft. If he can be fired by a board most people have never heard of, then for sure he can get fired by the CEO of Microsoft. 

People seem to think he is somehow a linchpin of building AGI. Remind me... how many of OpenAI's key papers did he coauthor? Paul Graham says if you dropped him into an island of cannibals he would be king in 5 years. Seems plausible. Paul Graham did not say he would've figured out how to engineer a raft good enough to get him out of there. If there were any Manifold markets on "Sama is the linchpin to building AGI", I would short them for sure. 

We already have strong suspicion from the open letter vote counts there's a personality cult around Sama at OpenAI (no democratic election ever ends with a vote of 97% in favor). It also makes sense people in the LessWrong sphere would view AGI as the central thing to the future of the world and on everyone's minds, and thus fall in the trap of also viewing Sama as the most important thing at Microsoft. (Question to ask yourself about such a belief: who does it benefit? And is that beneficiary also a powerful agent deliberately attempting to shape narratives to their own benefit?) 

Satya Nadella might have a very different perspective than that, on what's important for Microsoft and who's running it.

Comment by jacobjacob on Vote on worthwhile OpenAI topics to discuss · 2023-11-21T00:07:19.469Z · LW · GW

It would be a promising move, to reduce existential risk, for Anthropic to take over what will remain of OpenAI and consolidate efforts into a single project. 

Comment by jacobjacob on Vote on worthwhile OpenAI topics to discuss · 2023-11-21T00:04:25.779Z · LW · GW

EAs need to aggressively recruit and fund additional ambitious Sam's, to ensure there's one to sacrifice for Samsgiving November 2024. 

Comment by jacobjacob on Vote on worthwhile OpenAI topics to discuss · 2023-11-21T00:00:30.045Z · LW · GW

New leadership should shut down OpenAI. 

Comment by jacobjacob on Vote on worthwhile OpenAI topics to discuss · 2023-11-20T23:58:45.742Z · LW · GW

If there was actually a spooky capabilities advance that convinced the board that drastic action was needed, then the board's actions were on net justified, regardless of what other dynamics were at play and whether cooperative principles were followed.

Comment by jacobjacob on Vote on worthwhile OpenAI topics to discuss · 2023-11-20T23:53:54.954Z · LW · GW

Open-ended: A dialogue between an OpenAI employee who signed the open letter, and someone outside opposed to the open letter, about their reasoning and the options. 

(Up/down-vote if you're interested in reading discussion of this. React paperclip if you have an opinion and would be up for dialoguing)

Comment by jacobjacob on Vote on worthwhile OpenAI topics to discuss · 2023-11-20T23:49:54.238Z · LW · GW

If the board did not abide by cooperative principles in the firing nor acted on substantial evidence to warrant the firing in line with the charter, and nonetheless were largely EA motivated, then EA should be disavowed and dismantled. 

Comment by jacobjacob on Vote on worthwhile OpenAI topics to discuss · 2023-11-20T23:42:17.200Z · LW · GW

The events of the OpenAI board CEO-ousting on net reduced existential risk from AGI.

Comment by jacobjacob on Vote on worthwhile OpenAI topics to discuss · 2023-11-20T21:50:19.517Z · LW · GW

Open-ended: If >50% of employees end up staying at OpenAI: how, if at all, should OpenAI change its structure and direction going forwards? 

(Up/down-vote if you're interested in reading discussion of this. React paperclip if you have an opinion and would be up for discussing)

Comment by jacobjacob on Vote on worthwhile OpenAI topics to discuss · 2023-11-20T21:48:37.309Z · LW · GW

Open-ended: If >90% of employees leave OpenAI: what plan should Emmett Shear set for OpenAI going forwards? 

(Up/down-vote if you're interested in reading discussion of this. React paperclip if you have an opinion and would be up for discussing)

Comment by jacobjacob on Vote on worthwhile OpenAI topics to discuss · 2023-11-20T21:44:40.911Z · LW · GW

It is important that the board release another public statement explaining their actions, and providing any key pieces of evidence. 

Comment by jacobjacob on New LessWrong feature: Dialogue Matching · 2023-11-18T03:19:48.429Z · LW · GW

Yeah I'm gonna ship a fix to that now. No more monologues!  

Comment by jacobjacob on New LessWrong feature: Dialogue Matching · 2023-11-17T20:29:29.071Z · LW · GW

(If others want this too, upvote @faul_sname's comment as a vote! It would be easy to build, most of my uncertainty is in how it would change the experience)

Comment by jacobjacob on New LessWrong feature: Dialogue Matching · 2023-11-17T18:48:56.231Z · LW · GW

Those are some interesting papers, thanks for linking. 

In the case at hand, I do disagree with your conclusion though. 

In this situation, the most a user could find out is who checked them in dialogues. They wouldn't be able to find any data about checks not concerning themselves. 

If they happened to be a capable enough dev and were willing to go through the schleps to obtain that information, then, well... we're a small team and the world is on fire, and I don't think we should really be prioritising making Dialogue Matching robust to this kind of adversarial cyber threat for information of comparable scope and sensitivity! Folks with those resources could probably uncover all kinds of private vote data already, if they wanted to

Comment by jacobjacob on New LessWrong feature: Dialogue Matching · 2023-11-16T20:16:14.244Z · LW · GW

On data privacy

Here's some quick notes on how I think of LessWrong user data. 

Any data that's already public -- reacts, tags, comments, etc -- is fair game. It just seems nice to do some data science and help folks uncover interesting patterns here. 

On the other side of the spectrum, me and the team generally never look at users' up and downvotes, except in cases where there's strong enough suspicion of malicious voting behavior (like targeted mass downvoting). 

Then there's stuff in the middle.  Like, what if we tell a user "you and this user frequently upvote each other"? That particular example currently feels like it reveals too much private data.  As another example, the other day me and a teammate had a discussion of whether, on the matchmaking page, we could show people recently active users who already checked you, to make it more likely you'd find a match. We tenatively postulated it would be fine to do this as long as seeing a name on your match page gave no more than like a 5:1 update about those people having checked you. We sketched out some algorithms to implement this, that would also be stable under repeated refreshing and similar. (We haven't implemented the algorithm nor the feature yet.)

So my general take on features "in the middle" is for now to treat them on a case by case basis, with some principles like "try hard to avoid revealing anything that's not already public, and if doing so, try to leave plausible deniability bounded by some number of leaked bits, only reveal metadata or aggregate data, reveal it only to one other or a smaller set of users, think about whether this is actually a piece of info that seems high or low stakes, and see if you can get away with just using data from people who opted in to revealing it". 

Comment by jacobjacob on 'Theories of Values' and 'Theories of Agents': confusions, musings and desiderata · 2023-11-15T20:56:53.283Z · LW · GW

I can't quite tell how that's different from embeddedness. (Also if you have links to other places it's explained feel free to share them.)

Comment by jacobjacob on 'Theories of Values' and 'Theories of Agents': confusions, musings and desiderata · 2023-11-15T19:55:04.135Z · LW · GW

bounded, embedded, enactive, nested.

I know about boundedness, embededness, and I guess nestedness is about hierarchical agents. 

But what's enactive?

Comment by jacobjacob on A bet on critical periods in neural networks · 2023-11-06T23:39:36.776Z · LW · GW
Comment by jacobjacob on Does davidad's uploading moonshot work? · 2023-11-04T03:16:54.723Z · LW · GW

Space flight doesn't involve a 100 percent chance of physical death

I think historically folks have gone to war or on other kinds of missions that had death rates of like, at least, 50%. And folks, I dunno, climb Mount Everest, or figured out how to fly planes before they could figure out how to make them safe. 

Some of them were for sure fanatics or lunatics. But I guess I also think there's just great, sane, and in many ways whole, people, who care about things greater than their own personal life and death, and are psychologically consituted to be willing to pursue those greater things. 

Comment by jacobjacob on Does davidad's uploading moonshot work? · 2023-11-04T03:08:36.281Z · LW · GW

Hm, here's a test case: 

GPT4 can't solve IMO problems. Now take an IMO gold medalist about to walk into their exam, and upload them at that state into an Em without synaptic plasticity. Would the resulting upload would still be able to solve the exam at a similar level as the full human?

I don't have a strong belief, but my intuition is that they would. I recall once chatting to @Neel Nanda  about how he solved problems (as he is in fact an IMO gold winner), and recall him describing something that to me sounded like "introspecting really hard and having the answers just suddenly 'appear'..." (though hopefully he can correct that butchered impression)

Do you think such a student Em would or would not perform similarly well in the exam? 

Comment by jacobjacob on Does davidad's uploading moonshot work? · 2023-11-04T02:54:18.354Z · LW · GW

I have an important appointment this weekend that will take up most of my time, but hope to come back to this after that, but wanted to quickly note: 

but definitely are not back propagation.


Last time I looked into this 6 years ago seemed like an open question and it could plausibly be backprop or at least close enough:

3yrs ago Daniel Kokotajlo shared some further updates in that direction:

Comment by jacobjacob on Does davidad's uploading moonshot work? · 2023-11-03T20:56:58.667Z · LW · GW

Separately, I'm kind of awed by the idea of an "uploadonaut": the best and brightest of this young civilisation, undergoing extensive mental and research training to have their minds able to deal with what they might experience post upload, and then courageously setting out on a dangerous mission of crucial importance for humanity.

(I tried generating some Dall-E 1960's style NASA recruitment posters for this, but they didn't come out great. Might try more later)

Comment by jacobjacob on Does davidad's uploading moonshot work? · 2023-11-03T20:44:11.878Z · LW · GW

Noting that I gave this a weak downvote as I found this comment to be stating many strong claims but without correspondingly strong (or sometimes not really any) arguments. I am still interested in the reasons you believe these things though (for example, like a fermi on inferece cost at runtime). 

Comment by jacobjacob on Does davidad's uploading moonshot work? · 2023-11-03T20:36:51.504Z · LW · GW

I don't think you're going to get a lot of volunteers for destructive uploading (or actually even for nondestructive uploading). Especially not if the upload is going to be run with limited fidelity. Anybody who does volunteer is probably deeply atypical and potentially a dangerous fanatic.

Seems falsified by the existence of astronauts? 

Comment by jacobjacob on [deleted post] 2023-10-25T18:33:09.675Z

Comment by jacobjacob on Anthropic, Google, Microsoft & OpenAI announce Executive Director of the Frontier Model Forum & over $10 million for a new AI Safety Fund · 2023-10-25T18:09:30.665Z · LW · GW

Reference class: I'm old enough to remember the founding of the Partnership on AI. My sense from back in the day was that some (innocently misguided) folks wanted in their hearts for it to be an alignment collaboration vehicle. But I think it's decayed into some kind of epiphenomenal social justice thingy. (And for some reason they have 30 staff. I wonder what they all do all day.)

I hope Frontier Model Forum can be something better, but my hopes ain't my betting odds. 

Comment by jacobjacob on Book Review: Going Infinite · 2023-10-25T17:38:37.941Z · LW · GW

that the misalignment issues involved would have almost certainly destroyed us all, or all that we care about.


Comment by jacobjacob on AI #34: Chipping Away at Chip Exports · 2023-10-19T18:00:20.624Z · LW · GW

What played essentially no role in any of it, as far as I can tell, was AI.

One way I would expect it to play a role at this stage, would be armies of bot commenters on X and elsewhere, giving monkey brain the impression of broad support for a position. (Basically what Russia has been doing for ages, but now AI enabled.)

I haven't been able to tell whether or not that happened. Have you?

Comment by jacobjacob on Announcing Dialogues · 2023-10-18T20:53:29.681Z · LW · GW

lol, sure, invited you to a dialogue on that :)

Comment by jacobjacob on Daniel Kokotajlo's Shortform · 2023-10-18T19:49:11.340Z · LW · GW

Rootclaim is super cool, glad to finally see others mention it too! 

Comment by jacobjacob on How should TurnTrout handle his DeepMind equity situation? · 2023-10-17T17:57:56.979Z · LW · GW

Hm, I was a bit confused reading this. My impression was "seems like there are multiple viable solutions", but then they were discarded for reasons that seemed kind of tangential, or not dealbreakers to me, where some extra fiddling might've done the trick? 

If I get the time later will write up more concretely why some of them still seemed promising. 

Comment by jacobjacob on Announcing Dialogues · 2023-10-16T20:19:37.049Z · LW · GW

We just made some Manifold markets on how dialogues will shape up:

And a similar market for arbitrage action:

And a meta-market for the fun of it :)

Comment by jacobjacob on Announcing Dialogues · 2023-10-07T02:59:40.329Z · LW · GW

If you don't want to put your questions in public, there's a form you can fill in, where only the lesswrong team sees your suggestions, and will do the matchmaking to connect you with someone only if it seems like a good mutual fit :) 

Here's the dialogue matchmaking form:

Comment by jacobjacob on Announcing Dialogues · 2023-10-07T02:59:23.614Z · LW · GW

I'm potentially up for another dialogue. 

I'm often very up for interviewing people (as I've done here and here) -- if I have some genuine interest in the topic, and if it seems like lesswrong readers would like to see the interview happen. So if people are looking for someone-to-help-you-get-your-thoughts-out, I might be down. (See also the interview request form.)

For interviewing folks, I'm interested in more stuff than I can list. I'll just check what others are into being interviewed about and if there's a match. 

For personally dialogueing, topics I might want to do (writing these out, quickly and roughly, rather than not at all!)

  • metaphors and isomorphisms between lean manufacturing <> functional programming <> maneuver warfare, as different operational philosophies in different domains that still seem to be "climbing the same mountain from different directions"
  • state space metaphors applied to operations. i have some thoughts here, for thinking about how effective teams and organisations function, drawing upon a bunch of concepts and abstractions that I've mostly found around writing by wentworth and some agent foundations stuff... I can't summarise it succinctly, but if someone is curious upon hearing this sentence, we could chat
  • strategically, what are technologies such that 1) in our timeline they will appear late (or too late) on the automation tree, 2) they will be blocking for accomplishing certain things, and 3) there's tractable work now for causing them to happen sooner? For example: will software and science automation progress to a point where we will be able to solve a hard problem like uploading, in a way that then leaves us blocked on something like "having 1000 super-microscopes"? And if so, should someone just go try to build those microscopes now? Are there are other examples like this? 
  • I like flying and would dialogue about it :) 
  • enumerative safety sounds kind of bonkers... but what if it isn't? And beyond that, what kind of other ambitious, automated, alignment experiments would it be good if someone tried? 
  • Cyborg interfaces. What are natural and/or powerful ways of exploring latent space? I've been thinking about sort of mindlessly taking some interfaces over which I have some command -- a guitar, a piano, a stick-and-rudder -- and hooking them up something allowing me to "play latent space" or "cruise through latent space". What other metaphors are there here? What other interfaces might be cool to play around with? 
Comment by jacobjacob on Related Discussion from Thomas Kwa's MIRI Research Experience · 2023-10-05T23:08:45.712Z · LW · GW

So if some boss often drove his employees to tears, as long as he was pretty insightful, you don't think that the employees should be able to know before taking the job? Surely that's not your position. But then what is? 

I wanted to add a perspective to the conversation that I didn't see mentioned, moreso than advocating a very thought out position. I have conflicting intuitions, and the territory seems messy! 

On the one hand, it does seem to me like there should be some kind of "heads up about intensity". It's real bad to create hidden slippery slopes along the intensity scale. It's real bad to first make people dependent on you (by, say, paying most of their salary in yet-to-be-vested equity, making them work long enough that they can't explore external opportunities and maintain outside friends, ...) and then shifting into a potentially abusive stance (heavily frame controlling, demoralising, etc). It is when these kinds of pressures are applied that I think things move into unacceptable territory. (And my suggested community response would probably be something like "Sandbox culprit in ways where they're able to remain highly productive while doing less damage, give people accurate indications about their style (conveying this might actually fall on someone else than the culprit to do -- that division of labor might just be our only way to get all the good stuff here!), and avoid giving people inaccurate impressions or being a wide-eyed feeder school."

For comparison, when I imagine pursuing a career in investment banking, it seems like I'd be opting into a shark tank. I'm just kind of accepting there'll be some real abusive folks around, following the $$$, and I'll be betting on my ability to navigate that without losing myself in the process. Being part of a healthy community means somehow having people around me who can help me see these things. I do think there are some young undergrads who naively will believe the faceless Goldman ads. I feel like Taleb would have a word for them -- the "sucker" or the "Intellectual Yet Idiot". They'll get hurt, and this is bad, and the recruiting ads that led them into this are immoral. 

(From that perspective, I'm pretty into my straw version of military ads, which is more like "You'll have the worst time of your life and be tested to your limits. You're too weak for this. But you'll gain glory. Sign up here.")

On the other hand, I also have the intuition that requesting of individual researchers that they signpost and warn about their unusual communication style seems to be locating the onus of this in the wrong location... and I kind of just don't expect it to work, empirically? I feel like the getting-a-job-at-MIRI pipeline should somehow make it clear to people what level of f*ckery is about to happen to them, insofar as it is. I currently don't know whose responsibility I think that is (and I'm shipping this comment in a confused state, rather than not shipping it at all). 

Comment by jacobjacob on Thomas Kwa's MIRI research experience · 2023-10-05T17:45:56.150Z · LW · GW

Curated! (And here's the late curation notice) 

I don’t really know what I think about retrospectives in general, and I don’t always find them that helpful, because causal attribution is hard. Nonetheless, here are some reasons why I wanted to curate this:

  • I like that it both covers kind of a broad spectrum of stuff that influences a research project, and also manages to go into some interesting detail: high-level research direction and its feasibility, concrete sub-problems attempted and the outcome, particular cognitive and problem-solving strategies that were tried, as well as motivational and emotional aspects of the project. Hearing about what it was like when the various agent confusions collided with the various humans involved, was quite interesting and I feel like it actually gave me a somewhat richer view of both
  • It discusses some threads that seem important and that I’ve heard people talk about offline, but that I’ve seen less discussed online recently (the impact of infohazard policies, ambiguous lines between safety and capabilities research and how different inside views might cause one to pursue one rather than the other, people’s experience of interfacing with the MIRI_Nate way of doing research and communicating)
  • It holds different perspectives from the people involved in the research group, and I like how the dialogues feature kind of allows each to coexist without feeling a need to squeeze them into a unified narrative (the way things might feel if one were to coauthor a post or paper).

(Note: I chose to curate this, and I am also listed as a coauthor. I think this is fine because ultimately the impetus for writing up this content came from Thomas. Me and Raemon mostly just served as facilitators and interlocutors helping him get this stuff into writing.)

Comment by jacobjacob on Thomas Kwa's MIRI research experience · 2023-10-05T06:08:01.178Z · LW · GW

I did undergrad and grad school in neuroscience and can at the very least say that this was also my conclusion.

I remember the introductory lecture for the Cognitive Neuroscience course I took at Oxford. I won't mention the professor's name, because he's got his own lab and is all senior and stuff, and might not want his blunt view to be public -- but his take was "this field is 95% nonsense. I'll try to talk about the 5% that isn't". Here's a lecture slide:

Comment by jacobjacob on Related Discussion from Thomas Kwa's MIRI Research Experience · 2023-10-05T04:48:27.903Z · LW · GW

Wanted to briefly add a perspective I didn't see mentioned yet -- 

First -- seems like you had a particularly rough interaction, and I do want to express empathy for that. I feel like I recognise some of the things you point to, and think it's plausible that I might have been similarly demoralised by that situation, and that would really suck for me and I'd be really sad. So, genuinely sorry about that. I hope you'll find ways to regain motivation that was unfairly lost, and the ability to draw on insights that ended up involuntarily screened off from you. 

Second, the perspective I've come to hold for these situations is...  Basically the world does seem full of people who are extraordinarily productive in important ways, and who also... are kind of d*cks. (Important footnote: [1]

As such:

  • I think exceptional people are often sufficiently rare that, as things go, I'd rather take a bunch of productive d*cks than tune down their cognitive spikiness at the cost of mulling the productive peaks
  • I observe and am strategic about how I allocate my soul and motivation points to things. In the past I would kind of always pour full soul into things, but that led to a lot of sadness, because other people by default might not be able to hold things that are precious to me, and if I unilaterally pour it on them, they also really don't have a responsibility to hold it! Ouch. 
  • I try to satisfy different needs from different people. In various professional domains I'll be pretty thick skinned and put up with a lot of nonsense to extract interesting insights from people or get things done. Then with my partner or close friends I'll do a bunch of stuff that's emotionally nurturing and care a lot about holding aspects of each other's experience in ways that aren't rude. 
  • I beware of people employing dynamics that get inside and mess with my OODA loop, and have various allergic responses to this, and might often explicitly limit interactions, or hold the interaction in a particular way.[2] Regardless of whether they've advertised being unusual in this regard, I just kind of have a way of holding my guard up

I think holding this stance is my best strategy for getting around. Man, sometimes you gain so much great stuff from people who are rude, or weird, or norm-violating in various other ways, and I think "developing your own set of personal strategies that allow you to put up with stuff" can be a pretty decent superpower, judiciously deployed.

You say: 

if not, face the normal consequences for being rude, like 'proportional loss of social regard

So in light of the above, the way I orient to this would be something like: if someone is really killing it in terms of intellectual insight, or just getting important shit done -- that's the primary thing I care about around here (on LessWrong and the broader ecosystem). I'll try hard to carve a space for them to get those things out. If they're also a d*ck, I'll proably avoid inviting them to friendly gatherings I organise, and I might even just not have them work closely on my team specifically, because it'll mess too much with my OODA loop, and I want a certain culture. 

But I do not think they have a responsibility to proactively inform people about their style. 

On a community-wide level, the ratio of reward I'd give out for insight/productivity vs punishment for rudeness is like at least 30:1 or something, on some imaginary scale. I don't like rudeness and work best among people who are pretty empathetic and nurturing; but hey, the world is what it is, I'll take what I can get, and think this is the best point on the tradeoff curve. 

(And again, to reiterate, I do hope you also have or will find a way to orient to these things where you can gain the good insights + motivation, and avoid taking annoying hit points!)

  1. ^

    Note: I don't want to make any strong claims here about how insightful, or how much of a d*ck, this one particular empirical guy Nate is, with whom I have only interacted very little (though I like his blog posts!). Don't construe my comment as claiming that he is actually either of those things! 

  2. ^

    Though I do find OODA loop fuckery and frame control much worse than "rudeness" 

Comment by jacobjacob on The 5 Pillars of Happiness · 2023-10-04T17:54:51.735Z · LW · GW
  • Eating raw guinea pig organs


Comment by jacobjacob on Thomas Kwa's MIRI research experience · 2023-10-03T18:11:29.186Z · LW · GW

Reacted with "examples", but curious about examples/papers/etc both of things you think give lots of bits and things that don't. 

Comment by jacobjacob on Thomas Kwa's MIRI research experience · 2023-10-03T05:56:45.042Z · LW · GW

Flagging inconvenient acronym clash between SLT used for Sharp Left Turn and Singular Learning Theory (have seem both)!

Comment by jacobjacob on Related Discussion from Thomas Kwa's MIRI Research Experience · 2023-10-03T02:25:33.601Z · LW · GW

even send us a guide on communicating with him

Woah! That's like 10x more effort than I expect >90% of difficult-to-communicate-with people will go through. 

Kudos to Nate for that.

Comment by jacobjacob on How to Catch an AI Liar: Lie Detection in Black-Box LLMs by Asking Unrelated Questions · 2023-09-29T18:20:53.508Z · LW · GW

So, when a human lies over the course of an interaction, they'd be holding a hidden state in mind throughout. However, an LLM wouldn't carry any cognitive latent state over between telling the lie, and then responding to the elicitation question. I guess it feels more like "I just woke up from amnesia, and seems I have just told a lie. Okay, now what do I do..."

Stating this to:

  1. Verify that indeed this is how the paper works, and there's no particular way of passing latent state that I missed, and
  2. Any thoughts on how this affects the results and approach?
Comment by jacobjacob on Atoms to Agents Proto-Lectures · 2023-09-28T18:10:14.676Z · LW · GW
    • A program-like data structure is natural for representing locality + symmetry

Didn't quite get this from the lecture. For one, every rookie programmer has probably experienced that programs can work in ways with mysterious interactions that sure don't seem very local... but maybe in your case you'd still say that at the end of the day it would all just be unpackable into a graph of function calls, respecting locality at each step? 

Question: what's an example of a data structure very similar to program-like ones, while failing to respect locality + symmetry?

Comment by jacobjacob on ARENA 2.0 - Impact Report · 2023-09-26T17:38:44.615Z · LW · GW

I was only able to quickly skim this during my morning meeting, so might have missed a relevant point addressing this; but my first thought on seeing the results is "Sounds like you successfully trained a cohort of potential capabilities researchers"

Comment by jacobjacob on A Golden Age of Building? Excerpts and lessons from Empire State, Pentagon, Skunk Works and SpaceX · 2023-09-26T01:42:22.195Z · LW · GW

Making the building simple, with repeated components (the window example was a great one) is a better answer

Yeah... I was once working on a remodeling project, and had the "clever" idea that we could save time by only selectively demoing certain sections. "Tear down this wall, but leave this window-sill, and this doorframe looks good, leave that too, oh and maybe leave this section of drywall which looks fine"... 

Terrible idea. Crews got confused and paralyzed. I now believe it's much faster to just give clear and simple instructions -- "tear it all down to the studs". In the chaos and complexity of dealing with a building, simple instructions allow crews to move more independently and make their own decisions, and also makes it more feasible to deploy more labor (as it's easier to onboard and delegate). 

Comment by jacobjacob on A Golden Age of Building? Excerpts and lessons from Empire State, Pentagon, Skunk Works and SpaceX · 2023-09-21T17:46:13.820Z · LW · GW

Glad to hear :) If you've got any anecdotes about how the examples in the post similar/different to things you've seen in your job, I'd pretty curious

Comment by jacobjacob on [deleted post] 2023-09-21T15:28:33.836Z

Hello hello. 

Comment by jacobjacob on [deleted post] 2023-09-21T15:27:38.840Z

a comment