Posts

"Open Source AI" isn't Open Source 2024-02-15T08:59:59.034Z
Technologies and Terminology: AI isn't Software, it's... Deepware? 2024-02-13T13:37:10.364Z
Safe Stasis Fallacy 2024-02-05T10:54:44.061Z
AI Is Not Software 2024-01-02T07:58:04.992Z
Public Call for Interest in Mathematical Alignment 2023-11-22T13:22:09.558Z
What is autonomy, and how does it lead to greater risk from AI? 2023-08-01T07:58:06.366Z
A Defense of Work on Mathematical AI Safety 2023-07-06T14:15:21.074Z
"Safety Culture for AI" is important, but isn't going to be easy 2023-06-26T12:52:47.368Z
"LLMs Don't Have a Coherent Model of the World" - What it Means, Why it Matters 2023-06-01T07:46:37.075Z
Systems that cannot be unsafe cannot be safe 2023-05-02T08:53:35.115Z
Beyond a better world 2022-12-14T10:18:26.810Z
Far-UVC Light Update: No, LEDs are not around the corner (tweetstorm) 2022-11-02T12:57:23.445Z
Announcing AISIC 2022 - the AI Safety Israel Conference, October 19-20 2022-09-21T19:32:35.581Z
Rehovot, Israel – ACX Meetups Everywhere 2022 2022-08-25T18:01:16.106Z
AI Governance across Slow/Fast Takeoff and Easy/Hard Alignment spectra 2022-04-03T07:45:57.592Z
Arguments about Highly Reliable Agent Designs as a Useful Path to Artificial Intelligence Safety 2022-01-27T13:13:11.011Z
Elicitation for Modeling Transformative AI Risks 2021-12-16T15:24:04.926Z
Modelling Transformative AI Risks (MTAIR) Project: Introduction 2021-08-16T07:12:22.277Z
Maybe Antivirals aren’t a Useful Priority for Pandemics? 2021-06-20T10:04:08.425Z
A Cruciverbalist’s Introduction to Bayesian reasoning 2021-04-04T08:50:07.729Z
Systematizing Epistemics: Principles for Resolving Forecasts 2021-03-29T20:46:06.923Z
Resolutions to the Challenge of Resolving Forecasts 2021-03-11T19:08:16.290Z
The Upper Limit of Value 2021-01-27T14:13:09.510Z
Multitudinous outside views 2020-08-18T06:21:47.566Z
Update more slowly! 2020-07-13T07:10:50.164Z
A Personal (Interim) COVID-19 Postmortem 2020-06-25T18:10:40.885Z
Market-shaping approaches to accelerate COVID-19 response: a role for option-based guarantees? 2020-04-27T22:43:26.034Z
Potential High-Leverage and Inexpensive Mitigations (which are still feasible) for Pandemics 2020-03-09T06:59:19.610Z
Ineffective Response to COVID-19 and Risk Compensation 2020-03-08T09:21:55.888Z
Link: Does the following seem like a reasonable brief summary of the key disagreements regarding AI risk? 2019-12-26T20:14:52.509Z
Updating a Complex Mental Model - An Applied Election Odds Example 2019-11-28T09:29:56.753Z
Theater Tickets, Sleeping Pills, and the Idiosyncrasies of Delegated Risk Management 2019-10-30T10:33:16.240Z
Divergence on Evidence Due to Differing Priors - A Political Case Study 2019-09-16T11:01:11.341Z
Hackable Rewards as a Safety Valve? 2019-09-10T10:33:40.238Z
What Programming Language Characteristics Would Allow Provably Safe AI? 2019-08-28T10:46:32.643Z
Mesa-Optimizers and Over-optimization Failure (Optimizing and Goodhart Effects, Clarifying Thoughts - Part 4) 2019-08-12T08:07:01.769Z
Applying Overoptimization to Selection vs. Control (Optimizing and Goodhart Effects - Clarifying Thoughts, Part 3) 2019-07-28T09:32:25.878Z
What does Optimization Mean, Again? (Optimizing and Goodhart Effects - Clarifying Thoughts, Part 2) 2019-07-28T09:30:29.792Z
Re-introducing Selection vs Control for Optimization (Optimizing and Goodhart Effects - Clarifying Thoughts, Part 1) 2019-07-02T15:36:51.071Z
Schelling Fences versus Marginal Thinking 2019-05-22T10:22:32.213Z
Values Weren't Complex, Once. 2018-11-25T09:17:02.207Z
Oversight of Unsafe Systems via Dynamic Safety Envelopes 2018-11-23T08:37:30.401Z
Collaboration-by-Design versus Emergent Collaboration 2018-11-18T07:22:16.340Z
Multi-Agent Overoptimization, and Embedded Agent World Models 2018-11-08T20:33:00.499Z
Policy Beats Morality 2018-10-17T06:39:40.398Z
(Some?) Possible Multi-Agent Goodhart Interactions 2018-09-22T17:48:22.356Z
Lotuses and Loot Boxes 2018-05-17T00:21:12.583Z
Non-Adversarial Goodhart and AI Risks 2018-03-27T01:39:30.539Z
Evidence as Rhetoric — Normative or Positive? 2017-12-06T17:38:05.033Z
A Short Explanation of Blame and Causation 2017-09-18T17:43:34.571Z

Comments

Comment by Davidmanheim on My Clients, The Liars · 2024-03-12T08:29:34.414Z · LW · GW

Again, I think it was a fine and enjoyable post.

But I didn't see where you "demonstrate how I used very basic rationalist tools to uncover lies," which could have improved the post, and I don't think this really explored any underappreciated parts of "deception and how it can manifest in the real world" - which I agree is underappreciated. Unfortunately, this post didn't provide much clarity about how to find it, or how to think about it. So again, it's a fine post, good stories, and I agree they illustrate being more confused by fiction than reality, and other rationalist virtues, but as I said, it was not "the type of post that leads people to a more nuanced or better view of any of the things discussed." 

Comment by Davidmanheim on My Clients, The Liars · 2024-03-10T10:35:34.025Z · LW · GW

I disagree with this decision, not because I think it was a bad post, but because it doesn't seem like the type of post that leads people to a more nuanced or better view of any of the things discussed, much less a post that provided insight or better understanding of critical things in the broader world. It was enjoyable, but not what I'd like to see more of on Less Wrong.

(Note: I posted this response primarily because I saw that lots of others also disagreed with this, and think it's worth having on the record why at least one of us did so.)

Comment by Davidmanheim on The World in 2029 · 2024-03-05T11:34:13.977Z · LW · GW

"Climate change is seen as a bit less of a significant problem"
 

That seems shockingly unlikely (5%) - even if we have essentially eliminated all net emissions (10%), we will still be seeing continued warming (99%) unless we have widely embraced geoengineering (10%). If we have, it is a source of significant geopolitical contention (75%) due to uneven impacts (50%) and pressure from environmental groups (90%) worried that it is promoting continued emissions and / or causes other harms. Progress on carbon capture is starting to pay off (70%) but is not (90%) deployed at anything like the scale needed to stop or reverse warming.

Adaptation to climate change has continued (99%), but it is increasingly obvious how expensive it is and how badly it is impacting developing world. The public still seems to think this is the fault of current emissions (70%) and carbon taxes or similar legal limits are in place for a majority of G7 countries (50%) but less than half of other countries (70%).

Comment by Davidmanheim on On coincidences and Bayesian reasoning, as applied to the origins of COVID-19 · 2024-02-22T15:26:35.129Z · LW · GW

To start, the claim that it was found 2 miles from the facility is an important mistake, because WIV is 8 miles from the market. For comparison to another city people might know better, in New York, that's the distance between World Trade Center and either Columbia University, or Newark Airport. Wuhan's downtown is around 16 miles across. 8 miles away just means it was in the same city. 

And you're over-reliant on the evidence you want to pay attention to. For example, even rstricting ourselves to "nearby coincidence" evidence, the Hunan the market is the largest in central China - so what are the odds that a natural spillover events occurs immediately surrounding the largest animal market? If the disease actually emerged from WIV, what are the odds that the cases centered around the Hunan market, 8 miles away, instead of the Baishazhou live animal market, 3 miles away, or the Dijiao market, also 8 miles away?

So I agree that an update can be that strong, but this one simply isn't.

Comment by Davidmanheim on On coincidences and Bayesian reasoning, as applied to the origins of COVID-19 · 2024-02-20T22:12:27.135Z · LW · GW

Yeah, but I think that it's more than not taken literally, it's that the exercise is fundamentally flawed when being used as an argument instead of very narrowly for honest truth-seeking, which is almost never possible in a discussion without unreasonably high levels of trust and confidence in others' epistemic reliability.

Comment by Davidmanheim on On coincidences and Bayesian reasoning, as applied to the origins of COVID-19 · 2024-02-19T14:28:53.309Z · LW · GW
  1. What is the relevance of the "posterior" that you get after updating on a single claim that's being chosen, post-hoc, as the one that you want to use as an example?
  2. Using a weak prior biases towards thinking the information you have to update with is strong evidence. How did you decide on that particular prior? You should presumably have some reference class for your prior. (If you can't do that, you should at least have equipoise between all reasonable hypotheses being considered. Instead, you're updating "Yes Lableak" versus "No Lableak" - but in fact, "from a Bayesian perspective, you need an amount of evidence roughly equivalent to the complexity of the hypothesis just to locate the hypothesis in theory-space. It’s not a question of justifying anything to anyone.") 
  3. How confident are you in your estimate of the bayes factor here? Do you have calibration data for roughly similar estimates you have made? Should you be adjusting for less than perfect confidence? 
Comment by Davidmanheim on On coincidences and Bayesian reasoning, as applied to the origins of COVID-19 · 2024-02-19T13:53:55.928Z · LW · GW

Thank you for writing this.

I think most points here are good points to make, but I also think it's useful as a general caution against this type of exercise being used as an argument at all! So I'd obviously caution against anyone taking your response itself as a reasonable attempt at an estimate of the "correct" Bayes factors, because this is all very bad epistemic practice!  Public explanations and arguments are social claims, and usually contain heavily filtered evidence (even if unconsciously). Don't do this in public.

That is, this type of informal Bayesian estimate is useful as part of a ritual for changing your own mind, when done carefully. That requires a significant degree of self-composure, a willingness to change one's mind, and a high degree of justified confidence n your own mastery of unbiased reasoning.

Here, though, it is presented as an argument, which is not how any of this should work. And in this case, it was written by someone who already had a strong view of what the outcome should be, repeated publicly frequently, which makes it doubly hard to accept the implicit necessary claim that it was performed starting from an unbiased point at face value! At the very least, we need strong evidence that it was not an exercise in motivated reasoning, that the bottom line wasn't written before the evaluation started - which statement is completely missing, though to be fair, it would be unbelievable if it had been stated.

Comment by Davidmanheim on "Open Source AI" isn't Open Source · 2024-02-16T07:59:30.042Z · LW · GW

I agree that releasing model weights is "partially open sourcing" - in much the same way that freeware is "partially open sourcing" software, or restrictive licences with code availability is.

But that's exactly the point; you don't get to call something X because it's kind-of-like X, it needs to actually fulfill the requirements in order to get the label. What is being called Open Source AI doesn't actually do the thing that it needs to.

Comment by Davidmanheim on "Open Source AI" isn't Open Source · 2024-02-15T20:55:55.385Z · LW · GW

Thanks - I agree that this discusses the licenses, which would be enough to make LlaMa not qualify, but I think there's a strong claim I put forward in the full linked piece that even if the model weights were released using a GPL license, those "open" model weights wouldn't make it open in the sense that Open Source means elsewhere.

Comment by Davidmanheim on "Open Source AI" isn't Open Source · 2024-02-15T20:51:38.471Z · LW · GW

I agree that the reasons someone wants the dataset generally aren't the same reasons they'd want to compile from source code. But there's a lot of utility for research in having access to the dataset even if you don't recompile. Checking whether there was test-set leakage for metrics, for example, or assessing how much of LLM ability is stochastic parroting of specific passages versus recombination. And if it was actually open, these would not be hidden from researchers.

And supply chain is a reasonable analogy - but many open-source advocates make sure that their code doesn't depend on closed / proprietary libraries. It's not actually "libre" if you need to have a closed source component or pay someone to make the thing work. Some advocates, those who built or control quite a lot of the total open source ecosystem, also put effort into ensuring that the entire toolchain needed to compile their code is open, because replicability shouldn't be contingent on companies that can restrict usage or hide things in the code. It's not strictly required, but it's certainly relevant.

Comment by Davidmanheim on "Open Source AI" isn't Open Source · 2024-02-15T20:44:46.435Z · LW · GW

The vast majority of uses of software are via changing configuration and inputs, not modifying code and recompiling the software. (Though lots of Software as a Service doesn't even let you change configuration directly.) But software is not open in this sense unless you can recompile, because it's not actually giving you full access to what was used to build it.

The same is the case for what Facebook call open-source LLMs; it's not actually giving you full access to what was used to build it.

Comment by Davidmanheim on "Open Source AI" isn't Open Source · 2024-02-15T15:49:52.875Z · LW · GW

Thanks - Redpajama definitely looks like it fits the bill, but it shouldn't need to bill itself as making "fully-open, reproducible models," since that's what "open source" is already supposed to mean. (Unfortunately, the largest model they have is 7B.)

Comment by Davidmanheim on "Open Source AI" isn't Open Source · 2024-02-15T15:45:06.113Z · LW · GW

Yes, agreed - as I said in the post, "Open Source AI simply means that the models have the model weights released - the equivalent of software which makes the compiled code available. (This is otherwise known as software.)"

Comment by Davidmanheim on "Open Source AI" isn't Open Source · 2024-02-15T13:57:39.654Z · LW · GW

"Freely remixable" models don't generally have open datasets used for training. If you know of one, that's great, and would be closer to open source. (Not Mistral. And Phi-2 is using synthetic data from other LLMs - I don't know what they released about the methods used to generate or select the text, but it's not open.)

But the entire point is that weights are not the source code for an LLM, they are the compiled program. Yes, it's modifiable via LoRA and similar, but that's not open source! Open source would mean I could replicate it, from the ground up. For facebook's models, at least, the details of the training methods, the RLHF training they do, where they get the data, all of those things are secrets. But they call it "Open Source AI" anyways.

Comment by Davidmanheim on Technologies and Terminology: AI isn't Software, it's... Deepware? · 2024-02-13T16:51:37.116Z · LW · GW

Good point, and I agree that it's possible that what I see as essential features might go away - "floppy disks" turned out to be a bad name when they ended up inside hard plastic covers, and "deepware" could end up the same - but I am skeptical that it will.

I agree that early electronics were buggy until we learned to build them reliably - and perhaps we can solve this for gradient-descent based learning, though many are skeptical of that, since many of the problems have been shown to be pretty fundamental. I also agree that any system is inscrutable until you understand it, but unlike early electronics, no-one understands these massive lists of numbers that produce text, and human brains can't build them, they just program a process to grow them. (Yes, composable NNs could solve some of this, as you point out when mentioning separable systems, but I still predict they won't be well understood, because the components individually are still deepware.)

Comment by Davidmanheim on Safe Stasis Fallacy · 2024-02-11T19:29:32.807Z · LW · GW

You talk about "governance by Friendly AGI" as if it's a solved problem we're just waiting to deploy, not speculation that might simply not be feasible even if we solve AGI alignment, which itself is plausibly unsolvable in the near term. You also conflate AI safety research with AI governance regimes. And note that the problems with governance generally aren't a lack of intelligence by those in charge, it's largely conflicting values and requirements. And with that said, you talk about modern liberal governments as if they are the worst thing we've experienced, "riddled with brokenness," as if that's the fault of the people in charge, not the deeply conflicting mandates that the populace gives them. And to the extent that the systemic failure is the fault of the untrustworthy incentives of those in charge, why would controllable or aligned AGI fix that?

Yes, stasis isn't safe by default, but undirected progress isn't a panacea, and governance certainly isn't any closer to solved just because we have AI progress.

Comment by Davidmanheim on Brute Force Manufactured Consensus is Hiding the Crime of the Century · 2024-02-11T19:21:38.765Z · LW · GW

Thanks. I was unaware of the law, and yes, that does seem to be strong evidence that the agencies in question don't have any evidence specific enough to come to any conclusion. That, or they are foolishly risking pissing off Congress, which can subpoena them, and seems happy to do exactly that in other situations - and they would do so knowing that it's eventually going to come out that they withheld evidence?!?

Again, it's winter, people get sick, that's very weak Bayesian evidence of an outbreak, at best. On priors, how many people at an institute that size get influenza every month during the winter?

And the fact that it was only 3 people, months earlier, seems to indicate moderately strongly it wasn't the source of the full COVID-19 outbreak, since if it were, given the lack of precautions against spread at the time, if it already infected 3 different people, it seems likely it would have spread more widely within China starting at that time.

Comment by Davidmanheim on Brute Force Manufactured Consensus is Hiding the Crime of the Century · 2024-02-11T07:17:30.260Z · LW · GW

Sorry, I'm having trouble following. You're saying that 1) it's unlikely to be a lab leak known to US Intel because it would have been known to us via leaks, and 2) you think that Intel agencies have evidence about WIV employees having COVID and that it's being withheld?

First, I think you're overestimating both how much information from highly sensitive sources would leak, and how much Chinese leaders would know if it were a lab leak. This seems on net to be mostly uninformative. 

Second, if they have evidence about WIV members having COVID, (and not, you know, any other respiratory disease in the middle of flu/cold season,) I still don't know why you think you would know that it was withheld from congress. Intel agencies share classified information with certain members of Congress routinely, but you'd never know what was or was not said. You think a lack of a leak is evidence that  would have been illegally withheld from congress - but it's not illegal for Intel agencies to keep information secret, in a wide variety of cases. 

And on that second point, even without the above arguments, not having seen such evidence publicly leaked can't plausibly be more likely in a world where it was a lab leak that was hidden, than it would be in a world where it wasn't a lab leak and the evidence you're not seeing simply doesn't exist!

Comment by Davidmanheim on Brute Force Manufactured Consensus is Hiding the Crime of the Century · 2024-02-09T13:30:01.457Z · LW · GW

State department isn't part of "US intelligence agencies and military," and faces very, very different pressures. And despite this, as you point out there are limits to internal pressures in intel agencies - which at least makes it clear that the intel agencies don't have strong and convincing non-public evidence for the leak hypothesis.

Comment by Davidmanheim on Brute Force Manufactured Consensus is Hiding the Crime of the Century · 2024-02-09T13:27:08.545Z · LW · GW

I'm not saying it's impossible, I'm saying it's implausible. (So if this is a necessary precondition for believing in a lab leak, it is clear evidence against it.)

Comment by Davidmanheim on Brute Force Manufactured Consensus is Hiding the Crime of the Century · 2024-02-07T15:18:50.700Z · LW · GW

"(and likely parts of the US intelligence agencies and military) desperately wanted this to not be a lab leak."

 

As I said in another comment, that seems very, very hard to continue to believe, even if it might have seemed plausible on priors.

Comment by Davidmanheim on Brute Force Manufactured Consensus is Hiding the Crime of the Century · 2024-02-07T15:16:38.454Z · LW · GW

Whoever publishes or sends out notices may or may not have others they check with. That's sometimes the local health authority directly, but may go through the national government. I don't know enough about how that works in China to say in general who might have been able to tell Wuhan Municipal Health Committee or WCDC what they were and were not supposed to say when they made their announcements. However, we have lots of information about what was said in the public statements and hospital records from that time, most of which is mentioned here. (You don't need to trust him much, the descriptions of the systems and what happened when are well known.) But data is also disseminated informally through lots of channels, and I don't know who would have been getting updates from colleagues or sending data to the WHO or US CDC.

Comment by Davidmanheim on Brute Force Manufactured Consensus is Hiding the Crime of the Century · 2024-02-07T13:46:42.080Z · LW · GW

But the government would need to have started the coverup while they were suppressing evidence. It's weird to think they simultaneously were covering up transmission, and faking the data about the cases to make it fit the claim it originated in the wet market.

Comment by Davidmanheim on Brute Force Manufactured Consensus is Hiding the Crime of the Century · 2024-02-07T13:41:15.167Z · LW · GW

Most very large changes is viral evolution is lateral transfer between viruses, rather than accumulation of point mutations. The better claim would be that this was acquired by a proto-SARS-CoV-2 virus that way, not that it was the result of cross-species changes alone.

Comment by Davidmanheim on Brute Force Manufactured Consensus is Hiding the Crime of the Century · 2024-02-07T13:36:44.678Z · LW · GW

Strong evidence against this theory and predictions is the actual statements by intel orgs, which were notably less skeptical than other sources of lab origin.

Comment by Davidmanheim on Brute Force Manufactured Consensus is Hiding the Crime of the Century · 2024-02-07T13:31:13.619Z · LW · GW

Looks like such data doesn't exist, and post-2020 wildlife trading ban, new data won't tell us anything about pre-ban conditions - but we know there is lots of cross-border and long distance transport of wildlife. See, for example, this. And elsewhere in Asia, we see similar descriptions of very large volume of wildlife trade over long distances.

Comment by Davidmanheim on Brute Force Manufactured Consensus is Hiding the Crime of the Century · 2024-02-07T07:41:20.094Z · LW · GW

Partly disagree - the relevant question isn't distance, it's the amount of wildlife from specific places. New York is further from Atlanta than from Litchfield, CT, but there are more people from Atlanta in New York at any given time. And we know that there's a lot of trade in wildlife in Wuhan from distant places, which is the critical question.

Comment by Davidmanheim on Newcomb's Problem and Regret of Rationality · 2024-02-04T06:55:26.406Z · LW · GW

For the future, perhaps this once again updated link may help: Updated link

Citation: LEDWIG, Marion, 2000. Newcomb's problem [Dissertation]. Konstanz: University of Konstanz

Comment by Davidmanheim on Surgery Works Well Without The FDA · 2024-01-28T07:52:33.350Z · LW · GW

"Regarding surgery there are immediate outcomes and they are treating a specific problem so it is easy to measure"

There are longer term impacts of surgeries, though, that aren't easily measured, and surgeries tend to score far more poorly on those fronts. Fixing a (sometimes minor) problem in ways that make people need more healthcare later is at best a mixed bag.

Comment by Davidmanheim on Surgery Works Well Without The FDA · 2024-01-28T07:50:01.140Z · LW · GW

One problem with this analysis is that the FDA actually regulates a ton of what happens in surgeries, which sharply reduces how much this example actually proves. Specifically, they approve all of the devices that get used during surgery, from the scalpels to the monitoring equipment to the surgical sponges, all the laparoscopic instruments used in modern surgeries, and all the replacement joints and implanted or assistive devices which are being put in in many surgeries. Yes, this is different than regulating the surgery itself, but it manages to cut off a huge range of surgical interventions that could be more dangerous or less reliable, while it raises costs and slows innovation. So I'm less sure how well surgery serves as an example of working well without the FDA.

Comment by Davidmanheim on Redirecting one’s own taxes as an effective altruism method · 2024-01-16T08:15:42.337Z · LW · GW

This is not talking about pre-tax investment.

Comment by Davidmanheim on Redirecting one’s own taxes as an effective altruism method · 2024-01-16T08:10:13.646Z · LW · GW

Please do not advocate violating the law in order to "do good" on the forum.

Comment by Davidmanheim on AI Is Not Software · 2024-01-05T14:02:57.211Z · LW · GW

You're making a far different claim than I was - that everything using statistics for decisionmaking is not software. And I'd mostly agree, but don't think AI is particularly similar to statistics either, and the two both need intuition and understanding, but the intuitions needed differ greatly, so the point seems separate.

Comment by Davidmanheim on AI Is Not Software · 2024-01-03T21:09:53.170Z · LW · GW

Yes, I think you're now saying something akin to what I was trying to say. The AI, as a set of weights and activation funtions, is a different artifact than the software being used to multiply the matrices, much less the program used to output the text. (But I'm not sure this is quite the same as a different level of abstraction, the way humans versus atoms are - though if we want to take that route, I think gjm's comment about humans and chemistry makes this clearer.)

Comment by Davidmanheim on AI Is Not Software · 2024-01-03T21:04:26.454Z · LW · GW

In manufacturing or other high-reliability processes, human inputs are components which need the variability in their outputs controlled. I'm drawing a parallel, not saying that AI is responsible for humans.

And I'm completely unsure why you think that engineered systems that use AI are being built so carefully, but regardless, it doesn't mean that the AI is controlled, just that it's being constrained by the system. (And to briefly address your points, which aren't related to what I had been saying, we know how poorly humans take to being narrowly controlled, so to the extent that LLMs and similar systems are human-like, this seems like a strange way to attempt to ensure safety with increasingly powerful AI.)

Comment by Davidmanheim on AI Is Not Software · 2024-01-03T07:15:15.629Z · LW · GW

I'm not talking about AI doom right now, I'm talking about understanding the world. At the system level, you claim we need to design the overall system to purpose, and that's fine for controlling a system - but the same goes for when humans are part of the system, and we're using six sigma to reduce variance in process outcomes. And at that point, it's strange to say that more generally, humans are just components to control, and we can ignore that they are different than actuators and motors, or software scripts. Instead, we have different categories - which was the point originally.

Comment by Davidmanheim on AI Is Not Software · 2024-01-03T07:08:21.299Z · LW · GW

I'm not saying that I can force breaking of category boundaries, I'm asking whether the categories are actually useful for thinking about the systems. I'm saying it isn't, and we need to stop trying to use categories in this way.

And your reply didn't address they key point - is the thing that controls the body being shown in the data being transmitted software, or data? And parallel to that, is the thing that controls the output of the AI system software or data?

Comment by Davidmanheim on AI Is Not Software · 2024-01-02T20:39:58.152Z · LW · GW

Yes, this is part of the intuition I was trying to get across - thanks! 

Comment by Davidmanheim on AI Is Not Software · 2024-01-02T20:30:38.218Z · LW · GW

All of the abstractions you are using are just as relevant to system design for a factory, or a construction project. You could do all of the things you're discussing about system design for a mechanical system! And yes, present engineers are treating these systems like software, but to the extent they are doing so, they are missing lots of critical issues, and as we've seen, they keep needing to add on bits to their conceptual model of what they need to deal with in order to have it be relevant. (Which is why I'm saying we need to stop thinknig of it as software.)

So on your final set of questions, I certainly agree that the systems can be analyzed via similar approaches to system engineering, but I think talking about the AI systems as software is far less helpful than saying that are a different thing, with different failure modes, requiring a different conceptual model, and doing your systems engineering on that basis.

Comment by Davidmanheim on AI Is Not Software · 2024-01-02T20:24:10.715Z · LW · GW

I'm asking the question of how we should think about the systems, and claiming "software" is very much the wrong conceptual model. Yes, AI can work poorly because of a software issue, for example, timeouts with the API, or similar. But the thing we're interested in discussing is the AI, not the software component - and as you point out in the case of photography, the user's skill with the software, and with everything else about taking photographs, is something that occurs and should be discussed not in terms of the software being used.

Comment by Davidmanheim on AI Is Not Software · 2024-01-02T20:20:39.842Z · LW · GW

We can play the game of recategorizing certian things, and saying data and software are separate - but the question is whether it adds insight, or not. And I think that existing categories are more misleading than enlightening, hence my claim.

For example, is my face being picked up by the camera during the videoconference "data" in a meaningful sense? Does that tell you something useful about how to have a videoconference? If not, should we call it software, or shift our focus elsewhere when discussing it?

Comment by Davidmanheim on A Defense of Work on Mathematical AI Safety · 2023-12-24T08:05:27.669Z · LW · GW

I think th dispute here is that you're interpreting mathematical too narrowly, and almost all of the work happening in agent foundations and similar is exactly what was being worked on by "mathematical AI research" 5-7 years ago. The argument was that those approaches have been fruitful, and we should expect them to continue to be so - if you want to call that "foundational conceptual research" instead of "Mathematical AI research," that's fine..

Comment by Davidmanheim on On infinite ethics · 2023-12-24T07:33:58.031Z · LW · GW

First, I said I was frustrated that you didn't address the paper, which I intended to mean was a personal frustration, not blame for not engaging, given the vast number of things you could have focused on. I brought it up only because I don't want to make it sound like I thought it was relevant for those reading my comment, to appreciate that this was a personal motive, not a dispassionate evaluation.


Howeer, to defend my criticism, for decisionmakers with finite computational power / bounded time and limited ability to consider issues, I think that there's a strong case to dismiss arguments based on plausible relevance. There are, obviously, an huge (but, to be fair, weighted by a simplicity prior, effectively finite) number of potential philosophies or objections, and a smaller amount of time to make decisions than would be required to evaluate each. So I think we need a case for relevance, and I have two reasons / partial responses to the above that I think explain why I don't think there is such a case.

  1. There are (to simplify greatly,) two competing reasons for a theory to have come to our attention enough to be considered; plausibility, or interestingness. If a possibility is very cool seeming, and leads to lots of academic papers and cool sounding ideas, the burden of proof for plausibility is, ceteris pariubus, higher.

    This is not to say that we should strongly dismiss these questions, but it is a reason to ask for more than just non-zero possibility that physics is wrong. (And in the paper, we make an argument that "physics is wrong" still doesn't imply that bounds we know of are likely to be revoked - most changes to physics which have occurred have constrained things more, not less.)
     
  2. I'm unsure why I should care that I have intuitions that can be expanded to implausible cases. Justifying this via intuitions built on constructed cases which work seems exactly backwards.

    As an explanation for why I think this is confused, Stuart Armstrong made a case that people fall prey to a failure mode in reasoning that parallels one we see in ML, which I'll refer to as premature rulemaking. In ML, that's seen when a model sees a small sample, and try to build a classification rule based on that, and apply it out of sample; all small black fuzzy object it has seen are cats, and it has seen to cats which are large or other colors, so it calls large grey housecats non-cats, and small black dogs cats. Even moving from that point, it it harder to change away from that mode; we can convince it that dogs are a different category, but the base rule gets expanded by default to other cases, and tigers are not cats, and black mice are, etc. Once we set up the problem as a classifier, trying to find rules, we spend time building systems, not judging cases on their merits. (The alternative he proposes in this context, IIRC, is something like trying to do grouping rather than build rules, and evaluate distance from the cluster rather than classification.)

    The parallel here is that people find utilitarianism / deontology / maximizing complexity plausible in a set of cases, and jump to using it as a rule. This is the premature rulemaking. People then try to modify the theory to fit a growing number of cases, ignoring that it's way out of sample for their intuitions. Intuitions then get reified, and people self-justify their new reified intuitions as obvious. (Some evidence for this: people have far more strongly contrasting intuitions in less plausible constructed cases.)

This has gone somewhat off track, I think, but in short, I'm deeply unsure why we should spend time on infinite ethics, have a theory for why people do so, and would want to see strong evidence of why to focus on the topic before considering it useful, as opposed to fun.

Comment by Davidmanheim on Murphyjitsu: an Inner Simulator algorithm · 2023-12-21T11:56:45.232Z · LW · GW

This is an important post which I think deserves inclusion in the best-of compilation because despite it's usefulness and widespread agreement about that fact, it seems notto have been highlighted well to the community. 

Comment by Davidmanheim on Beware boasting about non-existent forecasting track records · 2023-12-21T11:54:47.729Z · LW · GW

This makes an important point that I find myself consistently referring to - almost none of the confidence in predictions, even inside the rationalist community, is based on actual calibration data. Experts forecast poorly, and we need to stop treating expertise or argumentation as strong stand-alone reasons to accept claims which are implicitly disputed by forecasts.

On the other hand, I think that this post focused far too much on Eliezer. In fact, there are relatively few people in the community who have significant forecasting track records, and this community does tremendously better than most. This leads to lots of strong opinions based on "understanding" which refuse to defer to forecaster expectations or even engage much with why they would differ.

Comment by Davidmanheim on On infinite ethics · 2023-12-21T11:41:48.401Z · LW · GW

I did not see this post when it was first put on the forum, but reading it now, my personal view of this post is that it continues a trend of wasting time on a topic that is already a focus of too much effort, with little relevance to actual decisions, and no real new claim that the problems were relevant or worth addressing.

I was even more frustrated that it didn't address most of the specific arguments put forward in our paper from a year earlier on why value for decisionmaking was finite, and then put forward seeral arguments we explicitly gave reasons to dismiss - including dismissing ordinal preferences, and positing situations which assume infinities instead of showing they could be relevant to any decision.  I think that it doesn't actually engage with reality in ways that are falsifiable, and ignores that even with assuming infinite universes based on physics, the theories which posit infinities in physics are strongly in favor of the view that there is a finite *affectable* universe, making the relevance to ethics hard to justify. 

Comment by Davidmanheim on It’s Probably Not Lithium · 2023-12-21T11:21:48.904Z · LW · GW

This post publicly but non-confrontationally rebutting an argument that had been put forward and promoted by others was a tremendous community service, of a type we see too rarely, albeit far more often in this community than most. It does not engage in strawmanning, it clearly lays out both the original claim and the evidence, and it attempts to engage positively, including trying to find concrete predictions that the disputing party could agree with. 

I think this greatly moved community consensus on a moderately important topic in ways that were very valuable. I will note that it's unfortunate but indicative that SMTM never responded

Comment by Davidmanheim on Public Call for Interest in Mathematical Alignment · 2023-11-22T20:53:12.792Z · LW · GW

Thanks - and the fact that we don't know who is working on relevant things is exactly the reason we're doing this! 

Comment by Davidmanheim on Public Call for Interest in Mathematical Alignment · 2023-11-22T19:27:14.107Z · LW · GW

We are focused on mathematical research and building bridges between academia and research. I think the pathway to doing that type of research is usually through traditional academic channels, a PhD program, or perhaps a masters degree or a program like MATS, at which point the type of research promotion and academic bridge building we are focused on become far more relevant. That said, we do have undergrad as an option, and are certainly OK with people at any level of seniority signaling their interest.

Comment by Davidmanheim on Introduction and current research agenda · 2023-11-20T19:42:28.312Z · LW · GW

Do you have any description of your research agenda, or is this just supposed to provide background?