Posts
Comments
Again, I think it was a fine and enjoyable post.
But I didn't see where you "demonstrate how I used very basic rationalist tools to uncover lies," which could have improved the post, and I don't think this really explored any underappreciated parts of "deception and how it can manifest in the real world" - which I agree is underappreciated. Unfortunately, this post didn't provide much clarity about how to find it, or how to think about it. So again, it's a fine post, good stories, and I agree they illustrate being more confused by fiction than reality, and other rationalist virtues, but as I said, it was not "the type of post that leads people to a more nuanced or better view of any of the things discussed."
I disagree with this decision, not because I think it was a bad post, but because it doesn't seem like the type of post that leads people to a more nuanced or better view of any of the things discussed, much less a post that provided insight or better understanding of critical things in the broader world. It was enjoyable, but not what I'd like to see more of on Less Wrong.
(Note: I posted this response primarily because I saw that lots of others also disagreed with this, and think it's worth having on the record why at least one of us did so.)
"Climate change is seen as a bit less of a significant problem"
That seems shockingly unlikely (5%) - even if we have essentially eliminated all net emissions (10%), we will still be seeing continued warming (99%) unless we have widely embraced geoengineering (10%). If we have, it is a source of significant geopolitical contention (75%) due to uneven impacts (50%) and pressure from environmental groups (90%) worried that it is promoting continued emissions and / or causes other harms. Progress on carbon capture is starting to pay off (70%) but is not (90%) deployed at anything like the scale needed to stop or reverse warming.
Adaptation to climate change has continued (99%), but it is increasingly obvious how expensive it is and how badly it is impacting developing world. The public still seems to think this is the fault of current emissions (70%) and carbon taxes or similar legal limits are in place for a majority of G7 countries (50%) but less than half of other countries (70%).
To start, the claim that it was found 2 miles from the facility is an important mistake, because WIV is 8 miles from the market. For comparison to another city people might know better, in New York, that's the distance between World Trade Center and either Columbia University, or Newark Airport. Wuhan's downtown is around 16 miles across. 8 miles away just means it was in the same city.
And you're over-reliant on the evidence you want to pay attention to. For example, even rstricting ourselves to "nearby coincidence" evidence, the Hunan the market is the largest in central China - so what are the odds that a natural spillover events occurs immediately surrounding the largest animal market? If the disease actually emerged from WIV, what are the odds that the cases centered around the Hunan market, 8 miles away, instead of the Baishazhou live animal market, 3 miles away, or the Dijiao market, also 8 miles away?
So I agree that an update can be that strong, but this one simply isn't.
Yeah, but I think that it's more than not taken literally, it's that the exercise is fundamentally flawed when being used as an argument instead of very narrowly for honest truth-seeking, which is almost never possible in a discussion without unreasonably high levels of trust and confidence in others' epistemic reliability.
- What is the relevance of the "posterior" that you get after updating on a single claim that's being chosen, post-hoc, as the one that you want to use as an example?
- Using a weak prior biases towards thinking the information you have to update with is strong evidence. How did you decide on that particular prior? You should presumably have some reference class for your prior. (If you can't do that, you should at least have equipoise between all reasonable hypotheses being considered. Instead, you're updating "Yes Lableak" versus "No Lableak" - but in fact, "from a Bayesian perspective, you need an amount of evidence roughly equivalent to the complexity of the hypothesis just to locate the hypothesis in theory-space. It’s not a question of justifying anything to anyone.")
- How confident are you in your estimate of the bayes factor here? Do you have calibration data for roughly similar estimates you have made? Should you be adjusting for less than perfect confidence?
Thank you for writing this.
I think most points here are good points to make, but I also think it's useful as a general caution against this type of exercise being used as an argument at all! So I'd obviously caution against anyone taking your response itself as a reasonable attempt at an estimate of the "correct" Bayes factors, because this is all very bad epistemic practice! Public explanations and arguments are social claims, and usually contain heavily filtered evidence (even if unconsciously). Don't do this in public.
That is, this type of informal Bayesian estimate is useful as part of a ritual for changing your own mind, when done carefully. That requires a significant degree of self-composure, a willingness to change one's mind, and a high degree of justified confidence n your own mastery of unbiased reasoning.
Here, though, it is presented as an argument, which is not how any of this should work. And in this case, it was written by someone who already had a strong view of what the outcome should be, repeated publicly frequently, which makes it doubly hard to accept the implicit necessary claim that it was performed starting from an unbiased point at face value! At the very least, we need strong evidence that it was not an exercise in motivated reasoning, that the bottom line wasn't written before the evaluation started - which statement is completely missing, though to be fair, it would be unbelievable if it had been stated.
I agree that releasing model weights is "partially open sourcing" - in much the same way that freeware is "partially open sourcing" software, or restrictive licences with code availability is.
But that's exactly the point; you don't get to call something X because it's kind-of-like X, it needs to actually fulfill the requirements in order to get the label. What is being called Open Source AI doesn't actually do the thing that it needs to.
Thanks - I agree that this discusses the licenses, which would be enough to make LlaMa not qualify, but I think there's a strong claim I put forward in the full linked piece that even if the model weights were released using a GPL license, those "open" model weights wouldn't make it open in the sense that Open Source means elsewhere.
I agree that the reasons someone wants the dataset generally aren't the same reasons they'd want to compile from source code. But there's a lot of utility for research in having access to the dataset even if you don't recompile. Checking whether there was test-set leakage for metrics, for example, or assessing how much of LLM ability is stochastic parroting of specific passages versus recombination. And if it was actually open, these would not be hidden from researchers.
And supply chain is a reasonable analogy - but many open-source advocates make sure that their code doesn't depend on closed / proprietary libraries. It's not actually "libre" if you need to have a closed source component or pay someone to make the thing work. Some advocates, those who built or control quite a lot of the total open source ecosystem, also put effort into ensuring that the entire toolchain needed to compile their code is open, because replicability shouldn't be contingent on companies that can restrict usage or hide things in the code. It's not strictly required, but it's certainly relevant.
The vast majority of uses of software are via changing configuration and inputs, not modifying code and recompiling the software. (Though lots of Software as a Service doesn't even let you change configuration directly.) But software is not open in this sense unless you can recompile, because it's not actually giving you full access to what was used to build it.
The same is the case for what Facebook call open-source LLMs; it's not actually giving you full access to what was used to build it.
Thanks - Redpajama definitely looks like it fits the bill, but it shouldn't need to bill itself as making "fully-open, reproducible models," since that's what "open source" is already supposed to mean. (Unfortunately, the largest model they have is 7B.)
Yes, agreed - as I said in the post, "Open Source AI simply means that the models have the model weights released - the equivalent of software which makes the compiled code available. (This is otherwise known as software.)"
"Freely remixable" models don't generally have open datasets used for training. If you know of one, that's great, and would be closer to open source. (Not Mistral. And Phi-2 is using synthetic data from other LLMs - I don't know what they released about the methods used to generate or select the text, but it's not open.)
But the entire point is that weights are not the source code for an LLM, they are the compiled program. Yes, it's modifiable via LoRA and similar, but that's not open source! Open source would mean I could replicate it, from the ground up. For facebook's models, at least, the details of the training methods, the RLHF training they do, where they get the data, all of those things are secrets. But they call it "Open Source AI" anyways.
Good point, and I agree that it's possible that what I see as essential features might go away - "floppy disks" turned out to be a bad name when they ended up inside hard plastic covers, and "deepware" could end up the same - but I am skeptical that it will.
I agree that early electronics were buggy until we learned to build them reliably - and perhaps we can solve this for gradient-descent based learning, though many are skeptical of that, since many of the problems have been shown to be pretty fundamental. I also agree that any system is inscrutable until you understand it, but unlike early electronics, no-one understands these massive lists of numbers that produce text, and human brains can't build them, they just program a process to grow them. (Yes, composable NNs could solve some of this, as you point out when mentioning separable systems, but I still predict they won't be well understood, because the components individually are still deepware.)
You talk about "governance by Friendly AGI" as if it's a solved problem we're just waiting to deploy, not speculation that might simply not be feasible even if we solve AGI alignment, which itself is plausibly unsolvable in the near term. You also conflate AI safety research with AI governance regimes. And note that the problems with governance generally aren't a lack of intelligence by those in charge, it's largely conflicting values and requirements. And with that said, you talk about modern liberal governments as if they are the worst thing we've experienced, "riddled with brokenness," as if that's the fault of the people in charge, not the deeply conflicting mandates that the populace gives them. And to the extent that the systemic failure is the fault of the untrustworthy incentives of those in charge, why would controllable or aligned AGI fix that?
Yes, stasis isn't safe by default, but undirected progress isn't a panacea, and governance certainly isn't any closer to solved just because we have AI progress.
Thanks. I was unaware of the law, and yes, that does seem to be strong evidence that the agencies in question don't have any evidence specific enough to come to any conclusion. That, or they are foolishly risking pissing off Congress, which can subpoena them, and seems happy to do exactly that in other situations - and they would do so knowing that it's eventually going to come out that they withheld evidence?!?
Again, it's winter, people get sick, that's very weak Bayesian evidence of an outbreak, at best. On priors, how many people at an institute that size get influenza every month during the winter?
And the fact that it was only 3 people, months earlier, seems to indicate moderately strongly it wasn't the source of the full COVID-19 outbreak, since if it were, given the lack of precautions against spread at the time, if it already infected 3 different people, it seems likely it would have spread more widely within China starting at that time.
Sorry, I'm having trouble following. You're saying that 1) it's unlikely to be a lab leak known to US Intel because it would have been known to us via leaks, and 2) you think that Intel agencies have evidence about WIV employees having COVID and that it's being withheld?
First, I think you're overestimating both how much information from highly sensitive sources would leak, and how much Chinese leaders would know if it were a lab leak. This seems on net to be mostly uninformative.
Second, if they have evidence about WIV members having COVID, (and not, you know, any other respiratory disease in the middle of flu/cold season,) I still don't know why you think you would know that it was withheld from congress. Intel agencies share classified information with certain members of Congress routinely, but you'd never know what was or was not said. You think a lack of a leak is evidence that would have been illegally withheld from congress - but it's not illegal for Intel agencies to keep information secret, in a wide variety of cases.
And on that second point, even without the above arguments, not having seen such evidence publicly leaked can't plausibly be more likely in a world where it was a lab leak that was hidden, than it would be in a world where it wasn't a lab leak and the evidence you're not seeing simply doesn't exist!
State department isn't part of "US intelligence agencies and military," and faces very, very different pressures. And despite this, as you point out there are limits to internal pressures in intel agencies - which at least makes it clear that the intel agencies don't have strong and convincing non-public evidence for the leak hypothesis.
I'm not saying it's impossible, I'm saying it's implausible. (So if this is a necessary precondition for believing in a lab leak, it is clear evidence against it.)
"(and likely parts of the US intelligence agencies and military) desperately wanted this to not be a lab leak."
As I said in another comment, that seems very, very hard to continue to believe, even if it might have seemed plausible on priors.
Whoever publishes or sends out notices may or may not have others they check with. That's sometimes the local health authority directly, but may go through the national government. I don't know enough about how that works in China to say in general who might have been able to tell Wuhan Municipal Health Committee or WCDC what they were and were not supposed to say when they made their announcements. However, we have lots of information about what was said in the public statements and hospital records from that time, most of which is mentioned here. (You don't need to trust him much, the descriptions of the systems and what happened when are well known.) But data is also disseminated informally through lots of channels, and I don't know who would have been getting updates from colleagues or sending data to the WHO or US CDC.
But the government would need to have started the coverup while they were suppressing evidence. It's weird to think they simultaneously were covering up transmission, and faking the data about the cases to make it fit the claim it originated in the wet market.
Most very large changes is viral evolution is lateral transfer between viruses, rather than accumulation of point mutations. The better claim would be that this was acquired by a proto-SARS-CoV-2 virus that way, not that it was the result of cross-species changes alone.
Strong evidence against this theory and predictions is the actual statements by intel orgs, which were notably less skeptical than other sources of lab origin.
Looks like such data doesn't exist, and post-2020 wildlife trading ban, new data won't tell us anything about pre-ban conditions - but we know there is lots of cross-border and long distance transport of wildlife. See, for example, this. And elsewhere in Asia, we see similar descriptions of very large volume of wildlife trade over long distances.
Partly disagree - the relevant question isn't distance, it's the amount of wildlife from specific places. New York is further from Atlanta than from Litchfield, CT, but there are more people from Atlanta in New York at any given time. And we know that there's a lot of trade in wildlife in Wuhan from distant places, which is the critical question.
For the future, perhaps this once again updated link may help: Updated link
Citation: LEDWIG, Marion, 2000. Newcomb's problem [Dissertation]. Konstanz: University of Konstanz
"Regarding surgery there are immediate outcomes and they are treating a specific problem so it is easy to measure"
There are longer term impacts of surgeries, though, that aren't easily measured, and surgeries tend to score far more poorly on those fronts. Fixing a (sometimes minor) problem in ways that make people need more healthcare later is at best a mixed bag.
One problem with this analysis is that the FDA actually regulates a ton of what happens in surgeries, which sharply reduces how much this example actually proves. Specifically, they approve all of the devices that get used during surgery, from the scalpels to the monitoring equipment to the surgical sponges, all the laparoscopic instruments used in modern surgeries, and all the replacement joints and implanted or assistive devices which are being put in in many surgeries. Yes, this is different than regulating the surgery itself, but it manages to cut off a huge range of surgical interventions that could be more dangerous or less reliable, while it raises costs and slows innovation. So I'm less sure how well surgery serves as an example of working well without the FDA.
This is not talking about pre-tax investment.
Please do not advocate violating the law in order to "do good" on the forum.
You're making a far different claim than I was - that everything using statistics for decisionmaking is not software. And I'd mostly agree, but don't think AI is particularly similar to statistics either, and the two both need intuition and understanding, but the intuitions needed differ greatly, so the point seems separate.
Yes, I think you're now saying something akin to what I was trying to say. The AI, as a set of weights and activation funtions, is a different artifact than the software being used to multiply the matrices, much less the program used to output the text. (But I'm not sure this is quite the same as a different level of abstraction, the way humans versus atoms are - though if we want to take that route, I think gjm's comment about humans and chemistry makes this clearer.)
In manufacturing or other high-reliability processes, human inputs are components which need the variability in their outputs controlled. I'm drawing a parallel, not saying that AI is responsible for humans.
And I'm completely unsure why you think that engineered systems that use AI are being built so carefully, but regardless, it doesn't mean that the AI is controlled, just that it's being constrained by the system. (And to briefly address your points, which aren't related to what I had been saying, we know how poorly humans take to being narrowly controlled, so to the extent that LLMs and similar systems are human-like, this seems like a strange way to attempt to ensure safety with increasingly powerful AI.)
I'm not talking about AI doom right now, I'm talking about understanding the world. At the system level, you claim we need to design the overall system to purpose, and that's fine for controlling a system - but the same goes for when humans are part of the system, and we're using six sigma to reduce variance in process outcomes. And at that point, it's strange to say that more generally, humans are just components to control, and we can ignore that they are different than actuators and motors, or software scripts. Instead, we have different categories - which was the point originally.
I'm not saying that I can force breaking of category boundaries, I'm asking whether the categories are actually useful for thinking about the systems. I'm saying it isn't, and we need to stop trying to use categories in this way.
And your reply didn't address they key point - is the thing that controls the body being shown in the data being transmitted software, or data? And parallel to that, is the thing that controls the output of the AI system software or data?
Yes, this is part of the intuition I was trying to get across - thanks!
All of the abstractions you are using are just as relevant to system design for a factory, or a construction project. You could do all of the things you're discussing about system design for a mechanical system! And yes, present engineers are treating these systems like software, but to the extent they are doing so, they are missing lots of critical issues, and as we've seen, they keep needing to add on bits to their conceptual model of what they need to deal with in order to have it be relevant. (Which is why I'm saying we need to stop thinknig of it as software.)
So on your final set of questions, I certainly agree that the systems can be analyzed via similar approaches to system engineering, but I think talking about the AI systems as software is far less helpful than saying that are a different thing, with different failure modes, requiring a different conceptual model, and doing your systems engineering on that basis.
I'm asking the question of how we should think about the systems, and claiming "software" is very much the wrong conceptual model. Yes, AI can work poorly because of a software issue, for example, timeouts with the API, or similar. But the thing we're interested in discussing is the AI, not the software component - and as you point out in the case of photography, the user's skill with the software, and with everything else about taking photographs, is something that occurs and should be discussed not in terms of the software being used.
We can play the game of recategorizing certian things, and saying data and software are separate - but the question is whether it adds insight, or not. And I think that existing categories are more misleading than enlightening, hence my claim.
For example, is my face being picked up by the camera during the videoconference "data" in a meaningful sense? Does that tell you something useful about how to have a videoconference? If not, should we call it software, or shift our focus elsewhere when discussing it?
I think th dispute here is that you're interpreting mathematical too narrowly, and almost all of the work happening in agent foundations and similar is exactly what was being worked on by "mathematical AI research" 5-7 years ago. The argument was that those approaches have been fruitful, and we should expect them to continue to be so - if you want to call that "foundational conceptual research" instead of "Mathematical AI research," that's fine..
First, I said I was frustrated that you didn't address the paper, which I intended to mean was a personal frustration, not blame for not engaging, given the vast number of things you could have focused on. I brought it up only because I don't want to make it sound like I thought it was relevant for those reading my comment, to appreciate that this was a personal motive, not a dispassionate evaluation.
Howeer, to defend my criticism, for decisionmakers with finite computational power / bounded time and limited ability to consider issues, I think that there's a strong case to dismiss arguments based on plausible relevance. There are, obviously, an huge (but, to be fair, weighted by a simplicity prior, effectively finite) number of potential philosophies or objections, and a smaller amount of time to make decisions than would be required to evaluate each. So I think we need a case for relevance, and I have two reasons / partial responses to the above that I think explain why I don't think there is such a case.
- There are (to simplify greatly,) two competing reasons for a theory to have come to our attention enough to be considered; plausibility, or interestingness. If a possibility is very cool seeming, and leads to lots of academic papers and cool sounding ideas, the burden of proof for plausibility is, ceteris pariubus, higher.
This is not to say that we should strongly dismiss these questions, but it is a reason to ask for more than just non-zero possibility that physics is wrong. (And in the paper, we make an argument that "physics is wrong" still doesn't imply that bounds we know of are likely to be revoked - most changes to physics which have occurred have constrained things more, not less.)
- I'm unsure why I should care that I have intuitions that can be expanded to implausible cases. Justifying this via intuitions built on constructed cases which work seems exactly backwards.
As an explanation for why I think this is confused, Stuart Armstrong made a case that people fall prey to a failure mode in reasoning that parallels one we see in ML, which I'll refer to as premature rulemaking. In ML, that's seen when a model sees a small sample, and try to build a classification rule based on that, and apply it out of sample; all small black fuzzy object it has seen are cats, and it has seen to cats which are large or other colors, so it calls large grey housecats non-cats, and small black dogs cats. Even moving from that point, it it harder to change away from that mode; we can convince it that dogs are a different category, but the base rule gets expanded by default to other cases, and tigers are not cats, and black mice are, etc. Once we set up the problem as a classifier, trying to find rules, we spend time building systems, not judging cases on their merits. (The alternative he proposes in this context, IIRC, is something like trying to do grouping rather than build rules, and evaluate distance from the cluster rather than classification.)
The parallel here is that people find utilitarianism / deontology / maximizing complexity plausible in a set of cases, and jump to using it as a rule. This is the premature rulemaking. People then try to modify the theory to fit a growing number of cases, ignoring that it's way out of sample for their intuitions. Intuitions then get reified, and people self-justify their new reified intuitions as obvious. (Some evidence for this: people have far more strongly contrasting intuitions in less plausible constructed cases.)
This has gone somewhat off track, I think, but in short, I'm deeply unsure why we should spend time on infinite ethics, have a theory for why people do so, and would want to see strong evidence of why to focus on the topic before considering it useful, as opposed to fun.
This is an important post which I think deserves inclusion in the best-of compilation because despite it's usefulness and widespread agreement about that fact, it seems notto have been highlighted well to the community.
This makes an important point that I find myself consistently referring to - almost none of the confidence in predictions, even inside the rationalist community, is based on actual calibration data. Experts forecast poorly, and we need to stop treating expertise or argumentation as strong stand-alone reasons to accept claims which are implicitly disputed by forecasts.
On the other hand, I think that this post focused far too much on Eliezer. In fact, there are relatively few people in the community who have significant forecasting track records, and this community does tremendously better than most. This leads to lots of strong opinions based on "understanding" which refuse to defer to forecaster expectations or even engage much with why they would differ.
I did not see this post when it was first put on the forum, but reading it now, my personal view of this post is that it continues a trend of wasting time on a topic that is already a focus of too much effort, with little relevance to actual decisions, and no real new claim that the problems were relevant or worth addressing.
I was even more frustrated that it didn't address most of the specific arguments put forward in our paper from a year earlier on why value for decisionmaking was finite, and then put forward seeral arguments we explicitly gave reasons to dismiss - including dismissing ordinal preferences, and positing situations which assume infinities instead of showing they could be relevant to any decision. I think that it doesn't actually engage with reality in ways that are falsifiable, and ignores that even with assuming infinite universes based on physics, the theories which posit infinities in physics are strongly in favor of the view that there is a finite *affectable* universe, making the relevance to ethics hard to justify.
This post publicly but non-confrontationally rebutting an argument that had been put forward and promoted by others was a tremendous community service, of a type we see too rarely, albeit far more often in this community than most. It does not engage in strawmanning, it clearly lays out both the original claim and the evidence, and it attempts to engage positively, including trying to find concrete predictions that the disputing party could agree with.
I think this greatly moved community consensus on a moderately important topic in ways that were very valuable. I will note that it's unfortunate but indicative that SMTM never responded.
Thanks - and the fact that we don't know who is working on relevant things is exactly the reason we're doing this!
We are focused on mathematical research and building bridges between academia and research. I think the pathway to doing that type of research is usually through traditional academic channels, a PhD program, or perhaps a masters degree or a program like MATS, at which point the type of research promotion and academic bridge building we are focused on become far more relevant. That said, we do have undergrad as an option, and are certainly OK with people at any level of seniority signaling their interest.
Do you have any description of your research agenda, or is this just supposed to provide background?