LessWrong 2.0 Reader

That makes sense, I just evaluated the comment in isolation. But I believe that the first few downvotes were as "incorrect" (the red X) rather than regular downvotes (down arrow), which is why the feedback occured to me as simply mistaken (as the comment is not false).

I've noticed, by the way, that most comments posted tend to get downvoted initially and then return to 0 over time. There may be a few regular, highly active users with high standards or something, and less casual users with lower standards which balance them out over time. I've gone to -10 and back before. 

Hi! I joined LW in order to post a research paper that I wrote over the summer, but I figured I'd post here first to describe a bit of the journey that led to this paper.

I got into rationality around 14 years ago when I read a blog called "you are not so smart", which pushed me to audit potential biases in myself and others, and to try and understand ideas/systems end-to-end without handwaving.

I studied computer science at university, partially because I liked the idea that with enough time I could understand any code (unlike essays, where investigating bibliographic claims might lead to dead ends), and also because software pays well. I specialized in machine learning because I thought that algorithms that could make accurate predictions based on patterns in the world that were too complex for people to hardcode were cool. I had this sense that somewhere, someone must understand the "first principles" behind how to choose a neural network architecture, or that there was some way of reverse-engineering what deep learning models learned, just like we could with normal code. Later I realized that there weren't really first principles regarding optimizing training, and that spending time trying to hardcode priors into models representing high-dimensional data was less effective than just getting more data (and then never understanding what exactly the model had learned).

I did a couple of kaggle competitions and wanted to try industrial machine learning. I took a SWE job on a data-heavy team at a tech company working on the ETLs powering models, and then did some backend work which took me away from large datasets for a couple years. I decided to read through recent deep learning textbooks and re-implement research papers at a self-directed programming retreat. Eventually I was able to work on a large scale recommendation system, but I still felt a long way from the cutting edge, which had evolved to GPT-4. At this point, my initial fascination with the field had become tinged with concern, as I saw people (including myself) beginning to rely on language model outputs as if they were true without consulting primary sources. I wanted to understand what language models "knew" and whether we could catch issues with their "reasoning."

I considered grad school, but I figured I'd have a better application if I understood how ChatGPT was trained, and how far we'd progressed in reverse engineering neural networks' internal representations of their training data.

I participated in the AI Safety fundamentals course which covered both of these topics, focusing particularly on the mechanistic interpretability section. I worked through parts of the ARENA curriculum, found an opportunity to collaborate on a research project, and decided to commit to it over the summer, which led to the paper I mentioned in the beginning! I'll link to it below.

I can't think of better explanations.

Why do you expect difficulty thinking of explanations to correlate with the only one you can think of being correct? It seems obvious to me that if you have a general issue with thinking of explanations, the ones you do think of will also be worse than average.

Not sure I understand what you are saying.

If we can construct fake honeypots that are indistinguishable - to the policy - from real opportunities, and do this at scale, then that would suffice: we could drive the policy's P(deployment | great opportunity) very low. If this is what you are saying, I agree.

If we can construct fake attacks that are indistinguishable - to the untrusted monitor - from real attacks, then that possibly suffices: we could drive the monitor's prior P(deployment | attack) very low. (We have to worry about some other sources of evidence, though.) I guess one could call these honeypots for the monitor, and if this is what you are saying, I agree.

If you are saying that we don't need either of these - i.e. that we can get by with honeypots/attacks that are noticeably fake - then that's news to me, and I don't see how you can do that.

I don't think good and evil are objectively real as moral terms, but if something makes us select against certain behaviour, it may be because said behaviour results in organisms deleting themselves from existence. So that "evil" actually means "unsustainable". But this makes it situational (your sustainable expenditure depends on your income, for instance, so spending 100$ cannot be objectively good or evil).

Moral judgments vary between individuals, cultures and societies

Yes, and which actions result in you not existing will also vary. There's no universal morality for the same reason that there's no universal "best food" or "most fitting zoo enclosure", for "best" cannot exist on its own. Calling something "best" is a kind of shortcut, there's implicit things being referred to.
What's the best move in Tetris? The correct answer depends on the game state. When you're looking for "objectively correct universal moral rules" you might also be throwing away the game state on which the answer depends.

I'd go as far as to say that all situations where people are looking for universal solutions are mistaken, as there may (necessarily? I'm not sure) exist many local solutions which are objectively better in the smaller scope. For instance, you cannot design a tool which is the best tool for fixing any machine, instead you will have to create 100s of tools which are the best for each part of each machine. So hammers, saws, wrenches, etc. exist and you cannot unify all of them them to get something which is objectively better than any of them in any situation. But does this imply that tools are not objective? Does it not rather imply that good is a function taking at least two inputs (tool, object) and outputting a value based on the relation between the two? (a third input could be context, i.e. water is good for me in the context that I'm thirsty). 

If my take is right, then like 80% of all philosophical problems turn out to be nonsense. In other words, most unsolved problems might be due to flawed questions. I'm fairly certain in this take, but I don't know if it's obvious or profound.

It is a good summary, but the question is can we generalize it the idea that "I am more likely to be born in the world where life extensions technologies are developing and alignment is easy". Simple Bayesian update does not support this. 

However, if the future measure can somehow "propagate" back in time, it increases my chances to be born in the world where there is logical chances for survival: alignment is easy. 

A simple example of a world-model where measure "propagates" back in time is the simulation argument: if I survive in human-interested AI world, there will be many more my copies in the future who think that I am the past. 

However, there could be more interesting ways for measure to propagate back in time. One of them is that the object of anthropic selection is not observer-moments, but the whole observer-timelines. Another is two-thirder solution to sleeping beauty.  

I'll note that malicious compliance is a very common response to being provided a task that's not straightforwardly possible with the resources available, and no channel to simply communicate that without retaliation.  BS an answer, or technically correct/rules as written response, is often just the best available strategy if one isn't in a position to fix the evaluator's broken incentives.

An actual human's chain of thought would be a lot spicier if their boss ask them to produce a document with working links without providing internet access.

I didn't downvote it, but didn't upvote it either. This answer was posted after other answers that contained this answer as a subset, with deeper analysis.

The way I think of it, it's not quite that some abstractions are cheaper to use than others, but rather:

• One can in-principle reason at the "low(er) level", i.e. just not use any given abstraction. That reasoning is correct but costly.
• One can also just be wrong, e.g. use an abstraction which doesn't actually match the world and/or one's own lower level model. Then predictions will be wrong, actions will be suboptimal, etc.
• Reasoning which is both cheap and correct routes through natural abstractions. There's some degrees of freedom insofar as a given system could use some natural abstractions but not others, or be wrong about some things but not others.

Yeah I'm asking because downvotes are far too ambigious. I think they're ambigious to the point that they don't make for useful feedback (You can't update a worldview for the better if you don't know what's wrong with it). I don't think downvotes are necessarily bad as a concept though. And about humanity - sure, and on any other website I'd largely have agreed with your view, but when I talk about intellectual things I largely push my own humanity to the side. And even if somebody downvotes because of irrational feelings, I'm interested in what those feelings are.

But I know that people on here frequently value truth, and I'm quite brutal to those values as I think truth is about as valid of a concept as a semicolon (the language is just math/logic rather than English). And if we are to talk about Truth with a capital T, then we're speaking about reality, which is more fundamental than language (the territory, reality, is important. But I rarely see any good maps, even on this website. So when taoists seem to suggest throwing the map away entirely, I do think that's a good idea for every day life. It's only for science, research and tech that I value maps). That makes me an outlier though, haha.