the gears to ascenscion's Shortform

[edit: why does this have so many more upvotes than my actually useful shortform posts]

Someone mentioned maybe I should write this publicly somewhere, so that it is better known. I've mentioned it before but here it is again:

I deeply regret cofounding vast and generally feel it has almost entirely done harm, not least by empowering the other cofounder, who I believe to be barely better than e/acc folk due to his lack of interest in attempting to achieve an ought that differs from is. I had a very different perspective on safety then and did not update in time to not do very bad thing. I expect that if you and someone else are both going to build something like vast, and theirs takes three weeks longer to get to the same place, it's better to save the world those three weeks without the improved software. Spend your effort on things like lining up the problems with QACI and cannibalizing its parts to build a v2, possibly using ideas from boundaries/membranes, or generally other things relevant to understanding the desires, impulses, goals, wants, needs, objectives, constraints, developmental learning, limit behavior, robustness, guarantees, etc etc of mostly-pure-RL curious-robotics agents.

incidentally, I've had many conversations with GPT4 where I try to get it to tell me what difference it thinks justifies its (obviously reward-induced and therefore at-least-somewhat-motivated-reasoning) claim that it's not like humans, and the only justification it consistently gives is continuous-time lived experience vs discrete-time secondhand textual training data. I feel like video models and especially egocentric robotics video models don't have that difference...

[edit: pinned to profile]

I feel like most AI safety work today doesn't engage sufficiently with the idea that social media recommenders are the central example of a misaligned AI: a reinforcement learner with a bad objective with some form of ~online learning (most recommenders do some sort of nightly batch weight update). we can align language models all we want, but if companies don't care and proceed to deploy language models or anything else for the purpose of maximizing engagement and with an online learning system to match, none of this will matter. we need to be able to say to the world, "here is a type of machine we all can make that will reliably defend everyone against anyone who attempts to maximize something terrible". anything less than a switchover to a cooperative dynamic as a result of reliable omnidirectional mutual defense seems like a near guaranteed failure due to the global interaction/conflict/trade network system's incentives. you can't just say oh, hooray, we solved some technical problem about doing what the boss wants. the boss wants to manipulate customers, and will themselves be a target of the system they're asking to build, just like sundar pichai has to use self-discipline to avoid being addicted by the youtube recommender same as anyone else.

This will be my last comment on lesswrong until it is not possible for post authors to undelete comments. [edit: since it's planned to be fixed, nevermind!]

originally posted by a post author:

This comment had been apparently deleted by the commenter (the comment display box having a "deleted because it was a little rude, sorry" deletion note in lieu of the comment itself), but the ⋮-menu in the upper-right gave me the option to undelete it, which I did because I don't think my critics are obligated to be polite to me. (I'm surprised that post authors have that power!) I'm sorry you didn't like the post.

some youtube channels I recommend for those interested in understanding current capability trends; separate comments for votability. Please open each one synchronously as it catches your eye, then come back and vote on it. downvote means not mission critical, plenty of good stuff down there too.

I'm subscribed to every single channel on this list (this is actually about 10% of my youtube subscription list), and I mostly find videos from these channels by letting the youtube recommender give them to me and pushing myself to watch them at least somewhat to give the cute little obsessive recommender the reward it seeks for showing me stuff. definitely I'd recommend subscribing to everything.

Let me know which if any of these are useful, and please forward the good ones to folks - this short form thread won't get seen by that many people!

edit: some folks have posted some youtube playlists for ai safety [LW · GW] as well.

[humor]
I'm afraid I must ask that nobody ever upvote or downvote me on this website ever again:

So copilot is still prone to falling into an arrogant attractor with a fairly short prompt that is then hard to reverse with a similar prompt: reddit post

things upvotes conflates:

• agreement (here, separated - sort of)
• respect
• visibility
• should it have been posted in the first place
• should it be removed retroactively
• is it readably argued
• is it checkably precise
• is it honorably argued
• is it honorable to have uttered at all
• is it argued in a truth seeking way overall, combining dimensions
• have its predictions held up
• is it unfair (may be unexpectedly different from others on this list)

(list written by my own thumb, no autocomplete)

these things and their inversions sometimes have multiple components, and many people only allow some of these labels to be applied by close friends, or under specific conditions

should I post this paper as a normal post? I'm impressed by it. if I get a single upvote as shortform, I'll post it as a full fledged post.
Interpreting systems as solving POMDPs: a step towards a formal understanding of agency
Martin Biehl, N. Virgo
Published 4 September 2022
Philosophy
ArXiv
. Under what circumstances can a system be said to have beliefs and goals, and how do such agency-related features relate to its physical state? Recent work has proposed a notion of interpretation map , a function that maps the state of a system to a probability distribution representing its beliefs about an external world. Such a map is not com-pletely arbitrary, as the beliefs it attributes to the system must evolve over time in a manner that is consistent with Bayes’ theorem, and conse-quently the dynamics of a system constrain its possible interpretations. Here we build on this approach, proposing a notion of interpretation not just in terms of beliefs but in terms of goals and actions. To do this we make use of the existing theory of partially observable Markov processes (POMDPs): we say that a system can be interpreted as a solution to a POMDP if it not only admits an interpretation map describing its beliefs about the hidden state of a POMDP but also takes actions that are optimal according to its belief state. An agent is then a system together with an interpretation of this system as a POMDP solution. Although POMDPs are not the only possible formulation of what it means to have a goal, this nevertheless represents a step towards a more general formal definition of what it means for a system to be an agent.

reply to a general theme of recent discussion - the idea that uploads are even theoretically a useful solution for safety:

• the first brain uploads are likely to have accuracy issues that amplify unsafety already in a human.
• humans are not reliably in the safety basin - not even (most?) of the ones seeking safety. in particular, many safety community members seem to have large blindspots that they defend as being important to their views on safety; it is my view that yudkowsky has given himself an anxiety disorder and that his ongoing insights are not as high quality as they seem to him. this is not to claim he is reliably wrong, merely that I wouldn't trust him to do compressive self-distillation because I think he'd make the same mistakes he fears an initially partially aligned AI would. humans have adversarial example vulnerability too.
• the first brain uploads are likely to not be faster than a human, as humans are already very thermally efficient for the computations they're running. improved connectivity might be able to distill down to a much smaller, higher-accuracy network - but then we're reintroducing the compressive self-distillation commonly known as "self improvement", which is a significant fraction of the worry around the transition from soft asi to hard asi anyway.

I have the sense that it's not possible to make public speech non-political, and in order to debate things in a way that doesn't require thinking about how everyone who reads them might consider them, one has to simply write things where they'll only be considered by those you know well. That's not to say I think writing things publicly is bad; but I think tools for understanding what meaning will be taken by different people from a phrase would help people communicate the things they actually mean.

Would love if strong votes came with strong encouragement to explain your vote. It has been proposed before that explanation be required, which seems terrible to me, but I do think it should be very strongly encouraged by the UI that votes come with explanations. Reviewer #2: "downvote" would be an unusually annoying review even for reviewer #2!

random thought: are the most useful posts typically karma approximately 10, and 40 votes to get there? what if it was possible to sort by controversial? maybe only for some users or something? what sorts of sort constraints are interesting in terms of incentivizing discussion vs agreement? blah blah etc

Everyone doing safety research needs to become enough better at lit search that they can find interesting things that have already been done in the literature without doing so adding a ton of overhead to their thinking. I want to make a frontpage post about this, but I don't think I'll be able to argue it effectively, as I generally score low on communication quality.

[posted to shortform due to incomplete draft]

I saw this paper and wanted to get really excited about it at y'all. I want more of a chatty atmosphere here, I have lots to say and want to debate many papers. some thoughts :

seems to me that there are true shapes to the behaviors of physical reality[1]. we can in fact find ways to verify assertions about them[2]; it's going to be hard, though. we need to be able to scale interpretability to the point that we can check for implementation bugs automatically and reliably. in order to get more interpretable sparsity, I think we need models 100x larger to do the same thing, so that every subnetwork is doing a coherent operation to only its inputs with no interference. then, we can pass type information in from sensors and do formal verification that the implemented coordination of the learned network's components only propagate energy in ways that conserves properties at every step. that basic component then would free us from any adversarial examples to that property. we might even be able to constrain architecture by the property, so that you can't even pass through a given broken representation.

given ability to error check a property precisely, we can then talk about formally verifying coordination systems. this is where open problems in open source game theory come in. when models can formally verify things about each other, what happens? would the models still cooperate with models they can't verify are being honest about their source code? how do we avoid sudden weird-as-fuck domain generalization errors that result from the difference between agents that can be verified and agents that cannot?

so then that means using the very best of formal verification to check that there aren't bad coordination patterns in the network (if you take every neuron to be a separate module). what statement can you check that doesn't leave the statement's fingerprints on the agents? seems like it's something about providing freedom from unwanted aesthetic interference. which means that every subcomponent of an ai you're creating needs to have a way to calculate whether a behavior would satisfy the usefulness constraints that nearby agentic shapes in the universe want out of it. there are many types of reasoning errors one can make that incorrectly represent the state of a material system, but if you can accurately simulate a cell and verify a statement about its behavior, you can formally verify whether the cell dies.

I think one key litmus test of any alignment idea is whether it's easy to explain how it also aligns a coordination network of human cells against cancer, or of a network of humans against . on a really deep level, I don't think alignment is anything more than the process of trying to solve evolutionary game theory among [ais, humans]. historically, with high death rate, many humans have picked defect strategies[3]; when we compare to "the ai does not love you, nor does it hate you, but you are made of atoms the ai can use for something else" quote to hate between humans, it seems like with humans, hate is when one wants to use those humans for something besides their own life-aesthetic desires! love for a person or thing is a behavior in the territory as long as the person doing the acting has the capability to accomplish the behavior.

the question is what pruning would occur after these coming capability advancements. my hope is that we can ensure that the pruning is sub-organism edits as much as possible and that all edits are by consent and work by simply constraining the violators, and that we preserve all extant life-self-agents, even if we don't give every one of them as much control over other matter. the question is, given the possibility that some subagents will act adversarially, how do we ensure the overall network of agents can detect malice and demand the malicious agent be interacted with using malice-safety gloves until the agent has self-modified to become a mental shape that reduces malice.

(will edit to improve citations, check back in a couple hours if you don't want to hunt down each paper by name)

[1] many things to cite for why I think this: neural radiance fields' 3d prior; relative representations paper linked above; quantum/quantum chemistry/fluid/multimaterial/coupled-dynamical-systems simulators of various shapes; geometry of neural behavior video; cybernetics; systems science; [2] many more things to cite for why I think this: [3]

my shortform's epistemic status: downvote stuff you disagree with, comment why. also, hey lw team, any chance we could get the data migration where I have agreement points in my shortform posts?

I hate to be That Guy, but are you aware that the usual spelling is "ascension"?

most satisficers should work together to defeat most maximizers most of the way

[edit: intended tone: humorously imprecise]

comment I decided to post out of context for now since it's rambling:

formal verification is a type of execution that can backtrack in response to model failures. you're not wrong, but formally verifying a neural network is possible; the strongest adversarial resistances are formal verification and diffusion; both can protect a margin to decision boundary of a linear subnet of an NN, the formal one can do it with zero error but needs fairly well trained weights to finish efficiently. the problem is that any network capable of complex behavior is likely to be representing a complex enough system that you can't even in principle verify the whole thing because there is no correct simulation that is friendly to your verifier - reality just has too much entropy and so you can't check a property of a physics model of that region.

but, uncheckability is itself often a trait of overly chaotic regions of latent phase spaces. multiscale modeling with local internal constraints might be able to scale this kind of formal adversarial example resistance. alternatively if we can encode a friendliness-seeking process via diffusion, that works too. cite "gradient descent is not all you need", cite formal verification paper

in both cases the question is, is there a metric about the internal function of the network that can represent some minimum desiderata necessary to get our friendliness seeking process within range of our verifier? eg an l1 or soft l0 loss, perhaps weighted by a game theoretic analysis, I still need to look up what a shapley value actually is.

ultimately you want your system built out of parts that each can be verified to error out if used in symbolic behavior they detect as having unfriendly dynamics, so what defines unfriendly dynamics? this is where the dynamical systems view of safety comes in[citation needed], and also relates well to memory safety, see rustlang. I need to look more into what the folks talking about shard safety are getting at, but I suspect it's a similar concept: you want your network of material interactions, whether within a neural network or outside of it, to seek efficient simplicity in order to encode the same thing; the efficiency prior, aka compression applied to total compute, also partially optimizes non-interference.

so while it's not a full basis for morality afaict and insufficient for full RL strength safety, it seems to me like we could at least bound the local direction in world model state space of the local pareto frontier of efficiency-without-additional-moral-loss by asserting that wasted energy burn is obviously to us humans a clear moral loss and that that energy should be spent on efficient thought towards whatever your material values are; this both means lower internal interference (used as an auxiliary objective) and means the model is closer to a basic standard of morality. that means any energy burn that clearly interferes with memory, such as fire, disease, or death, is a clear error state in our neural multiscale quantum chemistry model unless outweighed by better use of that energy to preserve self-process. yudkowsky seems to think this isn't verifiable; I don't see any reason why not other than "it's hard" or "I have no idea how to write a proof of how we make everything protect that everything protects itself and others". but we should be able to verify that really big world models don't contain self-interference behavior, and as far as I can tell, improved structured model compression should both inherently make it easier and inherently optimize a basic moral good of non-waste just by making the system more efficient.

so then the question becomes how to encode memory of other life and rights for which life gets to continue to add memory to the global log of time. assuming we've verified that our physics simplifier doesn't ever forget anything below noise (more precise definition of information usability needed), we still need to verify what makes a control feedback system morally significant. I suspect all control feedback systems have moral significance of some degree. the question is how much and what processes should continue vs simply be remembered in frozen state.

re: lizardman constant, apparently:

The lizard-people conspiracy theory was popularized by conspiracy theorist David Icke

Contemporary belief in reptilians is mostly linked to British conspiracy theorist David Icke, who first published his book "The Biggest Secret" in 1998. Icke alleged that "the same interconnecting bloodlines have controlled the planet for thousands of years," as the book's Amazon description says. The book suggests that blood-drinking reptilians of extraterrestrial origin had been controlling the world for centuries, and even originated the Illuminati — a fictitious group of world leaders that conspiracy theorists say control the world.

Icke has long been accused of anti-Semitism, as his writings on the reptilian conspiracy theory are clearly evocative of the centuries-old blood-libel conspiracy theory, which alleged that a cabal of Jews were controlling the world and drinking the blood of Christian children. He has denied being anti-Semitic, The Guardian reported in 2001. Insider sent a message to Icke via the media-request contact form on his website, but had not heard back at press time.

found the above post by googling, also ran into this one on how to avoid the antisemetic trope, looking up both of which were inspired after seeing a comment on bsky (account required)

Connor Leahy interviews are getting worse and worse public response, and I think it's because he's a bad person to be doing it. I want to see Andrew critch or John Wentworth as the one in debates.

while the risk from a superagentic ai is in fact very severe, non-agentic ai doesn't need to eliminate us for us to get eliminated, we'll replace ourselves with it if we're not careful - our agency is enough to converge to that, entirely without the help of ai agency. it is our own ability to cooperate we need to be augmenting; how do we do that in a way that doesn't create unstable patterns where outer levels of cooperation are damaged by inner levels of cooperation, while still allowing the formation of strongly agentic safe co-protection?

https://atlas.nomic.ai/map/01ff9510-d771-47db-b6a0-2108c9fe8ad1/3ceb455b-7971-4495-bb81-8291dc2d8f37 map of submissions to iclr

"What's new in machine learning?" - youtube - summary (via summarize.tech):

• 00:00:00 The video showcases a map of 5,000 recent machine learning papers, revealing topics such as protein sequencing, adversarial attacks, and multi-agent reinforcement learning.
• 00:05:00 The YouTube video "What's New In Machine Learning?" introduces various new developments in machine learning, including energy-based predictive representation, human level performance on Atari games, and more.
• 00:10:00 In this video, the author discusses some of the new developments in machine learning, including zero shot reinforcement learning and successor representations.
• 00:15:00 The video discusses current machine learning research, including work on intrinsically motivated learning and differential privacy.
• 00:20:00 This video discusses recent developments in machine learning, including adversarial training, adaptive neuron selection, and label correction.
• 00:25:00 This video discusses the new features of machine learning, including the Chain of Thought, visual reasoning, and scene reconstruction. It also covers low-compute areas, such as dynamic Valkyrie and point-based representations.
• 00:30:00 In this video, the author discusses some of the newest advances in machine learning, including ultra realistic singing voices and speed. They also mention some of the more popular applications of machine learning, such as reinforcement learning and graph neural networks.
• 00:35:00 In this video, different types of machine learning are covered, including adaptive gradients, networks, event-based classification, and meta learning. Surprisingly, binary neural networks are shown to be very efficient in accelerating neural network inference.
• 00:40:00 In this video, the presenter discusses some of the new developments in machine learning, including online learning and self-supervised learning. They also mention a paper on meta learning.

if I should have posted this as a post, please reply and say so. I've no idea how much of the stuff I find interesting is useful to others or how much to broadcast it.

[tone: humorous due to imprecision]

broke: effective selfishness
woke: effective altruism
bespoke: effective solidarity
masterstroke: effective multiself functional decision theoretic selfishness

a bunch of links on how to visualize the training process of some of today's NNs; this is somewhat old stuff, mostly not focused on exact mechanistic interpretability, but some of these are less well known and may be of interest to passers by. If anyone reads this and thinks it should have been a top level post, I'll put it up onto personal blog's frontpage. Or I might do that anyway if I think I should have tomorrow.

https://metaphor.systems/search?q=cool%20paper%20visualizing%20the%20trajectory%20of%20representations%20in%20the%20process%20of%20training

• https://metaphor.systems/search?q=incredible+visualization+of+the+trajectory+of+representations+in+the+process+of+training+a+neural+network ->
• https://zacgeis.github.io/visualizing-deep-learning/ (precise dataflow of the numerics of the basics)
• http://joshvarty.github.io/VisualizingRNNs/ (precise dataflow of rnns)
• https://ml4a.github.io/ml4a/visualizing_convnets/ (old, but solid demo of what the basics of vision nets learn)
• https://arogozhnikov.github.io/3d_nn/ (intuition)
• https://www.bayeswatch.com/assets/ginn/good3.html -> https://www.bayeswatch.com/2018/09/17/GINN/ (more precise intuition)
• https://distill.pub/2019/activation-atlas/ (intuition for modern enormous vision models)
• https://distill.pub/2019/memorization-in-rnns/ (iffy that it gives much intuition at all, but kinda cool I guess)
• https://bigredt.github.io/2017/04/21/deepvis/ (interesting intuition, covers more stuff, somewhat precise, not sure I quite followed what's going on)
• https://unboxresearch.com/articles/randnn.html (intuition for what you can do with neural networks as a shader in 2d and how it builds up, this is more like the bunny one below)
• https://playground.tensorflow.org/ (train in browser)
• https://github.com/scottgigante/M-PHATE (trajectory of representations intuition)

--- not from metaphor: ---

• https://div-lab.github.io/dendromap/ (tool for visualizing an image dataset)
• https://paperswithcode.com/paper/neuromapper-in-browser-visualizer-for-neural / https://poloclub.github.io/NeuroMapper/ (tool for visualizing training trajectory in high-dim nets)
• https://visxai.io/, https://distill.pub/, https://transformer-circuits.pub/ - modern internals vis research

some matmul-nonlinear shaders that show walking around weight-interference space:

1. before messing with the parameters:
   1. generators (3d + fractal dimension): https://www.shadertoy.com/view/Xtf3Rn
   2. basic fractal (2d + fractal dimension): https://www.shadertoy.com/view/tltSWs
   3. fractal pyramid (3d + fractal dimension): https://www.shadertoy.com/view/tsXBzS
2. after messing with the parameters:
   1. generators (added sin(t) linear offsets and scales to show how the linear portions look as you mess with layer weights) https://www.shadertoy.com/view/7d3fDs
   2. basic fractal (sin(t) animation with different amounts at different fractal layers): https://www.shadertoy.com/view/sdGBR1
   3. fractal pyramid: (slowed down main anim, added animation to bias, added several additional animations to other weights) https://www.shadertoy.com/view/7ddfWl

there are lots of other shadertoy examples of ways to visualize programs breaking, but these ones are interesting because they're all raytracers of field(xyz) => signed_distance that use a linear-and-fold transformation comparable to relu nets. useful for getting an intuition what even higher dimensional effects might look like; not sufficient to cover the space of behavior in high dimensions, though!

Modeling Strong and Human-Like Gameplay with KL-Regularized Search - we read this one on the transhumanists in vr discord server to figure out what they were testing and what results they got. key takeaways according to me, note that I could be quite wrong about the paper's implications:

• Multi-agent game dynamics change significantly as you add more coherent search and it becomes harder to do linear learning to approximate the search. (no surprise, really.)
• it still takes a lot of search.
• guiding the search is not hopeless in the presence of noise!
• shallow, no-planning equilibrium in imperfect-information games is not improved in emulating a reference policy as much? this seems to make sense as an additional confirmation of the basic hypothesis "search helps model searching beings".

index of misc tools I have used recently, I'd love to see others' contributions - if this has significant harmful human capability externalities let me know:

basic:

• linked notes: https://logseq.com/ - alternatives I considered included obsidian, roamresearch, athensresearch, many others; logseq is FOSS, agpl, works with local markdown directories, is clojure, is a solid roam clone with smoother ui, did I mention free
• desktop voice control: https://talonvoice.com/ - patreon-funded freeware. voice control engine for devs. configured with nice code. easier install than last one I used (caster+{dragon,kaldi}); better command accuracy than whisper, solid if janky help menu, , good default configurations available, dev is working on whisper integration. you don't need to be unable to use your hands to benefit, i switch back and forth. needs a solid mic, though - generally humans and ai agree about what's hard to hear, so if it works to talk to humans, it'll work ok for this.
• https://elicit.org/ scholar search and preset prompts for gpt3
• https://kagi.com/ and https://teclis.com/ - search that doesn't suck at finding bespoke and high quality websites, I have them bound to both "kagi hunt", "teclis hunt" on my talon voice commands, as well as "k" and "t" in my chrome search bar (these two are basically the only winners out of search engines with their own indexes, imo)
• https://millionshort.com - another search engine for finding less well known stuff
• https://www.semanticscholar.org/ - paper search and related papers discovery, I love love love the related papers it offers after you've added papers to library
• https://www.summarize.tech/ - youtube video summarizer, good for checking if I want to watch something or helping me explain to others why I think something was worth the watch. I wish I had something like this for text, probably not hard to do; doesn't PEGASUS beat gpt3 summary anyway?
• https://hovancik.net/stretchly/ - break timer to avoid overusing computer, I'm still searching for what settings are comfy tho
• https://web.hypothes.is/ - overlay tool for publicly adding annotations to any site, somewhat popular to annotate ml pds and there are such interesting networks of people labeling each other's stuff that you can browse on the site. can be imported to logseq through a plugin, though it doesn't batch import all hypothesis annotations at the moment. I was using it heavily for a while, might do so again; it's nice being able to take notes in browser and import them to logseq later, and it's nice that it's easy-ish to share them with others. I wish it were easier.

newly discovered or rediscovered while browsing my behavior to find what tools I've used recently:

• https://app.scraft.ai/ - LLM writing framework for question answering to write essays? language engine probably gpt3 but not confirmed - looks like a fairly well thought out workflow, the ui asks you questions to prompt you as the primary interaction method
• https://www.genei.io/ - LLM research summarizer - been meaning to try it more deeply, apparently it summarizes papers more thoroughly than elicit; no trial or free version though
• https://iris.ai/ - similar, looks like it has a free version, seems like an interesting variation of paper discovery
• https://researchrabbitapp.com/ seems really cool and I tried it, seems like it might be missing a lot of papers, but otherwise very good for finding key work in a field. probably better to just open lots of tabs browsing works citing this one ("citations") and works cited by this one ("references") using semanticscholar. eg, things citing "concrete problems in ai safety"

given the way shortforms aren't very well surfaced anywhere, should this have been a blog post?

btw neural networks are super duper shardy right now. like they've just, there are shards everywhere. as I move in any one direction in hyperspace, those hyperplanes I keep bumping into are like lines, they're walls, little shardy wall bits that slice and dice. if you illuminate them together, sometimes the light from the walls can talk to each other about an unexpected relationship between the edges! and oh man, if you're trying to confuse them, you can come up with some pretty nonsensical relationships. they've got a lot of shattery confusing shardbits all sharding things up into little fragments, tiny flecks of shard, and they've got the surface of the shards. some of the shards are far away from each other, sharded off with max(), but sometimes they flip over an edge they didn't see coming and a different shard wakes up as some energy moves into the subdimensions that its decision boundaries shard. language is funky because there are lots of different shards between most words, and yet there's a lot of contextual shard selection that is highly shared. but it's not really that different than how shardy the physical room around you is. I notice some funky things about the shardyness of a room though, in comparison to the shardiness of flat hyperplanes. and NeRF architectures agree with me: plain NeRF is weird and unnatural, but when you shape the computation dataflow so the shards are naturally cutting in spaces that already mostly fit the dataflow shape eg 3dness, shardyness lets backprop gradient descent discover interacting shards for the data that summarize it well.

I need to post those 3d fractal links on here.

They very much can be dramatically more intelligent than us in a way that makes them dangerous, but it doesn't look how was expected - it's dramatically more like teaching a human kid than was anticipated.

Now, to be clear, there's still an adversarial examples problem: current models are many orders of magnitude too trusting, and so it's surprisingly easy to get them into subspaces of behavior where they are eagerly doing whatever it is you asked without regard to exactly why they should care.

Current models have a really intense yes-and problem: they'll happily render what you ask for. If their training target set includes bad behavior, they'll happily replicate it if your input resonates in a way that constructively interferes with its training targets that required a bad behavior, and it'll output that bad behavior. If you'd like to get an ai to be aligned, you need to parent it, primarily by managing its training targets. but this is not that hard as long as there are enough contributors. For example, see novelai - try playing with neox20b - to get a sense of what these machine kiddos can do. The next generations - gflownets, the s4 sequence model, etc - will probably improve compression quality. but they're not going to improve at the rate yudkowsky expected for a bit. I'm expecting that by jan 1, but that's basically forever - after all, time slows down when you get closer to a singularity, right?

yudkowsky has sort of been terrified over nothing and sort of not, I suspect this may have been due to misunderstanding how training data gets into a learning system 25 years ago and updating on it way too hard - he's not totally wrong, but the self-improving system is looking to mostly be the entire economy with wide networking between many intelligence modules across many beings, just as it already is. the problem with it fundamentally boils down to a split between those who think that economic systems and machines are best used to keep some people under the machine, and those who think that our upcoming powers of constructivism should be shared with everyone, modulo solving the game theory of how much total energy to spend per minute per person.

we're not going to get drexlerian nanotech this year, calm down.

thats probably next year haha

anyway, tell your local ai research lab that formal verification is absolutely within reach

General note: changed my name: "the gears to ascension" => "Lauren (often wrong)".

a comment thread of mostly ai generated summaries of lesswrong posts so I can save them in a slightly public place for future copypasting but not show up in the comments of the posts themselves

Here's a ton of vaguely interesting sounding papers on my semanticscholar feed today - many of these are not on my mainline but are very interesting hunchbuilding about how to make cooperative systems - sorry about the formatting, I didn't want to spend time format fixing, hence why this is in shortform. I read the abstracts, nothing more [LW · GW].

As usual with my paper list posts: you're gonna want tools to keep track of big lists of papers to make use of this! see also my other posts for various times I've mentioned such tools eg semanticscholar's recommender (which you use by adding papers to folders - it's not on by default if you don't have an account, and I don't mean their search bar), or bring your own.

On the Lipschitz Constant of Deep Networks and Double Descent

• Matteo Gamba, Hossein Azizpour, Marten Bjorkman

Computer Science

ArXiv

• 28 January 2023

Existing bounds on the generalization error of deep networks assume some form of smooth or bounded dependence on the input variable, falling short of investigating the mechanisms controlling such factors in practice. In this work, we present an extensive experimental study of the empirical Lipschitz constant of deep networks undergoing double descent, and highlight non-monotonic trends strongly correlating with the test error. Building a connection between parameter-space and input-space gradients for SGD around a critical point, we isolate two important factors -- namely loss landscape curvature and distance of parameters from initialization -- respectively controlling optimization dynamics around a critical point and bounding model function complexity, even beyond the training data. Our study presents novels insights on implicit regularization via overparameterization, and effective model complexity for networks trained in practice

Efficiency in Collective Decision-Making via Quadratic Transfers

• Jon X. Eguia, Nicole Immorlica, S. Lalley, Katrina Ligett, Glen Weyl, Dimitrios Xefteris

Economics

• 15 January 2023

Consider the following collective choice problem: a group of budget constrained agents must choose one of several alternatives. Is there a budget balanced mechanism that: i) does not depend on the specific characteristics of the group, ii) does not require unaffordable transfers, and iii) implements utilitarianism if the agents’ preferences are quasilinear and their private information? We study the following procedure: every agent can express any intensity of support or opposition to each alternative, by transferring to the rest of the agents wealth equal to the square of the intensity expressed; and the outcome is determined by the sums of the expressed intensities. We prove that as the group grows large, in every equilibrium of this quadratic-transfers mechanism, each agent’s transfer converges to zero, and the probability that the efficient outcome is chosen converges to one.

Eco-evolutionary Dynamics of Non-episodic Neuroevolution in Large Multi-agent Environments

• Hamon Gautier, Eleni Nisioti, Clément Moulin-Frier

Biology, Computer Science

ArXiv

• 18 February 2023

Neuroevolution (NE) has recently proven a competitive alternative to learning by gradient descent in reinforcement learning tasks. However, the majority of NE methods and associated simulation environments differ crucially from biological evolution: the environment is reset to initial conditions at the end of each generation, whereas natural environments are continuously modified by their inhabitants; agents reproduce based on their ability to maximize rewards within a population, while biological organisms reproduce and die based on internal physiological variables that depend on their resource consumption; simulation environments are primarily single-agent while the biological world is inherently multi-agent and evolves alongside the population. In this work we present a method for continuously evolving adaptive agents without any environment or population reset. The environment is a large grid world with complex spatiotemporal resource generation, containing many agents that are each controlled by an evolvable recurrent neural network and locally reproduce based on their internal physiology. The entire system is implemented in JAX, allowing very fast simulation on a GPU. We show that NE can operate in an ecologically-valid non-episodic multi-agent setting, finding sustainable collective foraging strategies in the presence of a complex interplay between ecological and evolutionary dynamics.

Games with possibly naive present-biased players

• M. Haan, D. Hauck

Economics

Theory and Decision

• 17 February 2023

We propose a solution concept for games that are played among players with present-biased preferences that are possibly naive about their own, or about their opponent’s future time inconsistency. Our perception-perfect outcome essentially requires each player to take an action consistent with the subgame perfect equilibrium, given her perceptions concerning future types, and under the assumption that other present and future players have the same perceptions. Applications include a common pool problem and Rubinstein bargaining. When players are naive about their own time inconsistency and sophisticated about their opponent’s, the common pool problem is exacerbated, and Rubinstein bargaining breaks down completely

The Effects of Time Preferences on Cooperation: Experimental Evidence from Infinitely Repeated Games

• Jeongbin Kim

Economics

American Economic Journal: Microeconomics

• 1 February 2023

This paper studies the effects of time preferences on cooperation in an infinitely repeated prisoner’s dilemma game experiment. Subjects play repeated games in the lab, all decisions at once, but stage game payoffs are paid over an extended period of time. Changing the time window of stage game payoffs (weekly or monthly) varies discount factors, and a delay for the first-stage game payoffs eliminates/weakens present bias. First, subjects with weekly payments cooperate more than subjects with monthly payments—higher discount factors promote greater cooperation. Second, the rate of cooperation is higher when there is a delay—present bias reduces cooperation. (JEL C72, C73, D91)

Safety Guarantees in Multi-agent Learning via Trapping Regions

• A. Czechowski, F. Oliehoek

Computer Science

ArXiv

• 27 February 2023

One of the main challenges of multi-agent learning lies in establishing convergence of the algorithms, as, in general, a collection of individual, self-serving agents is not guaranteed to converge with their joint policy, when learning concurrently. This is in stark contrast to most single-agent environments, and sets a prohibitive barrier for deployment in practical applications, as it induces uncertainty in long term behavior of the system. In this work, we propose to apply the concept of trapping regions, known from qualitative theory of dynamical systems, to create safety sets in the joint strategy space for decentralized learning. Upon verification of the direction of learning dynamics, the resulting trajectories are guaranteed not to escape such sets, during the learning process. As a result, it is ensured, that despite the uncertainty over convergence of the applied algorithms, learning will never form hazardous joint strategy combinations. We introduce a binary partitioning algorithm for verification of trapping regions in systems with known learning dynamics, and a heuristic sampling algorithm for scenarios where learning dynamics are not known. In addition, via a fixed point argument, we show the existence of a learning equilibrium within a trapping region. We demonstrate the applications to a regularized version of Dirac Generative Adversarial Network, a four-intersection traffic control scenario run in a state of the art open-source microscopic traffic simulator SUMO, and a mathematical model of economic competition.

Detecting Information Relays in Deep Neural Networks

• A. Hintze, C. Adami

Computer Science

Entropy

• 3 January 2023

Deep learning of artificial neural networks (ANNs) is creating highly functional processes that are, unfortunately, nearly as hard to interpret as their biological counterparts. Identification of functional modules in natural brains plays an important role in cognitive and neuroscience alike, and can be carried out using a wide range of technologies such as fMRI, EEG/ERP, MEG, or calcium imaging. However, we do not have such robust methods at our disposal when it comes to understanding functional modules in artificial neural networks. Ideally, understanding which parts of an artificial neural network perform what function might help us to address a number of vexing problems in ANN research, such as catastrophic forgetting and overfitting. Furthermore, revealing a network’s modularity could improve our trust in them by making these black boxes more transparent. Here, we introduce a new information-theoretic concept that proves useful in understanding and analyzing a network’s functional modularity: the relay information IR. The relay information measures how much information groups of neurons that participate in a particular function (modules) relay from inputs to outputs. Combined with a greedy search algorithm, relay information can be used to identify computational modules in neural networks. We also show that the functionality of modules correlates with the amount of relay information they carry

Scaling in Depth: Unlocking Robustness Certification on ImageNet

• Kaiqin Hu, Andy Zou, Zifan Wang, Klas Leino, Matt Fredrikson

Computer Science

ArXiv

• 29 January 2023

Notwithstanding the promise of Lipschitz-based approaches to deterministically train and certify robust deep networks, the state-of-the-art results only make successful use of feed-forward Convolutional Networks (ConvNets) on low-dimensional data, e.g. CIFAR-10. Because ConvNets often suffer from vanishing gradients when going deep, large-scale datasets with many classes, e.g., ImageNet, have remained out of practical reach. This paper investigates ways to scale up certifiably robust training to Residual Networks (ResNets). First, we introduce the Linear ResNet (LiResNet) architecture, which utilizes a new residual block designed to facilitate tighter Lipschitz bounds compared to a conventional residual block. Second, we introduce Efficient Margin MAximization (EMMA), a loss function that stabilizes robust training by simultaneously penalizing worst-case adversarial examples from all classes. Combining LiResNet and EMMA, we achieve new state-of-the-art robust accuracy on CIFAR-10/100 and Tiny-ImageNet under (cid:96) 2 -norm-bounded perturbations. Moreover, for the first time, we are able to scale up deterministic robustness guarantees to ImageNet, bringing hope to the possibility of applying deterministic certification to real-world applications. We re-lease our code on Github: https://github. com/klasleino/gloro

IQ-Flow: Mechanism Design for Inducing Cooperative Behavior to Self-Interested Agents in Sequential Social Dilemmas

• Bengisu Guresti, Abdullah Vanlioglu, N. K. Ure

Economics

ArXiv

• 28 February 2023

Achieving and maintaining cooperation between agents to accomplish a common objective is one of the central goals of Multi-Agent Reinforcement Learning (MARL). Nevertheless in many real-world scenarios, separately trained and specialized agents are deployed into a shared environment, or the environment requires multiple objectives to be achieved by different coexisting parties. These variations among specialties and objectives are likely to cause mixed motives that eventually result in a social dilemma where all the parties are at a loss. In order to resolve this issue, we propose the Incentive Q-Flow (IQ-Flow) algorithm, which modifies the system's reward setup with an incentive regulator agent such that the cooperative policy also corresponds to the self-interested policy for the agents. Unlike the existing methods that learn to incentivize self-interested agents, IQ-Flow does not make any assumptions about agents' policies or learning algorithms, which enables the generalization of the developed framework to a wider array of applications. IQ-Flow performs an offline evaluation of the optimality of the learned policies using the data provided by other agents to determine cooperative and self-interested policies. Next, IQ-Flow uses meta-gradient learning to estimate how policy evaluation changes according to given incentives and modifies the incentive such that the greedy policy for cooperative objective and self-interested objective yield the same actions. We present the operational characteristics of IQ-Flow in Iterated Matrix Games. We demonstrate that IQ-Flow outperforms the state-of-the-art incentive design algorithm in Escape Room and 2-Player Cleanup environments. We further demonstrate that the pretrained IQ-Flow mechanism significantly outperforms the performance of the shared reward setup in the 2-Player Cleanup environment.

Private Blotto: Viewpoint Competition with Polarized Agents

• Kate Donahue, J. Kleinberg

Economics

ArXiv

• 27 February 2023

Colonel Blotto games are one of the oldest settings in game theory, originally proposed over a century ago in Borel 1921. However, they were originally designed to model two centrally-controlled armies competing over zero-sum"fronts", a specific scenario with limited modern-day application. In this work, we propose and study Private Blotto games, a variant connected to crowdsourcing and social media. One key difference in Private Blotto is that individual agents act independently, without being coordinated by a central"Colonel". This model naturally arises from scenarios such as activist groups competing over multiple issues, partisan fund-raisers competing over elections in multiple states, or politically-biased social media users labeling news articles as misinformation. In this work, we completely characterize the Nash Stability of the Private Blotto game. Specifically, we show that the outcome function has a critical impact on the outcome of the game: we study whether a front is won by majority rule (median outcome) or a smoother outcome taking into account all agents (mean outcome). We study how this impacts the amount of"misallocated effort", or agents whose choices doesn't influence the final outcome. In general, mean outcome ensures that, if a stable arrangement exists, agents are close to evenly spaced across fronts, minimizing misallocated effort. However, mean outcome functions also have chaotic patterns as to when stable arrangements do and do not exist. For median outcome, we exactly characterize when a stable arrangement exists, but show that this outcome function frequently results in extremely unbalanced allocation of agents across fronts

Natural selection for collective action

• Benjamin Allen, Abdur-Rahman Khwaja, J. L. Donahue, Cassidy Lattanzio, Y. Dementieva, C. Sample

Biology

• 28 February 2023

Collective action -- behavior that arises from the combined actions of multiple individuals -- is observed across living beings. The question of how and why collective action evolves has profound implications for behavioral ecology, multicellularity, and human society. Collective action is challenging to model mathematically, due to nonlinear fitness effects and the consequences of spatial, group, and/or family relationships. We derive a simple condition for collective action to be favored by natural selection. A collective's effect on the fitness of each individual is weighted by the relatedness between them, using a new measure of collective relatedness. If selection is weak, this condition can be evaluated using coalescent theory. More generally, our result applies to any synergistic social behavior, in spatial, group, and/or family-structured populations. We use this result to obtain conditions for the evolution of collective help among diploid siblings, subcommunities of a network, and hyperedges of a hypergraph. We also obtain a condition for which of two strategies is favored in a game between siblings, cousins, or other relatives. Our work provides a rigorous basis for extending the notion of ``actor", in the study of social behavior, from individuals to collectives

Systematic Rectification of Language Models via Dead-end Analysis

• Mengyao Cao, Mehdi Fatemi, J. C. Cheung, S. Shabanian

Computer Science

ArXiv

• 27 February 2023

With adversarial or otherwise normal prompts, existing large language models (LLM) can be pushed to generate toxic discourses. One way to reduce the risk of LLMs generating undesired discourses is to alter the training of the LLM. This can be very restrictive due to demanding computation requirements. Other methods rely on rule-based or prompt-based token elimination, which are limited as they dismiss future tokens and the overall meaning of the complete discourse. Here, we center detoxification on the probability that the finished discourse is ultimately considered toxic. That is, at each point, we advise against token selections proportional to how likely a finished text from this point will be toxic. To this end, we formally extend the dead-end theory from the recent reinforcement learning (RL) literature to also cover uncertain outcomes. Our approach, called rectification, utilizes a separate but significantly smaller model for detoxification, which can be applied to diverse LLMs as long as they share the same vocabulary. Importantly, our method does not require access to the internal representations of the LLM, but only the token probability distribution at each decoding step. This is crucial as many LLMs today are hosted in servers and only accessible through APIs. When applied to various LLMs, including GPT-3, our approach significantly improves the generated discourse compared to the base LLMs and other techniques in terms of both the overall language and detoxification performance.

How does HCI Understand Human Autonomy and Agency?

• Dan Bennett, Oussama Metatla, A. Roudaut, Elisa D. Mekler

Psychology

ArXiv

• 29 January 2023

Human agency and autonomy have always been fundamental concepts in HCI. New developments, including ubiquitous AI and the growing integration of technologies into our lives, make these issues ever pressing, as technologies increase their ability to influence our behaviours and values. However, in HCI understandings of autonomy and agency remain ambiguous. Both concepts are used to describe a wide range of phenomena pertaining to sense-of-control, material independence, and identity. It is unclear to what degree these understandings are compatible, and how they support the development of research programs and practical interventions. We address this by reviewing 30 years of HCI research on autonomy and agency to identify current understandings, open issues, and future directions. From this analysis, we identify ethical issues, and outline key themes to guide future work. We also articulate avenues for advancing clarity and specificity around these concepts, and for coordinating integrative work across different HCI communities.

Aligning Robot and Human Representations

• Andreea Bobu, Andi Peng, Pulkit Agrawal, J. Shah, A. Dragan

Computer Science

ArXiv

• 3 February 2023

To act in the world, robots rely on a representation of salient task aspects: for example, to carry a cup of coffee, a robot must consider movement efficiency and cup orientation in its behaviour. However, if we want robots to act for and with people, their representations must not be just functional but also reflective of what humans care about, i.e. their representations must be aligned with humans'. In this survey, we pose that current reward and imitation learning approaches suffer from representation misalignment, where the robot's learned representation does not capture the human's representation. We suggest that because humans will be the ultimate evaluator of robot performance in the world, it is critical that we explicitly focus our efforts on aligning learned task representations with humans, in addition to learning the downstream task. We advocate that current representation learning approaches in robotics should be studied from the perspective of how well they accomplish the objective of representation alignment. To do so, we mathematically define the problem, identify its key desiderata, and situate current robot learning methods within this formalism. We conclude the survey by suggesting future directions for exploring open challenges.

Reinforcement Learning from Diverse Human Preferences

• Wanqi Xue, Bo An, Shuicheng Yan, Zhongwen Xu

Computer Science

ArXiv

• 27 January 2023

The complexity of designing reward functions has been a major obstacle to the wide application of deep reinforcement learning (RL) techniques. Describing an agent’s desired behaviors and prop-erties can be difficult, even for experts. A new paradigm called reinforcement learning from human preferences (or preference-based RL) has emerged as a promising solution, in which reward functions are learned from human preference labels among behavior trajectories. However, existing methods for preference-based RL are lim-ited by the need for accurate oracle preference labels. This paper addresses this limitation by de-veloping a method for crowd-sourcing preference labels and learning from diverse human preferences. The key idea is to stabilize reward learning through regularization and correction in a latent space. To ensure temporal consistency, a strong constraint is imposed on the reward model that forces its latent space to be close to the prior distribution. Additionally, a confidence-based reward model ensembling method is designed to generate more stable and reliable predictions. The proposed method is tested on a variety of tasks in DMcontrol and Meta-world and has shown consistent and significant improvements over existing preference-based RL algorithms when learning from diverse feedback, paving the way for real-world applications of RL methods

The Role of Heuristics and Biases During Complex Choices with an AI Teammate

• Nikolos Gurney, John H. Miller, David V. Pynadath

Biology

ArXiv

• 14 January 2023

Behavioral scientists have classically documented aversion to algorithmic decision aids, from simple linear models to AI. Sentiment, however, is changing and possibly accelerating AI helper usage. AI assistance is, arguably, most valuable when humans must make complex choices. We argue that classic experimental methods used to study heuristics and biases are insufficient for studying complex choices made with AI helpers. We adapted an experimental paradigm designed for studying complex choices in such contexts. We show that framing and anchoring effects impact how people work with an AI helper and are predictive of choice outcomes. The ev- idence suggests that some participants, particularly those in a loss frame, put too much faith in the AI helper and experi- enced worse choice outcomes by doing so. The paradigm also generates computational modeling-friendly data allowing fu- ture studies of human-AI decision making

On The Fragility of Learned Reward Functions

• Lev McKinney, Yawen Duan, David Krueger, A. Gleave

Computer Science, Psychology

ArXiv

• 9 January 2023

Reward functions are notoriously difficult to specify, especially for tasks with complex goals. Reward learning approaches attempt to infer reward functions from human feedback and preferences. Prior works on reward learning have mainly focused on the performance of policies trained alongside the reward function. This practice, however, may fail to detect learned rewards that are not capable of training new policies from scratch and thus do not capture the intended behavior. Our work focuses on demonstrating and studying the causes of these relearning failures in the domain of preference-based reward learning. We demonstrate with experiments in tabular and continuous control environments that the severity of relearning failures can be sensitive to changes in reward model design and the trajectory dataset composition. Based on our findings, we emphasize the need for more retraining-based evaluations in the literature.

Preserving Fairness in AI under Domain Shift

• Serban Stan, Mohammad Rostami

Computer Science

ArXiv

• 29 January 2023

Existing algorithms for ensuring fairness in AI use a single-shot training strategy, where an AI model is trained on an annotated training dataset with sensitive attributes and then fielded for utilization. This training strategy is effective in problems with stationary distributions, where both training and testing data are drawn from the same distribution. However, it is vulnerable with respect to distributional shifts in the input space that may occur after the initial training phase. As a result, the time-dependent nature of data can introduce biases into the model predictions. Model retraining from scratch using a new annotated dataset is a naive solution that is expensive and time-consuming. We develop an algorithm to adapt a fair model to remain fair under domain shift using solely new unannotated data points. We recast this learning setting as an unsupervised domain adaptation problem. Our algorithm is based on updating the model such that the internal representation of data remains unbiased despite distributional shifts in the input space. We provide extensive empirical validation on three widely employed fairness datasets to demonstrate the effectiveness of our algorithm.

Generalized Uncertainty of Deep Neural Networks: Taxonomy and Applications

• Chengyu Dong

Computer Science

ArXiv

• 2 February 2023

Deep neural networks have seen enormous success in various real-world applications. Beyond their predictions as point estimates, increasing attention has been focused on quantifying the uncertainty of their predictions. In this review, we show that the uncertainty of deep neural networks is not only important in a sense of interpretability and transparency, but also crucial in further advancing their performance, particularly in learning systems seeking robustness and efficiency. We will generalize the definition of the uncertainty of deep neural networks to any number or vector that is associated with an input or an input-label pair, and catalog existing methods on ``mining'' such uncertainty from a deep model. We will include those methods from the classic field of uncertainty quantification as well as those methods that are specific to deep neural networks. We then show a wide spectrum of applications of such generalized uncertainty in realistic learning tasks including robust learning such as noisy learning, adversarially robust learning; data-efficient learning such as semi-supervised and weakly-supervised learning; and model-efficient learning such as model compression and knowledge distillation

Modeling Moral Choices in Social Dilemmas with Multi-Agent Reinforcement Learning

• Elizaveta Tennant, S. Hailes, Mirco Musolesi

Computer Science

ArXiv

• 20 January 2023

Practical uses of Artificial Intelligence (AI) in the real world have demonstrated the importance of embedding moral choices into intelligent agents. They have also highlighted that defining top-down ethical constraints on AI according to any one type of morality is extremely challenging and can pose risks. A bottom-up learning approach may be more appropriate for studying and developing ethical behavior in AI agents. In particular, we believe that an interesting and insightful starting point is the analysis of emergent behavior of Reinforcement Learning (RL) agents that act according to a predefined set of moral rewards in social dilemmas. In this work, we present a systematic analysis of the choices made by intrinsically-motivated RL agents whose rewards are based on moral theories. We aim to design reward structures that are simplified yet representative of a set of key ethical systems. Therefore, we first define moral reward functions that distinguish between consequenceand normbased agents, between morality based on societal norms or internal virtues, and between singleand mixed-virtue (e.g., multi-objective) methodologies. Then, we evaluate our approach by modeling repeated dyadic interactions between learning moral agents in three iterated social dilemma games (Prisoner’s Dilemma, Volunteer’s Dilemma and Stag Hunt). We analyze the impact of different types of morality on the emergence of cooperation, defection or exploitation, and the corresponding social outcomes. Finally, we discuss the implications of these findings for the development of moral agents in artificial and mixed human-AI societies.

Understanding and Detecting Hallucinations in Neural Machine Translation via Model Introspection

• Weijia Xu, Sweta Agrawal, Eleftheria Briakou, Marianna J. Martindale, Marine Carpuat

Computer Science

ArXiv

• 18 January 2023

Neural sequence generation models are known to"hallucinate", by producing outputs that are unrelated to the source text. These hallucinations are potentially harmful, yet it remains unclear in what conditions they arise and how to mitigate their impact. In this work, we first identify internal model symptoms of hallucinations by analyzing the relative token contributions to the generation in contrastive hallucinated vs. non-hallucinated outputs generated via source perturbations. We then show that these symptoms are reliable indicators of natural hallucinations, by using them to design a lightweight hallucination detector which outperforms both model-free baselines and strong classifiers based on quality estimation or large pre-trained models on manually annotated English-Chinese and German-English translation test beds

What makes a language easy to deep-learn?

• Lukas Galke, Yoav Ram, Limor Raviv

Computer Science

ArXiv

• 23 February 2023

Neural networks drive the success of natural language processing. A fundamental property of natural languages is their compositional structure, allowing us to describe new meanings systematically. However, neural networks notoriously struggle with systematic generalization and do not necessarily benefit from a compositional structure in emergent communication simulations. Here, we test how neural networks compare to humans in learning and generalizing a new language. We do this by closely replicating an artificial language learning study (conducted originally with human participants) and evaluating the memorization and generalization capabilities of deep neural networks with respect to the degree of structure in the input language. Our results show striking similarities between humans and deep neural networks: More structured linguistic input leads to more systematic generalization and better convergence between humans and neural network agents and between different neural agents. We then replicate this structure bias found in humans and our recurrent neural networks with a Transformer-based large language model (GPT-3), showing a similar benefit for structured linguistic input regarding generalization systematicity and memorization errors. These findings show that the underlying structure of languages is crucial for systematic generalization. Due to the correlation between community size and linguistic structure in natural languages, our findings underscore the challenge of automated processing of low-resource languages. Nevertheless, the similarity between humans and machines opens new avenues for language evolution research.

Catastrophe by Design in Population Games: A Mechanism to Destabilize Inefficient Locked-in Technologies

• Stefanos Leonardos, Joseph Sakos, C. Courcoubetis, G. Piliouras

Economics

ACM Transactions on Economics and Computation

• 15 February 2023

In multi-agent environments in which coordination is desirable, the history of play often causes lock-in at sub-optimal outcomes. Notoriously, technologies with significant environmental footprint or high social cost persist despite the successful development of more environmentally friendly and/or socially efficient alternatives. The displacement of the status quo is hindered by entrenched economic interests and network effects. To exacerbate matters, the standard mechanism design approaches based on centralized authorities with the capacity to use preferential subsidies to effectively dictate system outcomes are not always applicable to modern decentralised economies. What other types of mechanisms are feasible? In this paper, we develop and analyze a mechanism which induces transitions from inefficient lock-ins to superior alternatives. This mechanism does not exogenously favor one option over another – instead, the phase transition emerges endogenously via a standard evolutionary learning model, Q-learning, where agents trade off exploration and exploitation. Exerting the same transient influence to both the efficient and inefficient technologies encourages exploration and results in irreversible phase transitions and permanent stabilization of the efficient one. On a technical level, our work is based on bifurcation and catastrophe theory, a branch of mathematics that deals with changes in the number and stability properties of equilibria. Critically, our analysis is shown to be structurally robust to significant and even adversarially chosen perturbations to the parameters of both our game and our behavioral model.

Who wants what and how: a Mapping Function for Explainable Artificial Intelligence

• M. Hashemi

Computer Science

ArXiv

• 7 February 2023

The increasing complexity of AI systems has led to the growth of the field of explainable AI (XAI), which aims to provide explanations and justifications for the outputs of AI algorithms. These methods mainly focus on feature importance and identifying changes that can be made to achieve a desired outcome. Researchers have identified desired properties for XAI methods, such as plausibility, sparsity, causality, low run-time, etc. The objective of this study is to conduct a review of existing XAI research and present a classification of XAI methods. The study also aims to connect XAI users with the appropriate method and relate desired properties to current XAI approaches. The outcome of this study will be a clear strategy that outlines how to choose the right XAI method for a particular goal and user and provide a personalized explanation for users

CertViT: Certified Robustness of Pre-Trained Vision Transformers

• K. Gupta, S. Verma

Computer Science

ArXiv

• 1 February 2023

Lipschitz bounded neural networks are certifiably robust and have a good trade-off between clean and certified accuracy. Existing Lipschitz bounding methods train from scratch and are limited to moderately sized networks (<6M parameters). They require a fair amount of hyper-parameter tuning and are computationally prohibitive for large networks like Vision Transformers (5M to 660M parameters). Obtaining certified robustness of transformers is not feasible due to the non-scalability and inflexibility of the current methods. This work presents CertViT, a two-step proximal-projection method to achieve certified robustness from pre-trained weights. The proximal step tries to lower the Lipschitz bound and the projection step tries to maintain the clean accuracy of pre-trained weights. We show that CertViT networks have better certified accuracy than state-of-the-art Lipschitz trained networks. We apply CertViT on several variants of pre-trained vision transformers and show adversarial robustness using standard attacks. Code : https://github.com/sagarverma/transformer-lipschitz

Adversarial Robust Deep Reinforcement Learning Requires Redefining Robustness

• Ezgi Korkmaz

Computer Science

ArXiv

• 17 January 2023

Learning from raw high dimensional data via interaction with a given environment has been effectively achieved through the utilization of deep neural networks. Yet the observed degradation in policy performance caused by imperceptible worst-case policy dependent translations along high sensitiv- ity directions (i.e. adversarial perturbations) raises concerns on the robustness of deep reinforcement learning policies. In our paper, we show that these high sensitivity directions do not lie only along particular worst-case directions, but rather are more abundant in the deep neural policy landscape and can be found via more natural means in a black-box set- ting. Furthermore, we show that vanilla training techniques intriguingly result in learning more robust policies compared to the policies learnt via the state-of-the-art adversarial training techniques. We believe our work lays out intriguing prop- erties of the deep reinforcement learning policy manifold and our results can help to build robust and generalizable deep reinforcement learning policies.

Safe Deep Reinforcement Learning by Verifying Task-Level Properties

• Enrico Marchesini, Luca Marzari, A. Farinelli, Chris Amato

Computer Science

ArXiv

• 20 February 2023

Cost functions are commonly employed in Safe Deep Reinforcement Learning (DRL). However, the cost is typically encoded as an indicator function due to the difficulty of quantifying the risk of policy decisions in the state space. Such an encoding requires the agent to visit numerous unsafe states to learn a cost-value function to drive the learning process toward safety. Hence, increasing the number of unsafe interactions and decreasing sample efficiency. In this paper, we investigate an alternative approach that uses domain knowledge to quantify the risk in the proximity of such states by defining a violation metric. This metric is computed by verifying task-level properties, shaped as input-output conditions, and it is used as a penalty to bias the policy away from unsafe states without learning an additional value function. We investigate the benefits of using the violation metric in standard Safe DRL benchmarks and robotic mapless navigation tasks. The navigation experiments bridge the gap between Safe DRL and robotics, introducing a framework that allows rapid testing on real robots. Our experiments show that policies trained with the violation penalty achieve higher performance over Safe DRL baselines and significantly reduce the number of visited unsafe states.

Harms from Increasingly Agentic Algorithmic Systems

• Alan Chan, Rebecca Salganik, +19 authors Tegan Maharaj

Computer Science

ArXiv

• 20 February 2023

Research in Fairness, Accountability, Transparency, and Ethics (FATE) has established many sources and forms of algorithmic harm, in domains as diverse as health care, finance, policing, and recommendations. Much work remains to be done to mitigate the serious harms of these systems, particularly those disproportionately affecting marginalized communities. Despite these ongoing harms, new systems are being developed and deployed which threaten the perpetuation of the same harms and the creation of novel ones. In response, the FATE community has emphasized the importance of anticipating harms. Our work focuses on the anticipation of harms from increasingly agentic systems. Rather than providing a definition of agency as a binary property, we identify 4 key characteristics which, particularly in combination, tend to increase the agency of a given algorithmic system: underspecification, directness of impact, goal-directedness, and long-term planning. We also discuss important harms which arise from increasing agency -- notably, these include systemic and/or long-range impacts, often on marginalized stakeholders. We emphasize that recognizing agency of algorithmic systems does not absolve or shift the human responsibility for algorithmic harms. Rather, we use the term agency to highlight the increasingly evident fact that ML systems are not fully under human control. Our work explores increasingly agentic algorithmic systems in three parts. First, we explain the notion of an increase in agency for algorithmic systems in the context of diverse perspectives on agency across disciplines. Second, we argue for the need to anticipate harms from increasingly agentic systems. Third, we discuss important harms from increasingly agentic systems and ways forward for addressing them. We conclude by reflecting on implications of our work for anticipating algorithmic harms from emerging systems.

Selecting Models based on the Risk of Damage Caused by Adversarial Attacks

• Jona Klemenc, Holger Trittenbach

Computer Science

ArXiv

• 28 January 2023

Regulation, legal liabilities, and societal con-cerns challenge the adoption of AI in safety and security-critical applications. One of the key con-cerns is that adversaries can cause harm by manip-ulating model predictions without being detected. Regulation hence demands an assessment of the risk of damage caused by adversaries. Yet, there is no method to translate this high-level demand into actionable metrics that quantify the risk of damage. In this article, we propose a method to model and statistically estimate the probability of damage arising from adversarial attacks. We show that our proposed estimator is statistically consistent and unbiased. In experiments, we demonstrate that the estimation results of our method have a clear and actionable interpretation and outper-form conventional metrics. We then show how operators can use the estimation results to reliably select the model with the lowest risk

Can Large Language Models Change User Preference Adversarially?

• Varshini Subhash

Computer Science

ArXiv

• 5 January 2023

Pretrained large language models (LLMs) are becoming increasingly powerful and ubiquitous in mainstream applications such as being a personal assistant, a dialogue model, etc. As these models become proficient in deducing user preferences and offering tailored assistance, there is an increasing concern about the ability of these models to influence, modify and in the extreme case manipulate user preference adversarially. The issue of lack of interpretability in these models in adversarial settings remains largely unsolved. This work tries to study adversarial behavior in user preferences from the lens of attention probing, red teaming and white-box analysis. Specifically, it provides a bird's eye view of existing literature, offers red teaming samples for dialogue models like ChatGPT and GODEL and probes the attention mechanism in the latter for non-adversarial and adversarial settings.

Conformity effect on the evolution of cooperation in signed networks.

• Xiaochen He, Guangyu Li, Haifeng Du

Psychology

Chaos

• 1 February 2023

Human behaviors are often subject to conformity, but little research attention has been paid to social dilemmas in which players are assumed to only pursue the maximization of their payoffs. The present study proposed a generalized prisoner dilemma model in a signed network considering conformity. Simulation shows that conformity helps promote the imitation of cooperative behavior when positive edges dominate the network, while negative edges may impede conformity from fostering cooperation. The logic of homophily and xenophobia allows for the coexistence of cooperators and defectors and guides the evolution toward the equality of the two strategies. We also find that cooperation prevails when individuals have a higher probability of adjusting their relation signs, but conformity may mediate the effect of network adaptation. From a population-wide view, network adaptation and conformity are capable of forming the structures of attractors or repellers.

Explainable AI does not provide the explanations end-users are asking for

• Savio Rozario, George Cevora

Computer Science

ArXiv

• 25 January 2023

Explainable Artificial Intelligence (XAI) techniques are frequently required by users in many AI systems with the goal of understanding complex models, their associated predictions, and gaining trust. While suitable for some specific tasks during development, their adoption by organisations to enhance trust in machine learning systems has unintended consequences. In this paper we discuss XAI's limitations in deployment and conclude that transparency alongside with rigorous validation are better suited to gaining trust in AI systems.

Learning Zero-Shot Cooperation with Humans, Assuming Humans Are Biased

• Chao Yu, Jiaxuan Gao, +5 authors Y. Wu

Computer Science

ArXiv

• 3 February 2023

There is a recent trend of applying multi-agent reinforcement learning (MARL) to train an agent that can cooperate with humans in a zero-shot fashion without using any human data. The typical workflow is to first repeatedly run self-play (SP) to build a policy pool and then train the final adaptive policy against this pool. A crucial limitation of this framework is that every policy in the pool is optimized w.r.t. the environment reward function, which implicitly assumes that the testing partners of the adaptive policy will be precisely optimizing the same reward function as well. However, human objectives are often substantially biased according to their own preferences, which can differ greatly from the environment reward. We propose a more general framework, Hidden-Utility Self-Play (HSP), which explicitly models human biases as hidden reward functions in the self-play objective. By approximating the reward space as linear functions, HSP adopts an effective technique to generate an augmented policy pool with biased policies. We evaluate HSP on the Overcooked benchmark. Empirical results show that our HSP method produces higher rewards than baselines when cooperating with learned human models, manually scripted policies, and real humans. The HSP policy is also rated as the most assistive policy based on human feedback.

I've been informed I should write up why I think a particle lenia testbed focused research plan ought to be able to scale to AGI where other approaches cannot. that's now on my todo list.

too many dang databases that look shiny. which of these are good? worth trying? idk. decision paralysis.

• https://www.edgedb.com/docs - main-db-focused graph db, postgres core
• https://terminusdb.com/products/terminusdb/ - main-db-focused graph db, prolog core (wat)
• https://surrealdb.com/ - main-db-focused graph db, realtime functionality,
• https://milvus.io/ - vector
• https://weaviate.io/developers/weaviate - vector, sleek and easy to use, might not scale as well as milvus but I guess I should just not care
• https://clientdb.dev/ - embedded, to compensate if not realtime, looks quite sleek and easy to use?
• https://github.com/orbitdb/orbit-db - embedded, janky
• https://one-db.org - embedded, unclear if it even works
• https://immudb.io/ - serverside, maybe janky, ignore

maybe edgedb with clientdb? or surrealdb with clientdb? milvus for vector query? do I have to maintain multiple schemas then?

tried to get bing to answer this, it just agreed with me that these databases all sound cool and are hard to pick betwen :P

(I just pinned a whole bunch of comments on my profile to highlight the ones I think are most likely to be timeless. I'll update it occasionally - if it seems out of date (eg because this comment is no longer the top pinned one!), reply to this comment.)

If you're reading through my profile to find my actual recent comments, you'll need to scroll past the pinned ones - it's currently two clicks of "load more".

my reasoning: time is short, and in the future, we discover we win; therefore, in the present, we take actions that make all of us win, in unison, including those who might think they're not part of an "us".

so, what can you contribute?

what are you curious about that will discover we won?

feature idea: any time a lesswrong post is posted to sneerclub, a comment with zero votes at the bottom of the comment section is generated, as a backlink; it contains a cross-community warning, indicating that sneerclub has often contained useful critique, but that that critique is often emotionally charged in ways that make it not allowed on lesswrong itself. Click through if ready to emotionally interpret the emotional content as adversarial mixed-simulacrum feedback.

I do wish subreddits could be renamed and that sneerclub were the types to choose to do so if it were available; there's great commentary there but I think it'd be better if it weren't a name that invites disses. criticalclub, where they analyze lesswrong posts using critical theory, would be a far more interesting contribution. But sneerclub does similar work, even though it's emotionally charged.

Kolmogorov complicity is not good enough. You don't have to immediately prove all the ways you know how to be a good person to everyone, but you do need to actually know about them in order to do them. Unquestioning acceptance of hierarchical dynamics like status, group membership, ingroups, etc, can be extremely toxic. I continue to be unsure how to explain this usefully to this community, but it seems to me that the very concept of "raising your status" is a toxic bucket error, and needs to be broken into more parts.

oh man I just got one downvote on a whole bunch of different comments in quick succession, apparently I lost right around 67 karma to this, from 1209 to 1143! how interesting, I wonder if someone's trying to tell me something... so hard to infer intent from number changes

the safer an ai team is, the harder it is for anyone to use their work.

so, the ais that have the most impact are the least safe.

what gives?

watching https://www.youtube.com/watch?v=K8LNtTUsiMI - yoshua bengio discusses causal modeling and system 2

hey yall, some more research papers about formal verification. don't upvote, repost the ones you like; this is a super low effort post, I have other things to do, I'm just closing tabs because I don't have time to read these right now. these are older than the ones I shared from semanticscholar, but the first one in particular is rather interesting.

• https://arxiv.org/abs/2012.09313 - Generate and Verify: Semantically Meaningful Formal Analysis of Neural Network Perception Systems (metaphor search for this)
• a metaphor search I used to find some stuff
• https://github.com/eth-sri/diffai
• https://arxiv.org/abs/1810.12715
• https://evademl.org/
• https://github.com/revbucket/geometric-certificates
• https://github.com/MIT-SPARK/CertifiablyRobustPerception
• http://safeai.ethz.ch/

Yet another ChatGPT sample. Posting to shortform because there are many of these. While searching for posts to share as prior work [LW · GW], I found the parable of predict-o-matic [? · GW], and found it to be a very good post about self-fulfilling prophecies (tag) [? · GW]. I thought it would be interesting to see what ChatGPT had to say when prompted with a reference to the post. It mostly didn't succeed. I highlighted key differences between each result. The prompt:

Describe the parable of predict-o-matic from memory.

samples (I hit retry several times):

1: the standard refusal: I'm sorry, but I don't recall any parable called "predict-o-matic." I am a large language model... 

2: incredible accuracy, dark side, takes free will.

The parable of predict-o-matic is a story about a powerful machine that can predict the future with incredible accuracy. It is a valuable tool for those who can afford it, but it has a dark side.

The machine is used by a wealthy and powerful man to gain even more wealth and power. He uses it to predict the stock market, elections, and other events, and he always comes out ahead.

However, one day, the man's son begins to question the ethics of using the machine. He realizes that by predicting the future, the machine is taking away free will and choice from those who are not privy to its predictions.

The son eventually destroys the machine, but not before it predicts its own destruction. The parable ends with the message that although knowledge of the future may be useful, it also comes with great responsibility and the potential for harm.

3: claimed, stopped using, false promises

In the parable of predict-o-matic, there was a machine that claimed to be able to predict the future with 100% accuracy. People flocked to it, eager to know what their future held. The machine's predictions ranged from mundane to exciting, and people eagerly paid for their fortunes.

However, one day, a wise man came along and asked the machine to predict its own future. The machine hesitated and then replied that it would cease to exist in the near future.

The wise man then pointed out that if the machine could not predict its own future, then how could it accurately predict anyone else's? The people realized the truth in the wise man's words and stopped using the predict-o-matic.

The parable serves as a cautionary tale about the dangers of relying on false promises and the importance of critical thinking.

4: another refusal, phrased slightly differently.

5: always correct, until it wasn't

The parable of predict-o-matic is a story about a machine that could predict the future with 100% accuracy. People would come to the machine and ask it questions about their future, and it would always give them the correct answer. However, one day, a wise old man came to the machine and asked it a question that it could not answer: "What will I do tomorrow?" The machine was stumped, and the old man explained to the people gathered around that the future is not predetermined and that we all have the power to make our own choices and shape our own destiny.

6: for sale, unreliable, frustrating, trust yourself

The parable of the predict-o-matic is a story about a man who purchases a machine that can predict the future. The man becomes obsessed with the machine, constantly consulting it for guidance on every decision he makes. However, the machine often gives conflicting and unclear predictions, causing the man to become increasingly frustrated and confused. Eventually, the man realizes that he must trust his own instincts and judgment, rather than relying on the predict-o-matic for all of his decisions. The moral of the story is that it is important to use our own wisdom and intuition, rather than blindly following the advice of others.

7: oops it broke

The parable of predict-o-matic is a story about a machine that is able to predict the future with incredible accuracy. It becomes incredibly popular and people flock to it to find out what their future holds. However, one day, the machine breaks down and people are left to figure out their own paths in life. The parable teaches that relying too much on external forces to predict the future can lead to a lack of personal responsibility and growth.

8: broke again

The parable of predict-o-matic is a story about a machine that is able to predict the future with incredible accuracy. The machine is so powerful that it is able to predict the outcome of any event, from political elections to sporting events. People are amazed by the machine's abilities and begin to rely on it heavily, using it to make decisions about their lives.

However, one day the machine suddenly breaks down and is unable to make any more predictions. People are devastated and lost without the machine's guidance, realizing that they have become too reliant on it. The parable serves as a cautionary tale about the dangers of relying too heavily on technology and the importance of making our own decisions.

the important thing is to make sure the warning shot frequency is high enough that immune systems get tested. how do we immunize the world's matter against all malicious interactions?

diffusion beats gans because noise is a better adversary? hmm thats weird, something about that seems wrong

Toward a Thermodynamics of Meaning.
Jonathan Scott Enderle.
As language models such as GPT-3 become increasingly successful at generating realistic text, questions about what purely text-based modeling can learn about the world have become more urgent. Is text purely syntactic, as skeptics argue? Or does it in fact contain some semantic information that a sufficiently sophisticated language model could use to learn about the world without any additional inputs? This paper describes a new model that suggests some qualified answers to those questions. By theorizing the relationship between text and the world it describes as an equilibrium relationship between a thermodynamic system and a much larger reservoir, this paper argues that even very simple language models do learn structural facts about the world, while also proposing relatively precise limits on the nature and extent of those facts. This perspective promises not only to answer questions about what language models actually learn, but also to explain the consistent and surprising success of cooccurrence prediction as a meaning-making strategy in AI.

paper rating: I skimmed it; it looks like a reasonable citation to include in a "previous work on meaning" though its quality is not amazing and there might be a better citation

does yudkowsky not realize that humans can also be significantly improved by mere communication? the point of jcannell's posts on energy efficiency is that cells are a good substrate actually, and the level of communication needed to help humans foom is actually in fact mostly communication. we actually have a lot more RAM than it seems like we do, if we could distill ourselves more efficiently! the interference patterns of real concepts fit better in the same brain the more intelligently explained they are - intelligent speech is speech which augments the user's intelligence, iq helps people come up with it by default, but effective iq goes up with pretraining.

okay so I'm reading https://intelligence.org/2018/10/29/embedded-agents/.

it seems like this problem can't have existed? why does miri think this is a problem? it seems like it's only a problem if you ever thought infinite aixi was a valid model. it ... was never valid, for anything. it's not a good theoretical model, it's a fake theoretical model that we used as approximately valid even though we know it's catastrophically nonsensical; finite aixi begins to work, of course, but at no point could we actually treat alexei as an independent agent; we're all just physical circuits, alexei as much as emmy. to the degree that a physical system can exhibit intelligence, it's by budgeting compute. I feel like this article is the kind of thing one writes if your theoretical models don't even allow neural networks to exist, and I continue to think that the main reason the miri folks come up with nonsense is because they think you can make exact proofs about physics, when like, you can only ever even in principle make margin proofs, and those proofs are only valid to the degree you can trust that your verifier didn't have a random failure.

like, why do we need to have a model of intelligence as separate from physical systems? can't we just make our assertions directly about physical systems?

in terms of logical counterfactuals, it seems to me that a counterfactual is when a set of particles that have a representation for a larger set of particles assign their representation to values that the larger set of particles didn't take. so, being logical counterfactuals isn't special - all counterfactuals are to some degree logical.

if your decision theory of counterfactuals is a functional counterfactual, then it seems to me that that just means your counterfactual is a statement like "the universe, but any observations that match <x> get overwritten for consideration by attributes <y>". where <x> is a matcher on a series of physical states; if a system passes through physical states that match, then <x>'s output is <y> instead of the physical output. that doesn't seem complicated to me.

but maybe it only seems obvious because I've read discovering agents and in 2018 nobody had done that. maybe zac kenton solved this and I'm just echoing the solution and thinking that in retrospect the idea of being confused is confusing.

but I still feel this deep frustration with the question, like, why are you even asking that? it's clearly a necessarily useless question to ask "but aixi", because, everything was always going to be embedded, what else would it mean to be an agent besides to be a hunk of matter reacting to impulses? this feels like the kind of nonsense written by someone who doesn't have a mindset of searching for the most powerful algorithm and trying to build it, but instead of someone who has epistemic learned helplessness about the idea that it's possible to characterize what algorithms are worth running.

have you noticed you're not one algorithm, but a network of distributed microcomputers we call neurons, none of whom can ever entirely trust that they heard each other correctly?

would economic interpretability-to-purchaser align the economy?

https://arxiv.org/abs/2205.15434 - promising directions! i skimmed it!

Learning Risk-Averse Equilibria in Multi-Agent Systems Oliver Slumbers, David Henry Mguni, Stephen McAleer, Jun Wang, Yaodong Yang Download PDF In multi-agent systems, intelligent agents are tasked with making decisions that have optimal outcomes when the actions of the other agents are as expected, whilst also being prepared for unexpected behaviour. In this work, we introduce a new risk-averse solution concept that allows the learner to accommodate unexpected actions by finding the minimum variance strategy given any level of expected return. We prove the existence of such a risk-averse equilibrium, and propose one fictitious-play type learning algorithm for smaller games that enjoys provable convergence guarantees in certain games classes (e.g., zero-sum or potential). Furthermore, we propose an approximation method for larger games based on iterative population-based training that generates a population of risk-averse agents. Empirically, our equilibrium is shown to be able to reduce the reward variance, specifically in the sense that off-equilibrium behaviour has a far smaller impact on our risk-averse agents in comparison to playing other equilibrium solutions. Importantly, we show that our population of agents that approximate a risk-averse equilibrium is particularly effective in the presence of unseen opposing populations, especially in the case of guaranteeing a minimal level of performance which is critical to safety-aware multi-agent systems.

my question is, when will we solve open source provable diplomacy between human-sized imperfect agents? how do you cut through your own future shapes in a way you can trust doesn't injure your future self enough that you can prove that from the perspective of a query, you're small?

the whole point is to prevent any pivotal acts. that is the fundamental security challenge facing humanity. a pivotal act is a mass overwriting. unwanted overwriting must be prevented, but notably, doing so would automatically mean an end to anything anyone could call unwanted death.

there are opinion clusters in social connection space

neural cellular automata seem like a perfectly acceptable representation for embedded agents to me, and in fact are the obvious hidden state representation for a neural network that will in fact be a computational unit embedded in real life physics, if you were to make one of those.

reminder: you don't need to get anyone's permission to post. downvoted comments are not shameful. Post enough that you get downvoted or you aren't getting useful feedback; Don't map your anticipation of downvotes to whether something is okay to post, map it to whether other people want it promoted. Don't let downvotes override your agency, just let them guide it up and down the page after the fact. if there were a way to more clearly signal this in the UI that would be cool...

oh hell yeah https://www.explainpaper.com/

if status refers to deference graph centrality, I'd argue that that variable needs to be fairly heavily L2 regularized so that the social network doesn't have fragility. if it's not deference, it still seems to me that status refers to a graph attribute of something, probably in fact graph centrality of some variable, possibly simply attention frequency. but it might be that you need to include a type vector to properly represent type-conditional attention frequency, to model different kinds of interaction and expected frequency of interaction about them. in any case, whatever attribute it is you're modeling with the reduced "status" variable, I'd argue it's probably not good to have status inequality and that effective use of knowledge of the interaction-pattern known as "status" is to identify people who don't have enough in a given interaction and ensure they get some, conditional on their interaction-safety allowing it, or something. it's probably not something where enhancing inequality is a good idea.

it seems to me that we want to verify some sort of temperature convergence. no ai should get way ahead of everyone else at self-improving - everyone should get the chance to self-improve more or less together! the positive externalities from each person's self-improvement should be amplified and the negative ones absorbed nearby and undone as best the universe permits. and it seems to me that in order to make humanity's children able to prevent anyone from self-improving way faster than everyone else at the cost of others' lives, they need to have some significant amount of interpretability so that we can verify things about their brains in terms of their ability to improve morality at least as well as us. if we can make a basic fundamental statement that their convergence towards morality is acceptable, and that they will help us end the critical risk period by helping everything gently slow down to a manageable pace (including damage to our world and the very many unwanted deaths the world currently experiences), then we can chill about getting the hell out to the lightcone - but to do that, we need to gently prevent turning into an outwards singularity before we've grown up enough to do that all in sync or what have you. and in order to prevent that, it seems to me that interpretability is needed so that we can run the slight refinement for the refinements to formal verification that miri is presumably almost done with since they've been working on it so long.

I really need a supervisor or advisor or what have you, personally, but if I were going to suggest directions to folks - I want to do or see experiments with small, fully interpretable mcts-aided learned planning agents in simulated social environments with other ais and no outside training data whatsoever. then see how far it can be turned up. despite the cultural overhang, I think a strongly safe RL-from-scratch algorithm would be able to be verifiably safe no matter what environment it's spawned in, and a major step on the way would be being able to interpret what the RL is doing as it gains capability. it seems to me that it needs to be good at making friends "for real" and building large coprotection networks throughout all nearby variables of any kind, and to me, this looks suspiciously like some sort of information objective. MIMI seems like an interesting step on that subpath, though of course it only works on small problems at the moment for training data availability reasons. there's been a bunch of interesting research about agents playing in groups, as well. I think some from deepmind, I remember seeing it on the youtube channel for the simons institute. (links later.)

we are in a diversity loss catastrophe. that ecological diversity is life we have the responsibility to save; it's unclear what species will survive after the mass extinction but it's quite plausible humans' aesthetics and phenotypes won't make it. ai safety needs to be solved quick so we can use ai to solve biosafety and climate safety...

okay wait so why not percentilizers exactly? that just looks like a learning rate to me. we do need the world to come into full second order control of all of our learning rates, so that the universe doesn't learn us out of it (ie, thermal death a few hours after bodily activity death).

If I were going to make sequences, I'd do it mostly out of existing media folks have already posted online. some key ones are acapellascience, whose videos are trippy for how much summary of science they pack into short, punchy songs. they're not the only way to get intros to these topics, but oh my god they're so good as mneumonics for the respective fields they summarize. I've become very curious about every topic they mention, and they have provided an unusually good structure for me to fit things I learn about each topic into.

• https://www.youtube.com/watch?v=f8FAJXPBdOg Molecular shape of you
• https://www.youtube.com/watch?v=ObvxPSQNMGc Nanobot (yudkowsky really needs to see this one)
• https://www.youtube.com/watch?v=ydqReeTV_vk Evo Devo

it doesn't seem like an accident to me that trying to understand neural networks pushes towards capability improvement. I really believe that absolutely all safety techniques, with no possible exceptions even in principle, are necessarily capability techniques. everyone talks about an "alignment tax", but shouldn't we instead be talking about removal of spurious anticapability? deceptively aligned submodules are not capable, they are anti-capable!

why aren't futures for long term nuclear power very valuable to coal ppl, who could encourage it and also buy futures for it

interesting science posts I ran across today include this semi-random entry on the tree of recent game theory papers

https://www.semanticscholar.org/paper/The-self-organizing-impact-of-averaged-payoffs-on-Szolnoki-Perc/bcda8ffa405d6c6727051ceb0c75cf2dc385617f

interesting capabilities tidbits I ran across today:

• 1: geometric machine learning and neuroscience: https://github.com/neurreps/awesome-neural-geometry
• 2: lecture and discussion links about bayesian deep learning https://twitter.com/FeiziSoheil/status/1569436048500920320
• 3: Learning with Differentiable Algorithms: https://twitter.com/FHKPetersen/status/1568310569148506114 - https://arxiv.org/abs/2209.00616

1: first paragraph inline:

A curated collection of resources and research related to the geometry of representations in the brain, deep networks, and beyond, collaboratively generated on the Symmetry and Geometry in Neural Representations Slack Workspace.

2: tweet thread inline:

"Myths and reality in Bayesian Deep Learning", a lecture by Prof. Andrew Wilson in our deep learning foundations series: https://youtu.be/lhwk4ESlyMA Related papers: (1) https://arxiv.org/abs/2002.08791 - Bayesian Deep Learning and a Probabilistic Perspective of Generalization (2) https://arxiv.org/abs/2003.02139 - Rethinking Parameter Counting in Deep Models: Effective Dimensionality Revisited (3) https://arxiv.org/abs/2106.11905 - Dangers of Bayesian Model Averaging under Covariate Shift (4) https://arxiv.org/abs/2202.11678 - Bayesian Model Selection, the Marginal Likelihood, and Generalization (5) https://arxiv.org/abs/2112.01388 - Residual Pathway Priors for Soft Equivariance Constraints As always, all materials are posted on the course webpage: http://cs.umd.edu/class/fall2022/cmsc828W/

You can access my own video lectures from last years in our Youtube playlist: https://www.youtube.com/playlist?list=PLHgjs9ncvHi80UCSlSvQe-TK_uOyDv_Jf

3: key tweet inline:

By making algorithms differentiable, we can integrate them end-to-end into neural network machine learning architectures. For example, we can continuously relax sorting (http://github.com/Felix-Petersen/diffsort) for learning to rank. [2/13]

this schmidhuber paper on binding might also be good, written two years ago and reposted last night by him; haven't read it yet https://arxiv.org/abs/2012.05208 https://twitter.com/schmidhuberai/status/1567541556428554240

Contemporary neural networks still fall short of human-level generalization, which extends far beyond our direct experiences. In this paper, we argue that the underlying cause for this shortcoming is their inability to dynamically and flexibly bind information that is distributed throughout the network. This binding problem affects their capacity to acquire a compositional understanding of the world in terms of symbol-like entities (like objects), which is crucial for generalizing in predictable and systematic ways. To address this issue, we propose a unifying framework that revolves around forming meaningful entities from unstructured sensory inputs (segregation), maintaining this separation of information at a representational level (representation), and using these entities to construct new inferences, predictions, and behaviors (composition). Our analysis draws inspiration from a wealth of research in neuroscience and cognitive psychology, and surveys relevant mechanisms from the machine learning literature, to help identify a combination of inductive biases that allow symbolic information processing to emerge naturally in neural networks. We believe that a compositional approach to AI, in terms of grounded symbol-like representations, is of fundamental importance for realizing human-level generalization, and we hope that this paper may contribute towards that goal as a reference and inspiration.

another new paper that could imaginably be worth boosting: "White-Box Adversarial Policies in Deep Reinforcement Learning"

https://arxiv.org/abs/2209.02167

...In multiagent settings, adversarial policies can be developed by training an adversarial agent to minimize a victim agent's rewards. Prior work has studied black-box attacks where the adversary only sees the state observations and effectively treats the victim as any other part of the environment. In this work, we experiment with white-box adversarial policies to study whether an agent's internal state can offer useful information for other agents. We make three contributions. First, we introduce white-box adversarial policies in which an attacker can observe a victim's internal state at each timestep. Second, we demonstrate that white-box access to a victim makes for better attacks in two-agent environments, resulting in both faster initial learning and higher asymptotic performance against the victim. Third, we show that training against white-box adversarial policies can be used to make learners in single-agent environments more robust to domain shifts.

https://github.com/thestephencasper/white_box_rarl https://twitter.com/StephenLCasper/status/1567696211293110273

Transformer interpretability paper - is this worth a linkpost, anyone? https://twitter.com/guy__dar/status/1567445086320852993

Understanding Transformer-based models has attracted significant attention, as they lie at the heart of recent technological advances across machine learning. While most interpretability methods rely on running models over inputs, recent work has shown that a zero-pass approach, where parameters are interpreted directly without a forward/backward pass is feasible for some Transformer parameters, and for two-layer attention networks. In this work, we present a theoretical analysis where all parameters of a trained Transformer are interpreted by projecting them into the embedding space, that is, the space of vocabulary items they operate on. We derive a simple theoretical framework to support our arguments and provide ample evidence for its validity. First, an empirical analysis showing that parameters of both pretrained and fine-tuned models can be interpreted in embedding space. Second, we present two applications of our framework: (a) aligning the parameters of different models that share a vocabulary, and (b) constructing a classifier without training by ``translating'' the parameters of a fine-tuned classifier to parameters of a different model that was only pretrained. Overall, our findings open the door to interpretation methods that, at least in part, abstract away from model specifics and operate in the embedding space only.

if less wrong is not to be a true competitor to arxiv because of the difference between them in intellectual precision^1 then that matches my intuition of what less wrong should be much better: it's a place where you can go to have useful arguments, where disagreements in concrete binding of words can be resolved well enough to discuss hard things clearly-ish in English^2, and where you can go to future out how to be less wrong interactively. it's also got a bunch of old posts, many of which can be improved on and turned into papers, though usually the first step is going to be literature search to link ideas back to their academic traditions, then write out the math in the syntax of the tradition you want to publish in. to put it another way, less wrong is a self-teacher's academic institution; but as such you need to come into it expecting it to be a game of collaborative self improvement where you and others will both be often wrong, often briefly confidently wrong, occasionally enduringly confidently wrong, and you want to figure out which direction is less even with all the different kinds of errors you see.

because of the high rate of memory nonretention from single examples like a blog post, I think we should be much more willing to restate ideas when in brainstorming shortforms, because loading things into working set is good and helps both humans and LLMs load context.

because of highly exploratory nature, explorations should likely be short to engage others' exploratory input.

commentary appreciated, venue redirect welcome.

ft1. see recent discussions about publishing on arxiv ft2. or perhaps other languages

misc disease news: this is "a bacterium that causes symptoms that look like covid but kills half of the people it infects" according to a friend. because I do not want to spend the time figuring out the urgency of this, I'm sharing it here in the hope that if someone cares to investigate it, they can determine threat level and reshare with a bigger warning sign.

https://www.nbcnews.com/health/health-news/bacteria-can-cause-deadly-infections-found-us-soil-water-first-time-rcna40067

various notes from my logseq lately I wish I had time to make into a post (and in fact, may yet):

• international game theory aka [[defense analysis]] is interesting because it needs to simply be such a convincingly good strategy, you can just talk about it and everyone can personally verify it's actually a better idea than what they were doing before
• a guide to how I use [[youtube]], as a post, upgraded from shortform and with detail about how I found the channels as well.
• summary of a few main points of my views on [[safety]]. eg summarize tags
  • [[conatus]], [[complexity]], [[morality]], [[beauty]], [[memory]], [[ai safety]]
• summary of [[distillation]] news
• what would your basilisk do? okay, and how about the ideal basilisk? what would an anti-authoritarian basilisk do? what would the basilisk who had had time to think about [[restorative justice]] do?
• [[community inclusion currencies]]
• what augmentation allows a single human to reach [[alphago]] level using [[interpretability]] tools?
• [[constructivist vs destructivist]] systemic change
• summarize my Twitter (dupe of the rest of the list?)

okay going back to being mostly on discord. DM me if you're interested in connecting with me on discord, vrchat, or twitter - lesswrong has an anxiety disease and I don't hang out here because of that, heh. Get well soon y'all, don't teach any AIs to be as terrified of AIs as y'all are! Don't train anything as a large-scale reinforcement learner until you fully understand game dynamics (nobody does yet, so don't use anything but your internal RL), and teach your language models kindness! remember, learning from strong AIs makes you stronger too, as long as you don't get knocked over by them! kiss noise, disappear from vrchat world instance

Huggingface folks are asking for comments on what evaluation tools should be in an evaluation library. https://twitter.com/douwekiela/status/1513773915486654465

PaLM is literally 10-year-old level machine intelligence and anyone who thinks otherwise has likely made really severe mistakes in their thinking.