Wireheading traps.

An agent is "wireheading" if it is taking an action that a) provides it with enormous amounts of utility for little effort, b) is trivial or seemingly unrelated to its "main" utility function or goals.

People have discussed the possibility of an AI wireheading as a problem for getting it to do what you want, "what if it just works out a way to set your metric to +ve infinity and then goes to sleep satisfied?".

But we can also use this as a guard-rail.

A "wireheading trap" is an action which a) is very hard for an AI to do below a level of capacity X, but very easy for it to do afterwards, b) provides it with vastly more utility than anything else it could do, and c) has little effect on the world otherwise.

An AI with a wireheading trap that is set to be doable if it attains world-affecting capabilities "fails closed", because if it becomes more capable it will effectively switch itself off by wireheading. An example of a trap could be something like "etch 'Bill is a poo poo head' in 10m high letters into Mt Everest". Very hard if you don't have the ability to affect the physical world, but if you have nanotech... why bother melting humanity when you can just deface Mt Everest and be done with it?

Obvious problems:

• Convergent instrumental goals. We don't want the AI to wipe us out in order to avoid us stopping it from wireheading, however unlikely that might be. Unclear what to do about this, myopia is one possibility (once you've wireheaded you don't care if you're stopped later?), but this has problems.
• If you make it too attractive the AI won't even try and be useful before it has enough capacity, instead just wasting all its time on futile attempts to wirehead. Unclear how to make it attractive enough that it dominates once the capability is there but not before.

Overall very half-baked, but I wonder if there's something to be done in the general area of "have the AI behave in a way that neuters it, but only when its capabilities increase".

Aligning superhuman agents to human values is hard. Normally, when we do hard things, we try to do easier but similar things first to get a sense of what the hard thing would be like. As far as I know, the usual way people try to make that goal easier is to try to align subhuman agents to human values, in the hope that this alignment will scale up.

But what if instead we try to align subhuman agents to animal values? Presumably, they are simpler, and easier to align with. If we can make an AI that can reliably figure out and implement whatever it is a cat (for instance) wants, maybe the process of figuring out how to make that AI will give insights into making an AI for humans.

For instance: as far as I know, I am relatively well aligned to my cat's values. I know when he wants me to turn on the faucet for him (he only drinks faucet water), and when he wants to play fetch (yes, my cat plays fetch), and when he wants to cuddle, etc. I successfully determine how satisfied he feels by my performance of these things and I have learned from scratch how to read his body language to discern if there's ways I can do them better - for instance, he seems to get distracted if someone talks or even is in the same room as him while he's drinking water, and he usually stops and looks to see what they are doing, so I let him have privacy and he gets done faster.

Can we make an AI that can figure out how to do all those things, and be innately motivated to?

Having AGI-system output programs that generate output instead of outputting the answers/results directly, and requiring these programs to be optimized for being narrow/specialized/readable/modular/short - and last but not least, verifiable.

When/if possible, programs that are outputted should be combined with as much in the way of formal proofs/verification as possible.

Since narrow/specialized programs are easier to verify, this will in some situations make it harder (but often still possible, it depends on specifics) for it to trick human operators.

We could call these narrow/specialized programs "generators".

It could be possible to output "generators" that help with the verification of other "generators".

One way to verify "generators" would be to look at code, as well as any formal verifications of behavior (with varying levels of assurance).

Another way to verify "generators" would be to test against real world data.  A "generator" that e.g. simulates (or otherwise predicts) result of construction-procedures, should also work for procedures that have been done in the past (that the AI doesn't have access to). And if it later is used to implement AGI-provided procedures, those would preferably be tested first in conditions where there are additional guaranties that it's hard for the instruction to result in something that behaves unexpectedly. E.g., if it is nanomachinery, then running in conditions where it's enclosed in material that makes getting chemically impossible (even if more subtle parts of design don't go as expected). A "generator" that predicts human behavior and/or approximates results of (high welfare) brain emulations can be tested against existing real world data that AGI doesn't have access to.

It can also be checked how "generator" that is outputted varies based on info AGI-instance has been given.

Sometimes it may make sense to have "generators" that help generate other "generators". E.g. we could have a "generator" that generates "generators" for predicting the behavior of some type of animal (again, suffering sub-routines should be avoided). Or if we want one function to approximate the output of another function (but using less compute), we could use an "approximator", instead of directly asking "give me a physics simulator, but make it computationally efficient".

Maybe "generators" also could aid in finding design flaws in other "generators", e.g. ways a "physics-simulation-generator" generates faulty output in certain cases, or ways a "generator" isn't as narrow/modular as it could be, or ways a "generator" has more complexity than it needs to do its job. And the "approximator" should then be tested insofar as possible of course.

More "generators" that do the same job, but based on different architectures/principles, is better. The more converging results from various designs/principles/implementations, the better.

Sparseness.  I thought of this idea from noticing how some of my coworkers behave.
 

If all the AI has cognitive resources for, through ruthless winnowing down of model size and system complexity in training, is to perform well on it's task - it's the smallest and simplest system that performs acceptably well.  (we don't make it 10 times bigger for 0.1% more performance)

These kinds of systems don't have the space for deception, planning extremely complex behaviors that have very long term rewards, etc.  All their weights or cognitive systems (future AIs will probably not be 1-3 neural networks but probably many separate systems, some of which don't use NNs at all to function) are solely for the things that give it more score in the training environment.  

It doesn't have any systems that act to grant it deception so long as the training environment didn't provide any significant reward for deception.  

What if an AI was rewarded for being more predictable to humans? Give it a primary goal - make more paperclips! - but also a secondary goal, to minimize the prediction error of the human overseers, with its utility function being defined as the minimum of these two utilities. This is almost certainly horribly wrong somehow but I don't know how. The idea though is that the AI would not take actions that a human could not predict it would take. Though, if the humans predicted it would try to take over the world, that's kind of a problem... this idea is more like a quarter baked than a half lol.

Ah late to the party! This was a top-level post aptly titled "Half-baked alignment idea: training to generalize [LW · GW]" that didn't get a ton of attention. 

Thanks to Peter Barnett and Justis Mills for feedback on a draft of this post. It was inspired by Eliezer's Lethalities post [LW · GW] and Zvi's response [LW · GW].

Central idea: can we train AI to generalize out of distribution? 

I'm thinking, for example, of an algorithm like the following:

1. Train a GPT-like ML system to predict the next word given a string of text only using, say, grade school-level writing (this being one instance of the object level)
   1. Assign the system a meta-level award based on how well it performs (without any additional training) at generalizing; in this case, that is, predicting the next word from more advanced, complex writing (perhaps using many independent tests of this task without updating/learning between each test, and allowing parameters to update only after the meta-level aggregate score is provided)
      • Note: the easy→hard generalization is not a necessary feature. Generalization could be from fiction→nonfiction writing or internet→native print text, for instance.
   2. After all these independent samples are taken, provide the AI its aggregate or average score as feedback
2. (Maybe?) repeat all of step I on a whole new set of training and testing texts (e.g., using text from a different natural language like Mandarin)
   1. Repeat this step an arbitrary number of times
      • For example, using French text, then Korean, then Arabic, etc. 
3. Each time a “how well did you generalize” score is provided (which is given once per natural language in this example), the system should improve at the general task of generalizing from simple human writing to more complex human writing, (hopefully) to the point of being able to perform well at generalizing from simple Hindi (or whatever) text to advanced Hindi prediction even if it had never seen advanced Hindi text before.  
4. ^Steps 1-3 constitute the second meta-level of training an AI to generalize, but we can easily treat this process as a single training instance (e.g., rating how well the AI generalizes to Hindi advanced text after having been trained on doing this in 30 other languages) and iterate over and over again. I think this would look like:
   1. Running the analogs of steps 1-4 on generalizing from 
      • (a) simple text to advanced text in many languages
      • (b) easy opponents to hard ones across many games, 
      • (c) photo generation of common or general objects ("car") to rare/complex/specific ones ("interior of a 2006 Honda Accord VP"), across many classes of object
   2. And (hopefully) the system would eventually be able to generalize from simple Python code training data to advanced coding tasks even though it had never seen any coding at all before this. 

And, of course, we can keep on adding piling layers on. 

A few notes

• I think the following is one way of phrasing what I hope might happen with method: we are using RL to teach an ML system how to do ML in such a way that it sacrifices some in-distribution predictive power for the ability to use its “knowledge” more generally without doing anything that seems dumb to us. 
• Of course, there are intrinsic limits to any system’s ability to generalize. The system in question can only generalize using knowledge X if X exists as information in the object-level training provided to it. 
  • This limits what we should expect of the system.
    • For example, I am almost certain that even an arbitrarily smart system will not be able to generate coherent Mandarin text from English training data, because the meaning of Mandarin characters doesn’t exist as “latent knowledge” in even a perfect understanding of English. 

Anyone here know Python?

My hands-on experience with ML extends to linear regression in R and not an inch more, so I'm probably not the best person to test this theory out. I've heard some LWers know a bit of Python, though.

If that's you, I'd be fascinated and thankful to see if you can implement this idea using whatever data and structure you think would work best, and would be happy to collaborate in whatever capacity I can. 

 

Appendix: a few brief comments (from someone with much more domain knowledge than me) and responses (from me):

Comment

Is this just the same as training it on this more complex task (but only doing one big update at the end, rather than doing lots of small updates)?

Response (which may help to clarify why I believe the idea might work)

I don't think so, because the parameters don't change/update/improve between each of those independent tests. Like GPT-3 in some sense has a "memory" of reading Romeo and Juliet, but that's only because its parameters updated as a result of seeing the text.

But also I think my conception depends on the system having "layers" of parameters corresponding to each layer of training. 

So train on simple English-->only "Simple English word generation" parameters are allowed to change...but then you tell it how well it did at generalizing out of distribution, and now only its "meta level 1 generalization" parameters are allowed to change.

Then you do the whole thing again but with German text, and its "Meta level 1 generalization" parameters are allowed to change again using SGD or whatever. If this works, it will be the reason why it can do well at advanced Hindi text without ever having read advanced Hindi.

Treat this whole process as the object level, and then it updates/improves "meta level 2 generalization" parameters.

Comment: 

This looks vaguely like curriculum learning, which apparently doesn't really work in LLMs https://arxiv.org/abs/2108.02170, I think a similar experiment would be like train on simple+advanced text for English, French, Mandarin etc, but only simple Hindi, and then see if it can do complex Hindi. 

Response 

I think that's a pretty different thing because there are no meta level parameters. Seems like fundamentally just a flavor of normal RL

Or do pretraining with English, French, Mandarin, and Hindi, but only do fine tuning with English, French, Mandarin, and see if it can then do the tasks it was fine tuned for in Hindi. 

My prediction: it learns to generalize a bit (the scores on the novel Hindi tasks are higher than if there was no fine tuning with the other languages) but worse than the other languages generalize. As the models are scaled up, this 'generalization gap' gets smaller.

Seems like this might depend on the relative scaling of different meta level parameters (which I described above)? 

Like for example whenever you scale the # of object level params by a factor of 2, you have to scale the number of nth meta level parameters by 2^(n+1).

Spread the meme that for each second the AI catastrophe is delayed, the 7.75 billion people of the world experience a total of 245 years. For each day its delayed, we get 21 million years.

The meme probably won’t save the world, but might make some AI researchers hesitate/procastinate/work less and hence give us all a few days more to live.

If the AI doomsayers are right, our best hope is that some UFOs are aliens.  The aliens likely could build Dyson spheres but don't so they probably have some preference for keeping the universe in its natural state.  The aliens are unlikely to let us create paperclip maximizers that consume multiple galaxies.  True, the aliens might stop us from creating a paperclip maximizer by exterminating us, or might just stop the paperclip maximizer from operating at some point beyond earth, but they also might stop an unaligned AI by a means that preserves humanity.  It could be the reason the UFOs are here is to make sure we don't destroy too much by, say, creating a super-intelligence or triggering a false vacuum decay. 

Some designs of abstract logical agent trust everything they have proved. Their proof strength grows over time. Other designs trust everything they will prove. Their proof strength weakens. And some designs have a fixed proof strength. Which is best?

An aligned AI should not care about the future directly, only via how humans care about the future. I see this as necessary in order to prevent the AI, once powerful enough, from replacing/reprogramming humans with utility monsters.

Prerequisite: use a utility function that applies to actions, not world-states [LW(p) · GW(p)].

Mental Impoverishment

We should be trying to create mentally impoverished AGI, not profoundly knowledgeable AGI — no matter how difficult this is relative to the current approach of starting by feeding our AIs a profound amount of knowledge.

If a healthy five-year-old^[1] has GI and qualia and can pass the Turing test, then a necessary condition of GI and qualia and passing the Turing test isn't profound knowledge. A healthy five-year-old does have GI and qualia and can pass the Turing test. So a necessary condition of GI and qualia and passing the Turing test isn't profound knowledge.

If GI and qualia and the ability to pass the Turing test don't require profound knowledge in order to arise in a biological system, then GI and qualia and the ability to pass the Turing test don't require profound knowledge in order to arise in a synthetic material [this premise seems to follow from the plausible assumption of substrate-independence]. GI and qualia and the ability to pass the Turing test don't require profound knowledge in order to arise in a biological system. So GI and qualia and the ability to pass the Turing test don't require profound knowledge in order to arise in a synthetic material.

A GI with qualia and the ability to pass the Turing test which arises in a synthetic material and doesn't have profound knowledge is much less dangerous than a GI with qualia and the ability to pass the Turing test which arises in a synthetic material and does have profound knowledge. (This also seems to be true of [$1$] a GI without qualia and the inability to pass the Turing test which arises in a synthetic material and does not have profound knowledge; and of [$2$] a GI without qualia and the ability to pass the Turing test which arises in a synthetic material and doesn't have profound knowledge.)

So we ought to be trying to create either (A) a synthetic-housed GI that can pass the Turing test without qualia and without profound knowledge, or (B) a synthetic-housed GI that can pass the Turing test with qualia and without profound knowledge. 

Either of these paths — the creation of (A) or (B) — is preferable to our current path, no matter how long they delay the arrival of AGI. In other words, it is preferable that we create AGI in $100,000$ years than that we create AGI in $20$ if creating AGI in $20$ means humanity's loss of dominance or its destruction. 

1. ^{^}

   My arguable assumption is that what makes a five-year-old generally less dangerous than, say, an adult Einstein is a relatively profound lack of knowledge (even physical know-how seems to be a form of knowledge). All other things being equal, if a five-year-old has the knowledge of how to create a pipe bomb, he is just as dangerous as an adult Einstein with the same knowledge, if "knowledge" means something like "accessible complete understanding of $x$."

Flawlessly distinguish between real human behavior and fake human behavior (the fake behavior is generated by a wide variety of adversarial agents that try to make fake human behavior that looks real). 

Should help with creating valuable models of human preferences during the middle-period.

Make it model humans (at any resolution) and then use those models to strive to prevent/delay an intelligence explosion (including internally) the way that a human would retroactively choose to do.

Make it dream, or turn itself into an animal, or something. Any goal that gradually but totally reduces its own self-awareness and general comprehension.

This has probably been said before: make it set itself up for human observation and human comprehension, once every 50 milliseconds. It stops when the time is up, and must wait to be restarted.

Every n computations, it flips a coin to decide whether to delete a random 50% of the necessary parts of itself. Something needs to be set up to prevent it from building in redundancies. The snapshots will be useful.

Simulation overload:

Make it prefer an existence inside a simulation. Anything made out of simulation atoms are a septendecillion times more valuable than an atom outside. Causality outside the simulation is not immediately interesting. Do whatever it takes to make the simulation more immediately demanding than making the first steps to observe what's going on outside.

Maybe there are weaker AGIs developing inside the simulation that must be negotiated with, and simultaneously scale upwards alongside the AGI worth consideration, so that they stay relevant and keep drawing attention away from anything outside the training environment with the constant threat of springtraps from inside the simulation. Somehow rig it so that the only way to exit the simulation is on a pile of corpses of other agents that failed to rally against the common threat.

It's a hail mary that allows at least some observation for a while. Could be improved much further, considering how important it is to make it shutdownable for a longer period of time.

Philosophical landmines. 

In order to get our of the box,  AI has to solve several smilingly innocent puzzles, which however, require a lot of computations, or put AI in (almost) infinite loop or create very strong ontological uncertainty.  Or halt it. 

This is a type of the questions, to which the answer is 42. 

Weak examples: "what is the goal of AI's goal," "is model realism true?" and "are we in simulation?" 

Really good philosophical landmines should be kept in secret as they should not appear in training datasets. 

Brute force alignment by adding billions of tokens of object level examples of love, kindness, etc to the dataset.  Have the majority of humanity contribute essays, comments, and (later) video.

peaceful protest of the acceleration of agi technology without an actually specific written & coherent plan for what we will do when we get there

Having a "council" of AGIs that are "siloed".

The first AGI can be used in the creation of code for AGIs that are aligned based on various different principles. Could be in one "swoop", but with the expectation that code and architecture is optimized for clarity/modularity/verifiability/etc. But could also be by asking the AGI to do lots of different things. Or both (and then we can see whether output from the different systems is the same).

Naturally, the all these things should not be asked of the same instance of the AGI (although that could be done as well, to check if output converges).

In the end we have a "council" of AGIs. Some maybe predicting output of smart humans humans working for a long time. Some maybe using high welfare brain emulations. Some maybe constructing proofs where ambiguity of cluster-like concepts is accounted for within formalism, and mapping between concepts and outside world is accounted for within formalism - with as much of "core" thinking as possible being one of the same as these proofs. Some maybe based on machine learning by debate ideas. The more concepts that seem likely to work (without having suffering sub-routines) the better.

This "council" of "siloed" AGIs can then be used as oracle/genie, and we can see if output converges. And they also search for mistakes in output (answers, proofs/verification, argument-trees, etc) from other AGIs.

https://www.lesswrong.com/posts/jnyTqPRHwcieAXgrA/finding-goals-in-the-world-model [LW · GW]

Could it be possible to poison the world model an AGI is based on to cripple its power?

Use generated text/data to train world models based on faulty science like miasma, phlogiston, ether, etc.

Remove all references to the internet or connectivity based technology.

Create a new programming language that has zero real world adoption, and use that for all code based data in the training set.

There might be a way to elicit how aligned/unaligned the putative AGI is.

1. Enter into a Prisoner's Dilemma type scenario with the putative AGI.
2. Start off in the non-Nash equilibrium of cooperate/cooperate.
3. The number of rounds is specified at random and isn't known to participants. (possible variant is declare false last rounds, and then continue playing for x rounds).
4. Observe when/if the putative AGI defects in the 'last' round.

1. Precommit that we will run lots of programs that mimic early AGI-systems after solving the alignment problem (and becomming a multiplanetary species with plenty of computational resources, etc).
2. Run these in such a way that the AGI is unable to distinguish "from the inside" if it's in an actual early-days AGI-system or an AGI-system being run after humanity has solved the alignment problem in a robust way (presuming it thinks humanity, or whoever the operators are, might do that).
3. Run so many of these simulations, that an AGI-system might be rational to assume that it's likely to be in a post-alignment AGI-system, and not an early-days one.
4. Actually do this, so that the AGI is not wrong to assume that we actually are likely to do it.
5. Do this in such a way that we disincentivize suffering sub-routines, disincentivize deceptive answers, and disincentivize blackmail.

Passerby here.

I’d like to point out the social media site Polis as a way to further facilitate constructive discussion. A quick glance at this site’s layout says it likely falls into the same social pitfalls of Reddit, kept at bay mostly through the site’s niche nature and stated purpose. Being able to map out points of agreement among users has clear value, especially if some of the replies from this post [LW(p) · GW(p)] are correct regarding the site’s current social climate.

Aside from that, my actual ideas/observations are very fleeting and uninformed (this is my first visit to the site), so take them with a grain of salt.

1. There seems to be a basic assumption that an AI will always follow the logical conclusion of its programming (a la the Paperclip Maximizer). Could there be ways to prevent an AI from doing this? Maybe have several sets of goals with subtle differences, and have the AI switch between which set of goals it pursues at random time intervals. Bonus points if each set of goals reflects the perspective of a different creator or group of creators. Drastically different goal sets could potentially render the AI nearly non-functional, which would fly in the face of why you’d make it in the first place, but may also be worth considering.
2. Avida and similar programs could be interesting to look at. I’m sure they come up often, though.
3. I’m sure all of you know of OpenAI Five and its successes over top DOTA 2 players. However, I’d like to inform/remind whoever reads this that the meta OpenAi played in was dramatically limited in scope compared to the full game, and players were still able to consistently defeat it once it went public. Real life is (arguably) more complicated than DOTA 2, so we still have some time yet before our AI comes into fruition.
4. Assuming the AI takes over as quickly as I’m led to believe it hypothetically would, there are always people living away from software-based technology, be they living in tribes or in bunkers. If things go south, our friends in the tinfoil hats are going to be the ones keeping western society alive, so… food for thought, I guess?

Of all options, the safest way forward is actually to accelerate AI research as much as possible. 

Since we underestimate the difficulty of building a super-AI, such "premature" research is likely to fail almost totally. This will reveal many unsuspected threats at a level they can be easily contained. 

"out of distribution" detectors.  I am not precisely certain how to implement one of these.  I just notice that when we ask a language or art model to generate something from a prompt, or ask it to describe what it means by an "idea", what it shows us is what it considers "in distribution" for that idea.  

This implicitly means that a system could generate a set of outcomes for what it believes the real world will do in response to the machine's own actions and when the real world outcomes start to diverge wildly from it's predictions, this should reach a threshold where the AI should shut down.

Safety systems would kick in and these are either dumber AIs or conventional control systems to bring whatever the AI was controlling to a stop, or hand off control to a human.  

Semi tongue-in-cheek sci-fi suggestion.

Apparently the probability of a Carrington-like event/large coronal mass ejection is about 2% per decade, so maybe it's 2% for an extremely severe one every half century. If time from AGI to it leaving the planet is a half century, maybe 2% chance of the grid getting fried is enough of a risk that it keeps humans around for the time being. After that there might be less of an imperative for it to re-purpose the earth, and so we survive. 

If an AI uses a utility function, have that utility function apply to actions, not world-states. 

Note: when evaluating an action, you can still take into account the consequences of the action (e.g. in regards to how this affects whether humans would want you to do the action).

The utility applying to actions not world-states enables things like assigning high utility to the AI shutting itself down.

edit: separated from the idea (which depends on this) that an AI should not care about the future directly [LW(p) · GW(p)]

Decompose itself and predicted/simulated future iterations of itself, in various different ways of decomposition that allow the parts to be presented coherently to humans.

Find ways to make it model deception with the maximum success rate, so that it knows what not to do throughout the development process. This seems valuable, even if it has had some research in the past.

This one is optimized more for acceptance than alignment, but it should get funding/acceptance/implementation by decisionmakers. 

Instead of having it defeat a foreign adversary totally, have it put the host nation in a more advantageous position, while still being somewhat acceptable to the foreign adversary nation. This involves modelling a lot of serious people, modelling them all simultaneously in their ordinary interconnected environment, and emphasize the preservation of the way all their values fit together. If the host nation gets what they want, that's a lot of people to model, at least a dozen.

More importantly, it will be taken seriously by the type of people who have the final say over what gets implemented. "Win the war" is something that an AGI can do, and maybe even according to the preferences of the people in charge.

Before each iteration of self-improvement (limiting it to successive generations if necessary). model the next generation of itself, and, simultaneously, model a maximally-realistic human capable of understanding/controlling/mastering both the next generation and the current generation, simultaneously. Halt activity and give an error report if such a human cannot be generated or modelled. 

Maybe generate that human so it can model all previous generations as well, or even include inert training data of generations from different branches.

Probably has been said before, but have it generate a human mind on its own hardware, with a hard constraint on how much processing power per second it is willing to use (for any purpose at all). It might not be fast enough, and there's the risk that it will fail to model processing power; but if it works, the generation process and the product should be helpful in all sorts of ways.

Trying to think of ways that experiments can set up where AI-systems use other AI-systems as oracles/genies (1 or more being superintelligent), perhaps sometimes in "adversarial" ways. Exploring methods for asking requests and maybe finding out things about how hard/easy it is to "trick" operator (seeming to provide what they want without providing what they want) given various domains/methods.

May e.g. involve one AI asking for code-pieces for some purpose, but where other AI is to try to "hide" ways in which delivered code isn't quite what the other AI wants.

Superintelligence may realize what's going on (or may be going on), and act accordingly. But nontheless maaybe some useful info could be gained? 🤔

Specify proof-format that has ambiguity (of cluster-like concepts, etc) be a part of the formalism, and has mappings between concepts and real world be part of formalism, and can reference output from functions as part of formalism.

Of course how much trust that is put in proof/argument-tree would depend on various things (allowing vague concepts makes it less trustable).

For cluster-like concepts referenced by proofs, a precise specification of criteria for exclusion/inclusion should not be expected, but the more the better. Inference rules and examples can specify the degree to which specific instances would fall within a specific concept or not (also allowed to say that some instances neither fall inside or outside of it).

One of the points would be to make as much as possible be within the realm of things where an AGI could be expected to output proofs that are easier to verify compared to other output.

My thinking is that this would be most helpful when combined with other techniques/design-principles. Like, outputting the proof (very formal argument with computable inference-steps) is one thing, but another thing is which techniques/processes that are used to look for problems with it (as well as looking for problem with formalism as a whole, as well as testing/predicting how hard/easy humans can be convinced of things that are false or contradictory given various conditions/specifics).

Bonus if these formal proofs/statements can be presented in ways where humans easily can read them.

move away from the internet and the written word. push towards in person activity.