FactorialCode's Shortform 2019-07-30T22:53:24.631Z · score: 2 (1 votes)


Comment by factorialcode on Artificial general intelligence is here, and it's useless · 2019-10-24T02:09:46.893Z · score: -2 (3 votes) · LW · GW

I'm just gonna leave this here and highlight points 2 and 3.

Comment by factorialcode on Long-term Donation Bunching? · 2019-09-28T21:00:20.513Z · score: 1 (1 votes) · LW · GW

then what happens if the 'donor' doesn't choose to convert it to a donation?

Same thing that happens when you fail to meet the requirements for any other financial instrument. You go into debt, your credit rating takes a plunge, and debt collectors will start harassing you for money.

Comment by factorialcode on New Member · 2019-09-26T22:59:24.000Z · score: 12 (4 votes) · LW · GW

I'm not aware of anything on LW, but on reddit there's r/HPMOR for just HPMOR and r/rational for ratfics in general.

Comment by factorialcode on Running the Stack · 2019-09-26T20:59:19.746Z · score: 1 (1 votes) · LW · GW

I feel like a priority queue applies best to me.

Comment by factorialcode on Deducing Impact · 2019-09-25T00:10:04.178Z · score: 4 (3 votes) · LW · GW

I'll take a crack at this.

To a first order approximation, something is a "big deal" to an agent if it causes a "large" swing in its expected utility.

Comment by factorialcode on FactorialCode's Shortform · 2019-09-20T20:29:30.111Z · score: 22 (8 votes) · LW · GW

Inspired by the recent post on impact measures, I though of an example illustrating the subjective nature of impact.

Consider taking the action of simultaneously collapsing all the stars except our sun into black holes.(Suppose you can somehow do this without generating supernovas.)

To me, this seems like a highly impactful event, potentially vastly curtailing the future potential of humanity.

But to an 11th century peasant, all this would mean is that the stars in the night sky would slowly go out over the course of millenia. Which would have very little impact on the peasants life.

Comment by factorialcode on Sayan's Braindump · 2019-09-19T03:00:58.642Z · score: 2 (2 votes) · LW · GW

I find having a skateboard is a compact way to shave minutes off of the sections of my commute where I would otherwise have to walk. It turns a 15 minute walk to the bus stop into a 5 minute ride, which adds up in the long run.

Comment by factorialcode on Don't depend on others to ask for explanations · 2019-09-18T21:38:11.867Z · score: 13 (6 votes) · LW · GW

On the other end, when writing, I feel that recursively expanding upon your ideas to explain them and back them up is a skill that needs to be learned and practiced.

When I come up with an idea, I suspect that I do so with whatever abstractions and ideas my brain has on hand, but those are probably not the same as those of the target audience. When I start writing, I'll end up writing a ~1-2 sentence summary that I feel captures what I'm trying to get across. Then I need to make a conscious effort to unpack each of those component ideas and back them up with reasoning/examples to support my claims, this get's harder as I further unpack statements, because I'm more inclined to take those claims for granted. I suspect that this gets easier with practice, and that I'll be able to write progressively more detailed posts as time goes on.

Does anyone else feel that this is a bottleneck on their ability to explain things?

Comment by factorialcode on [AN #63] How architecture search, meta learning, and environment design could lead to general intelligence · 2019-09-18T18:53:16.608Z · score: 3 (1 votes) · LW · GW

Due to the vast data requirements, most of the environments would have to be simulated. I suspect that this will make the agenda harder than it may seem at first glance -- I think that the complexity of the real world was quite crucial, and that simulating environments that reach the appropriate level of complexity will be a very difficult task.

I'm skeptical of this. I think that it's well within our capabilities to create a virtual environment with a degree of complexity comparable to the ancestral environment. For instance, the development of minecraft with all of it's complexity can be upper bounded by the cost of paying ~25 developers over the course of 10 years. But the core features of the game, minecraft alpha, were done by a single person in his spare time over 2 years.

I think a smallish competent team with a 10-100 million dollar budget could easily throw together a virtual environment with ample complexity, possibly including developing FPGA's or ASICs to run it at the required speed.

Comment by factorialcode on The unexpected difficulty of comparing AlphaStar to humans · 2019-09-18T15:34:01.461Z · score: 2 (2 votes) · LW · GW

I found a youtube channel that has been providing commentary on suspected games of AlphaStar on the ladder. They're presented from a layman's perspective, but they might be valuable for people to get an idea of what the current AI is capable of.

Comment by factorialcode on Three Stories for How AGI Comes Before FAI · 2019-09-18T03:54:19.065Z · score: 5 (2 votes) · LW · GW

Trying to create an FAI from alchemical components is obviously not the best idea. But it's not totally clear how much of a risk these components pose, because if the components don't work reliably, an AGI built from them may not work well enough to pose a threat.

I think that using alchemical components in an possible FAI can lead to a serious risk if the people developing it aren't sufficiently safety conscious. Suppose that either implicitly or explicitly, the AGI is structured using alchemical components as follows:

  1. A module for forming beliefs about the world.
  2. A module for planning possible actions or policies.
  3. A utility or reward function.

In the process of building an AGI using alchemical means, all of the above will be incrementally improved to a point where they are "good enough". The AI is forming accurate beliefs about the world, and is making plans to get things that the researchers want. However, in a setup like this, all of the classic AI safety concerns come into play. Namely, the AI has an incentive to upgrade the first 2 modules and preserve the utility function. Since the utility function is only "good enough", this becomes the classic setup for Goodhart and we get UFAI.

Even in a situation where the AI does not participate in it's own further redesign, it's effective ability to optimise the world increases as it gets more time to interact with it. As a result, an initially well behaved AGI might eventually wander into a region of state space where it becomes unfriendly using only capabilities comparable to that of a human.

That said, remains to be seen if researchers will build AI's with this kind of architecture without additional safety precautions. But we do see model free RL variants of this general architecture such as Guided Cost Learning and Deep reinforcement learning from human preferences.

As a practical experiment to validate my reasoning, one could replicate the latter paper using a weak RL algorithm, and then see what happens if it's swapped out with a much stronger algorithm after learning the reward function. (Some version of MPC maybe?)

Comment by factorialcode on FactorialCode's Shortform · 2019-09-17T03:32:13.520Z · score: 5 (2 votes) · LW · GW

Meta-philosophy hypothesis: Philosophy is the process of reifying fuzzy concepts that humans use. By "fuzzy concepts" I mean things where we can say "I know it when I see it." but we might not be able to describe what "it" is.

Examples that I believe support the hypothesis:

  • This shortform is about the philosophy of "philosophy" and this hypothesis is an attempt at an explanation of what we mean by "philosophy".

  • In epistemology, Bayesian epistemology is a hypothesis that explains the process of learning.

  • In ethics, an ethical theory attempts to make explicit our moral intuitions.

  • A clear explanation of consciousness and qualia would be considered philosophical progress.

Comment by factorialcode on What You See Isn't Always What You Want · 2019-09-15T17:55:39.991Z · score: 3 (1 votes) · LW · GW

do you think any reasonable extension of these kinds of ideas could get what we want?

Conditional on avoiding Goodhart, I think you could probably get something that looks a lot like a diamond maximiser. It might not be perfect, the situation with the "most diamond" might not be the maximum of it's utility function, but I would expect the maximum of it's utility function will still contain a very large amount of diamond. For instance, depending on the representation, and the way the programmers baked in the utilty function, it might have a quirk in it's utility function of only recognizing something as a diamond if it's stereotypically "diamond shaped". This would bar it from just building pure carbon planets to achieve it's goal.

IMO, you'd need something else outside of the ideas presented to get a "perfect" diamond maximizer.

Comment by factorialcode on What You See Isn't Always What You Want · 2019-09-14T16:59:20.434Z · score: 5 (2 votes) · LW · GW

Do you think we could build a diamond maximizer using those ideas, though?

They're definitely not sufficient, almost certainly. A full fledged diamond maximizer would need far more machinery, if only to do the maximization and properly learn the representation.

The concern here is that the representation has to cleanly demarcate what we think of as diamonds.

I think this touches on a related concern, namely goodharting. If we even slightly miss-specify the utility function at the boundary and the AI optimize in an unrestrained fashion, we'll end up with weird situations that are totally de-correlated with what we we're initially trying to get the AI to optimize.

If we don't solve this problem, I agree, the problem is extremely difficult at best and completely intractable at worst. However, If we can reign in goodharting, then I don't think things are intractable.

To make the point, I think the problem of a AI goodharting a representation is very analogous to the problems being tackled in the field of adversarial perturbations for image classification. In this case, the "representation space" is the image itself. The boundaries are classification boundaries set by the classifying neural network. The optimizing AI that goodharts everyting is usually just some form or gradient decent.

The field started when people noticed that even tiny imperceptible perturbations to images in one class would fool a classifier into thinking it was an image from another class. The interesting thing is that when you take this further, you get deep dreaming and inceptionism. The lovecraftian dog-slugs that would arise from the process are are result of the local optimization properties of SGD combined with the flaws of the classifier. Which, I think, is analogous to goodharting in the case of a diamond maximiser with a learnt ontology. The AI will do something weird, it becomes convinced that the world is full of diamonds. Meanwhile, if you ask a human about the world it created, "lovecraftian" will probably precede "diamond" in the description.

However, the field of adversarial examples seems to indicate that it's possible to at least partially overcome this form of goodharting and, by anaogy, the goodharting that we would see with a diamond maximiser. IMO, the most promising and general solution seems to be to be more bayesian, and keep track of the uncertainty associated with class label. By keeping track of uncertainty in class labels, it's possible to avoid class boundaries altogether, and optimize towards regions of the space that are more likely to be part of the desired class label.

I can't seem to dig it up right now, but I once saw a paper where they developed a robust classifier. When they used SGD to change a picture from being classified as a cat to being classified as a dog, the result was that the underlying image went from looking like a dog to looking like a cat. By analogy, an diamond maximizer with a robust classification of diamonds in it's representation should actually produce diamonds.

Overall, adversarial examples seem to be a microcosm for evaluating this specific kind of goodharting. My optimism that we can do robust ontology identification is tied to the success of that field, but at the moment the problem doesn't seem to be intractable.

Comment by factorialcode on What You See Isn't Always What You Want · 2019-09-13T18:48:06.620Z · score: 5 (2 votes) · LW · GW

I'm personally far more optimistic about ontology identification. Work in representation learning, blog posts such as OpenAI's sentiment neuron, and style transfer, all indicate that it's at least possible to point at human level concepts in a subset of world models. Figuring out how to refine these learned representations to further correspond with our intuitions, and figuring out how to rebind those concepts to representations in more advanced ontologies are both areas that are neglected, but they're both problems that don't seem fundamentally intractable.

Comment by factorialcode on What You See Isn't Always What You Want · 2019-09-13T17:20:40.871Z · score: 3 (1 votes) · LW · GW

Under this view, alignment isn’t a property of reward functions: it’s a property of a reward function in an environment. This problem is much, much harder: we now have the joint task of designing a reward function such that the best way of stringing together favorable observations lines up with what we want. This task requires thinking about how the world is structured, how the agent interacts with us, the agent’s possibilities at the beginning, how the agent’s learning algorithm affects things…

I think there are ways of doing this that don't involve explicitly working through what observation sequences lead to good outcomes. AFAICT this was originally outlined in Model Based Rewards quite a while ago. Essentially, the idea is to make the reward (or even better, utilty) a function of the agent's internal model of the world. Then when the agent goes to make a decision, the utility of the worlds where the agent does and does not make take an action are compared. Doing things this way has a couple of nice properties, including eliminating the incentive to wirehead, and making it possible to specify utilities over possible worlds rather than just what the AI sees.

The relevant point however, is that it takes the problem from trying to pin down what chains of events lead to good outcomes, and splits it into a problem of identifying good and bad worldstates in the agents model and building an accurate model of the world. This is because an agent with an accurate model of the world will be able to figure out what sequence of actions and observations lead to any given worldstate.

Comment by factorialcode on What are concrete examples of potential "lock-in" in AI research? · 2019-09-09T17:31:34.285Z · score: 6 (3 votes) · LW · GW

This isn't quite "lock in", but it's related in the sense that an outside force shaped the field of "deep learning".

I suspect the videogame industry, and the GPUs we're developed for it has locked in the type of technologies we now know as deep learning. GPU's were originally ASICs developed for playing videogames, so there are specific types of operations they were optimized to perform.

I suspect that neural network architectures that leveraged these hardware optimizations outperformed other neural networks. Conv nets and Transformers are probably evidence of this. The former leverages convolution, and the latter leverages matrix multiplication. In turn, GPUs and ASICs have been optimized to run these successful neural networks faster, with NVIDIA rolling out Tensor Cores and Google deploying their TPUs.

Looking back, it's hard to say that this combination of hardware and software isn't a local optima, and that if we were to redesign the whole stack from the bottom up, that the technologies with the capabilities of modern "deep learning" wouldn't look completely different.

It's not even clear how one could find another optimum in the space of algorithms+hardware at this point either. The current stack benefits both from open source contributions and massive economies of scale.

Comment by factorialcode on Dual Wielding · 2019-08-27T20:08:30.774Z · score: 4 (5 votes) · LW · GW

This of course, leads naturally to a new app/os idea. We need a way to semi-seamlessly use two phones together as if they were a single phone. Like dual monitors, but with phones.

Comment by factorialcode on I'm interested in a sub-field of AI but don't know what to call it. · 2019-08-25T17:20:33.308Z · score: 8 (5 votes) · LW · GW

Can you list some papers that are vaguely in line with the kind of research you're looking for?

Comment by factorialcode on Is LW making progress? · 2019-08-24T18:05:26.071Z · score: 18 (10 votes) · LW · GW

I've been lurking on LW for many years, and overall, my impression is that there's been steady progress. At the end of a very relevant essay from Scott, way back in 2014, he states:

I find this really exciting. It suggests there’s this path to be progressed down, that intellectual change isn’t just a random walk. Some people are further down the path than I am, and report there are actual places to get to that sound very exciting. And other people are around the same place I am, and still other people are lagging behind me. But when I look back at where we were five years ago, it’s so far back that none of us can even see it anymore, so far back that it’s not until I trawl the archives that realise how many things there used to be that we didn’t know.

5 years later, I still think that this still applies. It explains some of the rehashing of topics that were previously discussed. All the things I'm going to point out below are some of the most notable insights I can remember.

When LW was relatively inactive, there were essays from the surrounding sphere that stuck with me. For instance, this essay by paul chrisiano. Which was, for me, the first clear examples of how epistemically irrational things that humans do can actually be instrumentally rational in the right setting, something that wasn't really discussed much in the original sequences.

I think LW has also started focusing a fair bit on group rationality, along with norms and systems that foster it. That can be seen by looking at how the site has changed, along with all of the meta discussion that follows. I think that in pursuit of this, there's also been quite a bit of discussion about group dynamics. Most notable for me was Scott's Meditations on Moloch and Toxoplasma of rage. Group rationality looks like a very broad topic, and insightful discussion about it are still happening now. Such as this discussion on simulacra levels.

On the AI safety side, I feel like there's been an enormous amount of progress. Most notably for me was Stuart Armstrong's post: Humans can be assigned any values whatsoever.. Along with all the discussion about the pros and cons, of different methods of achieving alignment, such as AI Safety Via Debate, HCH, and Value Learning.

As for the sequences, I don't have any examples off the top of my head, but I think at least some of the quoted psychology results that were referenced failed to replicate during the replication crisis. But I can't remember too much else about them, since it's been so long since I read them. Many of the core idea feel like they've become background knowledge that I take for granted, even if I've forgotten their original source.

Comment by factorialcode on Computational Model: Causal Diagrams with Symmetry · 2019-08-22T21:23:49.679Z · score: 5 (5 votes) · LW · GW

This isn’t quite the same as weighting by minimum description length in the Solomonoff sense, since we care specifically about symmetries which correspond to function calls - i.e. isomorphic subDAGs. We don’t care about graphs which can be generated by a short program but don’t have these sorts of symmetries.

Can you elaborate on this? What would be an example of a graph that can be generated by a short program, but that does not have these sorts of symmetries?

My intuition is that the class of processes your describing is Turing complete, and therefore can simulate any Turing machine, and is thus just another instance of Solomonoff induction with a different MDL constant.

Edit: Rule 110 would be an example.

Comment by factorialcode on Cerebras Systems unveils a record 1.2 trillion transistor chip for AI · 2019-08-20T22:27:23.272Z · score: 2 (3 votes) · LW · GW

Allow me to speculate wildly.

I don't actually think this is going to make that big of a difference, at least for current AI research. The main reason is because I think the main hardware bottlenecks to better AI performance are performance/$ and performance/W and memory bandwidth. This is because, so far, most large scale DL algorithms have shown almost embarrassingly parallel scaling, and a good amount of time is wasted just saving and loading NN activations for the back-prop algorithm.

This technology probably won't lead to any major performance improvements in terms of performance/$ or performance/W. Those will have already come from dedicated DL chips such as Google's TPUs, because this essentially a really big dedicated DL chip. The major place for improvement is memory bandwidth, which according to the article, is an impressive 9PB per second, and 10,000 times than what's on a V100 GPU, but with only 18GB of ram, that's going to severely constrain the size of models that can be trained, so I don't think it will be useful for training better models.

Might be good for inference though.

Comment by factorialcode on FactorialCode's Shortform · 2019-08-15T16:17:33.134Z · score: 4 (6 votes) · LW · GW

I notice that there's a fair bit of "thread necromancy" on LessWrong. I don't think it's a bad thing, but I think it would be cool to have an option to filter comments based on the time gap between when the post was made and when the comment was made. That way it's easier to see to see what the discussion was like around the time when the post was made.

On a related note, does LessWrong record when upvotes are made? It would also be cool to have a "time-machine" to see how up-votes and down-votes in a thread evolve over time. Could be good for analysing the behaviour of threads in the short term, and a way to see how community norms change in the long term.

Comment by factorialcode on "Designing agent incentives to avoid reward tampering", DeepMind · 2019-08-15T02:43:29.365Z · score: 4 (5 votes) · LW · GW

I think this is a good sign, this paper goes over many of the ideas that the RatSphere has discussed for years, and Deepmind is giving those ideas publicity. It also brings up preliminary solutions, of which, "Model Based Rewards" seems to go farthest in the right direction.(Although even the paper admits the idea's been around since 2011)

However, the paper is still phrasing things in terms of additive reward functions, which don't really naturally capture many kinds of preferences (such as those over possible worlds). I also feel that the causal influence diagrams, when unrolled for multiple time steps, needlessly complicate the issues being discussed. Most interesting phenomena in decision theory can be captured by simple 1 or 2 step games or decision trees. I don't see the need to phrase things as multi-timestep systems. The same goes for presenting the objectives in terms of grid worlds.

Overall, the authors seem to still be heavily influenced by the RL paradigm. It's a good start, we'll see if the rest of the AI community notices.

Comment by factorialcode on Could we solve this email mess if we all moved to paid emails? · 2019-08-11T17:51:10.769Z · score: 2 (2 votes) · LW · GW

I like this idea, and I think for it to take off, it would have to be implemented by easily piggy backing off of the existing email system. If I could download some kind of browser-extension that allowed me to accept payment for emails while letting me continue to use my existing email, I would consider having that option.

However, I think this could face some adoption problems. I could easily imagine there being negative social consequences to advertising having a paid email address. As it makes the statement "I am more likely to ignore your messages unless you pay me for my time." common knowledge.

Comment by factorialcode on AI Alignment Open Thread August 2019 · 2019-08-06T20:52:22.953Z · score: 1 (1 votes) · LW · GW

Interesting. I had the Nash equilibrium in mind, but it's true that unlike a dollar auction, you can de-escalate, and when you take into account how your opponent will react to you changing your strategy, doing so becomes viable. But then you end up with something like a game of chicken, where ideally, you want to force your opponent to de-escalate first, as this tilts the outcomes toward option C rather than B.

Comment by factorialcode on AI Alignment Open Thread August 2019 · 2019-08-06T12:40:16.533Z · score: 8 (4 votes) · LW · GW

At some point, there was definitely discussion about formal verification of AI systems. At the very least, this MIRIx event seems to have been about the topic.

From Safety Engineering for Artificial General Intelligence:

An AI built in the Artificial General Intelligence paradigm, in which the design is engineered de novo, has the advantage over humans with respect to transparency of disposition, since it is able to display its source code, which can then be reviewed for trustworthiness (Salamon, Rayhawk, and Kramár 2010; Sotala 2012). Indeed, with an improved intelligence, it might find a way to formally prove its benevolence. If weak early AIs are incentivized to adopt verifiably or even provably benevolent dispositions, these can be continually verified or proved and thus retained, even as the AIs gain in intelligence and eventually reach the point where they have the power to renege without retaliation (Hall 2007a).

Also, from section 2 of Agent Foundations for Aligning Machine Intelligence with Human Interests: A Technical Research Agenda:

When constructing intelligent systems which learn and interact with all the complexities of reality, it is not sufficient to verify that the algorithm behaves well in test settings. Additional work is necessary to verify that the system will continue working as intended in application. This is especially true of systems possessing general intelligence at or above the human level: superintelligent machines might find strategies and execute plans beyond both the experience and imagination of the programmers, making the clever oscillator of Bird and Layzell look trite. At the same time, unpredictable behavior from smarter-than-human systems could cause catastrophic damage, if they are not aligned with human interests (Yudkowsky 2008). Because the stakes are so high, testing combined with a gut-level intuition that the system will continue to work outside the test environment is insufficient, even if the testing is extensive. It is important to also have a formal understanding of precisely why the system is expected to behave well in application. What constitutes a formal understanding? It seems essential to us to have both (1) an understanding of precisely what problem the system is intended to solve; and (2) an understanding of precisely why this practical system is expected to solve that abstract problem. The latter must wait for the development of practical smarter than-human systems, but the former is a theoretical research problem that we can already examine.

I suspect that this approach has fallen out of favor as ML algorithms have gotten more capable while our ability to prove anything useful about those algorithms has heavily lagged behind. Although deep mind and a few others are is still trying.

Comment by factorialcode on AI Alignment Open Thread August 2019 · 2019-08-06T03:18:16.656Z · score: 9 (4 votes) · LW · GW

I agree that the coordination games between nukes and AI are different, but I still think that nukes make for a good analogy. But not after multiple parties have developed them. Rather I think key elements of the analogy is the game changing and decisive strategic advantage that nukes/AI grant once one party develops them. There aren't too many other technologies that have that property. (maybe the bronze-iron age transition?)

Where the analogy breaks down is with AI safety. If we get AI safety wrong there's a risk of large permanent negative consequences. A better analogy might be living near the end of WW2, but if you build a nuclear bomb incorrectly, it ignites the atmosphere and destroys the world.

In either case, under this model, you end up with the following outcomes:

  • (A): Either party incorrectly develops the technology
  • (B): The other party successfully develops the technology
  • (C): My party successfully develops the technology

and generally a preference ordering of A<B<C, although a sufficiently cynical actor might have B<A<C.

If there's a sufficiently shallow trade-off between speed of development and the risk of error, this can lead to a dollar auction like dynamic where each party is incentivized to trade a bit more risk in order to develop the technology first. In a symmetric situation without coordination, the equilibrium nash equilibrium is all parties advancing as quickly as possible to develop the technology and throwing caution to the wind.

Comment by factorialcode on FactorialCode's Shortform · 2019-08-04T17:28:29.870Z · score: 4 (2 votes) · LW · GW

I've been thinking about how one could get continuous physics from a discrete process. Suppose you had a differential equation, and you wanted to make a discrete approximation to it. Furthermore, suppose you had a discrete algorithms for simulating this differential equation that takes in a parameter, say, dt which controls the resolution of the simulation. As dt tends toward zero, the dynamics of the simulated diff eq will tend towards the dynamics of the real diff eq.

Now suppose, we have a a turing machine that implements this algorithm as a subroutine. More precisely, the turing machine runs a simulations of diff equation at a resolutions of 1 then 1/2, then 1/3 and so on and so forth.

Finally, suppose their we're a conscious observer in this simulation, at what resolution would they expect their physics to be simulated? Depending on one's notion of anthropics, one could argue that at any resolution, there is a finite amount of observers in lower resolution simulations, but an infinite amount in higher resolution simulations. Consequently, the observer should expect to live in a universe with continuous physics.


Comment by factorialcode on Tetraspace Grouping's Shortform · 2019-08-02T03:16:32.549Z · score: 8 (4 votes) · LW · GW

Was this meant to be a reply to my bit about the Solmonoff prior?

If so, in the algorithmic information literature, they usually fix the unnormalizability stuff by talking about Prefix Turing machines. Which corresponds to only allowing TM descriptions that correspond to a valid Prefix Code.

But it is a good point that for steeper discounting rates, you don't need to do that.

Comment by factorialcode on benwr's unpolished thoughts · 2019-08-01T20:49:03.819Z · score: 1 (1 votes) · LW · GW

I actually like the fact that I don't immediately know who is speaking to who in a thread. I feel like it prevents me from immediately biasing my judgment of what a person is saying before they say it.

Comment by factorialcode on FactorialCode's Shortform · 2019-08-01T20:29:45.200Z · score: 4 (3 votes) · LW · GW

The standard Solomonoff prior discounts hypotheses by 2^(-l), where l is the number of bits required to describe them. However, we can easily imagine a whole class of priors, each with a different discount rate. For instance, one could discount by 1/Z2^(-(2l)) where Z is a normalizing factor to get probabilities to add up to one. Why do we put special emphasis on this rate of discounting rather than any other rate of discounting?

I think that we can justify this discount rate with the principle of maximum entropy, as distributions with steeper asymptotic discounting rates will have lower entropy than distributions with shallower asymptotic discounting rates and any distribution with a shallower discounting rate than 2^(-l) would (probably) diverge and therefore constitute an invalid probability distribution.

Are there arguments/situations justifying steeper discounting rates?

Comment by factorialcode on FactorialCode's Shortform · 2019-07-30T22:53:24.756Z · score: 6 (4 votes) · LW · GW

What should be the optimal initial Karma for a post/comment?

By default on reddit and lesswrong, posts start with 1 karma, coming from the user upvoting themselves. On lesswrong right now, it appears that this number can be set higher by strongly upvoting yourself. But this need not be the case. Posts and comments could start with either positive or negative karma. If posts start with larger positive karma, this might incentivize people to post/comment more often. Likewise, if posts or comments start off with negative karma, this acts as a disincentive to post.

A related idea would be to create a special board where posts/comments start off with large negative karma, but each upvote from users would give the poster more karma than usual. As a result, people would only post there if they expected their post to "break-even" in terms of Karma.

Comment by factorialcode on Does it become easier, or harder, for the world to coordinate around not building AGI as time goes on? · 2019-07-30T14:46:22.187Z · score: 10 (5 votes) · LW · GW

I suspect that one of the factors that will make coordinating to not build AGI harder is that the incentive to build AGI will become greater for a larger amount of people. Right now, there's a large class of people who view AI as a benign technology, that will bring about large amounts of economic growth, and that it's effects are going to be widespread and positive. I think this position is best captured by Andew Ng when he says "AI is the new electricity". Likewise, the Whitehouse states "Artificial intelligence holds the promise of great benefits for American workers, with the potential to improve safety, increase productivity, and create new industries we can’t yet imagine.".

However, as time goes by AI capabilities grow and so will public demonstrations of what's possible with AI. This will cause people to revise upwards their beliefs about the impact/power of AI and AGI and drag far more actors into the game. I think that if the Whitehouse shared the views of DeepMind or OpenAI on AGI, they wouldn't hesitate to start the equivalent of a second Manhattan project.

Comment by factorialcode on Thoughts on the 5-10 Problem · 2019-07-19T00:52:53.376Z · score: 2 (1 votes) · LW · GW

How would f() map 10 to 0? Wouldn't that require that from

A() := 10
U() := 
     if A() = 10
         return 10
     if A() = 5
         return 5

there's a proof of

U() = 0


My understanding is that in the original formulation, the agent takes it's own definition along with a description of the universe and looks for proofs of the form

[A() = 10 -> U() = x] & [A() = 5 -> U() = y ]

But since "A()" is the same in both sides of the expression, one of the implications is guaranteed to be vacuously true. So the output of the program depends on the order in which it looks for proofs. But here f looks for theorems starting from different axioms depending on it's input, so "A()" and "U()" in f(5) can be different than "A()" and "U()" when f(10).

Comment by factorialcode on Thoughts on the 5-10 Problem · 2019-07-18T20:08:40.064Z · score: 2 (1 votes) · LW · GW

If I can try and make your solution concrete for the original 5-10 problem. Would it look something like this?

A() := 
    let f(x) :=
         Take A() := x as an axiom instead of A() := this function
         Take U() := to be the U() in the original 5-10 problem
         Look for a proof of "U() = y"
         return y
         return argmax f(x) 
    where x in {5,10}
Comment by factorialcode on Please give your links speaking names! · 2019-07-12T17:37:13.921Z · score: 2 (1 votes) · LW · GW

As others have said, one of the downsides of doing this is that adding a readable name interrupts the flow of the text. This is especially a problem if you're skimming the article, because the expanded text will catch your attention even if it's not a central point the article is trying to make. As for printing, I wonder what fraction of people actually do this, to see if the change is justified.

Comment by factorialcode on The AI Timelines Scam · 2019-07-12T16:57:19.476Z · score: 6 (4 votes) · LW · GW

I think a lot of the animosity that Gary Markus drew was less that some of his points were wrong, and more that he didn't seem to have a full grasp of the field before criticizing it. Here's an r/machinelearning thread on one of his papers. Granted, r/ML is not necessarily representative of the AI community, especially now, but you see some people agreeing with some of his points, and others claiming that he's not up to date with current ML research. I would recommend people take a look at the thread, to judge for themselves.

I'm also not inclined to take any twitter drama as strong evidence of the attitudes of the general ML community, mainly because twitter seems to strongly encourage/amplify the sort of argumentative shit-flinging pointed out in the post.

Comment by factorialcode on AGI will drastically increase economies of scale · 2019-07-06T00:22:28.629Z · score: 3 (2 votes) · LW · GW

I suppose that's true. Although assuming that the company has developed intent aligned AGI, I don't see why the entire branch couldn't be automated, with the exception of a couple of human figureheads. Even if the AGI isn't good enough to do AI research, or the company doesn't trust it to do that, there are other methods for the company to grow. For instance, it could set up fully automated mining operations and factories in the corrupted country.

Comment by factorialcode on AGI will drastically increase economies of scale · 2019-07-05T21:48:50.443Z · score: 3 (2 votes) · LW · GW

I suppose another way this could happen is that the company could set up a branch in a much poorer and easily corrupted nation, since it's not constrained by people, it could build up a very large amount of power in a place that's beyond the reach of a superpower's anti-trust institutions.

Comment by factorialcode on Contest: $1,000 for good questions to ask to an Oracle AI · 2019-07-04T23:12:42.471Z · score: 1 (1 votes) · LW · GW

Submission: Counterfactual Oracle:

Use the oracle to compress data according to the MDL Principle. Specifically, give the oracle a string and ask it to produce a program that, when run, outputs the original string. The reward to the oracle is large and negative if the program does not reproduce the string when run, or inversely proportional to the length of the program if it does. The oracle receives a reward after the program runs or fails to terminate in a sufficient amount of time.

Submission: Low Bandwidth Oracle:

Have the oracle predict the price of a commodity / security / sports bet at some point in the future from a list of plausible prices. Ideally, the oracle would spit out a probability distribution which can be scored using a proper scoring rule, but just predicting the nearest most likely price should also work. Either way, the length of the episode is the time until the prediction can be verified. From there, it shouldn't be too difficult to use those predictions to make money.

More generally, I suppose we can use the counterfactual oracle to solve any optimisation or decision problem that can be evaluated with a computer, such as protein folding, SAT problems, or formally checked maths proofs.

Comment by factorialcode on 2014 Less Wrong Census/Survey · 2014-10-26T19:16:14.088Z · score: 38 (38 votes) · LW · GW

Took the survey!

Comment by factorialcode on Roles are Martial Arts for Agency · 2014-08-09T05:31:46.454Z · score: 4 (6 votes) · LW · GW

You might want to rot13 that last bit there for anyone who plans to see the movie.