LW Open Source – Getting Started

post by Raemon · 2018-05-06T23:28:25.642Z · score: 84 (21 votes) · LW · GW · 29 comments


  Getting Started
      Assumed Background Knowledge
      Setting up Locally
    What Contributions Are Helpful?
      Creating Issues
  Understanding the Codebase
  Going Forward

LessWrong 2.0 is open source, but so far we haven’t done a great job at making contributing a good experience. Here’s our first attempt at fixing this, which covers the basics of "how to contribute" and "high level overview of the site architecture."


Getting Started

Assumed Background Knowledge

Currently, this guide assumes that you know your way around a terminal and github, and have worked with javascript, html, sass/css, and ReactJS.

If you know at least some of those things and are excited to contribute, you can probably make decent headway (and if you run into specific obstacles, comment on them on this post or ping us on intercom, and we’ll see if we can update things to be more clear)

Setting up Locally

[Note: So far, everyone who's installed Lesswrong2 locally has been using a fairly modern MacBook. If you are running Windows or Linux you'll likely run into problems. (Trying anyway and responding here with your solutions will be helpful to get a better sense of how to install on a variety of machines, but it may be a bit more annoying to get started)]

LessWrong is a fork of VulcanJS's example-forum. We've had to make some changes to VulcanJS itself as well as their example-forum. We try to keep the changes to each codebase distinct, and have factored our web app into two repos.

I recommend starting by creating a folder to store both of them:

mkdir lesswrongSuite
cd lesswrongSuite

Clone the two repos:

git clone https://github.com/LessWrong2/Lesswrong2

git clone https://github.com/LessWrong2/Vulcan.git

You'll mostly be working in Lesswrong2.

Install node dependencies in each of the repo folders:

cd Lesswrong2 
npm install
cd ../Vulcan
npm install

Start the development server:

cd ../Lesswrong2 
npm start

If it is NOT working, there is most likely some issues with your npm install process. If you are terminal-savvy you can attempt to resolve that yourself based on error messages. If you'd like help, you can ping the LessWrong team either by creating a github issue or pinging us on intercom on LessWrong itself. You may find some help in the README for Vulcan-Starter

[Further note: If the install process results in errors, check your version of node against the one in the .nvmrc file. If individual files failed to install, try installing them individually (i.e. try npm install underscore-1.8.3 if the underscore package failed to install). That should likely result in easier-to-parse error messages that you can then google and attempt to debug]

You’ll then have a local copy of lesswrong2 running on your computer. (accessible on http://localhost:3000/)

It will start out with an empty database. (This means that some of the hardcoded links on the frontpage, such as Eliezer’s Sequences or the Codex, will not work). You can create users via the normal sign up process (entering a fake email is fine). The first user you’ll create will be an admin, so you’ll probably want to create at least two users to check how the site looks for non-admins.

You can poke around the codebase. Eventually, to make serious contributions you’ll want to read about our site architecture.

What Contributions Are Helpful?

The most reliably helpful thing would be to tackle outstanding issues that have been tagged on github.

In particular, you can filter them by the tag “good first issue.” (Some of these might require some explanation, but I expect I can explain them fairly easily to a new contributor)

Creating Issues

You can create a new issue. If so, please leave it untagged for the time being (so that admins can quickly look for untagged issues, and sort through them)

BugsIf you run into a bug, the most helpful thing to do is to search for related keywords in the issue tracker. If you can’t find anything relevant, create a new issue. Try to provide as specific information as possible (your browser, exact links to the post or page where you had the issue, information on how to replicate the bug if you can)

Feature Requests – Feature requests will often need to undergo some discussion to get refined into something executable, and may sometimes need to be split into sub-features.

Features tend to cluster into either “things that are pretty straightforward and we almost certainly want to do” and “things that we have weird conceptual philosophical opinions about that may sometimes be hard to explain succinctly.” (An upcoming post will go into some of this).

After you’ve posted a feature, an admin will tag it (If it’s been a week and we haven’t done so, feel free to bug us about it. We’re still adapting to the role of “open source facilitators” so we may drup things a bit)

Understanding the Codebase

LessWrong 2 is built on VulcanJS, a generic framework for online forums. VulcanJS is in turn built on technologies including:

Eventually, it’ll be helpful to have a good understanding of each of those technologies (both to develop new features and fix many kinds of bugs). But for now, the most useful things to know are:

A Vulcan project is divided into packages. It comes with several base packages that provide core functionality (users, voting, routing, etc).

The two packages you’ll most likely need to pay attention to are:

Going Forward

There's a lot more worth saying – I plan to write a post and/or github documentation that go over all the most important chunks of the codebase to help new developers get started. That's a fairly big project. But meanwhile it seemed better to get this post up and see what questions people actually had.

I also plan to write a bit more about some our underlying site philosophy, which will be important for people who want to suggest new features or make front-end changes.

Copying the tl;dr one more time:


Comments sorted by top scores.

comment by Raemon · 2018-05-07T20:13:36.990Z · score: 11 (2 votes) · LW(p) · GW(p)

If you're at least vaguely interested in this (even if it feels a bit intimidating, or you don't want to commit to anything), could you reply here with a brief note about your level of interest, and anything in particular that's acting as a barrier-to-entry?

I have the sense it may be worth investing a lot of effort into making the open source system here good, but wanted some sense of how many people might be willing to help.

comment by gjm · 2018-05-07T22:06:38.104Z · score: 13 (3 votes) · LW(p) · GW(p)

I am at least vaguely interested. Possible barriers to entry in my case are (1) having other higher-priority calls on my time, (2) the machine I spend most time on being a Windows box rather than Linux or Mac (I have literally no idea whether that is actually an obstacle; I know it was for old-LW), (3) general disorganization and shortage of Round Tuits, and (4) unfamiliarity with some of the specific technologies used.

Of these, #1 and #3 probably aren't going to go away, #2 may be a non-issue (perhaps someone can comment?), and #4 serves mostly to make #1 and #3 more severe (because it makes getting started feel like a bigger investment of time).

The most effective way (of those actually available to LW developers) to make me[1] more likely to contribute is probably to provide high-level documentation of the system -- the sort of thing in the OP here, in fact -- so that getting into it is as painless-in-expectation as possible.

[1] Not that making me in particular more likely to contribute should be anyone's goal (unless maybe my own), but maybe there are others who feel the same way.

comment by gjm · 2018-05-08T14:53:03.876Z · score: 11 (2 votes) · LW(p) · GW(p)

On #2, I just tried the Obvious Thing of running the commands listed in the OP on my Windows machine (ordinary Windows command prompt, not MinGW or WSL or anything of the sort). Here is what happened.

The npm install command failed. What I believe to be the first informative line of the error message said Error: Can't find Python executable "python", you can set the PYTHON env variable. Indeed I don't have a PYTHON environment variable set. A bit of googling suggests that on systems where it is set it ought to point to the Python executable. So I said set PATH=C:\ProgramData\Anaconda3\python.exe which is where my Python interpreter lives, and I got the same error except that it said Can't find Python executable "C:\ProgramData\Anaconda3\python.exe" which, since that file is definitely there, I took to mean that it failed to work in some way.

I tried doing the same from an "Anaconda prompt" (i.e., a command prompt with all environment variables set up suitably) and it failed in pretty much the same way except that it had ".EXE" instead of ".exe" at the end of the Python executable name it said it couldn't find.

Perhaps we need Python 2 rather than Python 3. So I installed Python 2.7.15 (vanilla install from python.org rather than via Anaconda), set PYTHON=C:\Python27\python.exe, and tried again. This time it apparently worked, though there were a bunch of warnings.

Then npm start failed completely, because it tries to run a bash script. Oh well, never mind.

Next try: launch a WSL bash prompt, cd /mnt/c/Users/... and try again from the top. My Ubuntu-inside-Windows doesn't have any version of Python installed by default, so sudo apt install python-minimal (which uses Python 2 rather than Python 3) before proceeding.

Now running npm install fails in a different and more immediate way: before it even tries to do anything, three error messages, two of which are ... not found and the last of which is Syntax error: word unexpected (expecting "in"). I think what's happening is that my pseudo-Linux shell is trying to run npm from my Windows installation of npm, and then getting confused by DOS-style line endings or something of the kind. (There is quite good evidence for this in the actual error messages.)

Next try: install nodejs and npm inside pseudo-Linux and try again. The result is the same, which doesn't make any sense to me.

So, current suggestion: step 0 is "Be running Linux or macOS" for now. May also need "make sure you have Python 2 installed"?

I'll continue trying to get it working later, but right now I'm at work and need to be doing other things :-).

comment by crybx · 2018-05-09T01:07:53.146Z · score: 10 (2 votes) · LW(p) · GW(p)

If you try again, I think you can avoid needing bash. See my comment here [LW(p) · GW(p)].

comment by Raemon · 2018-05-08T15:10:21.899Z · score: 6 (1 votes) · LW(p) · GW(p)

Very much appreciate both giving it a try and then putting a fair chunk of effort into debugging.

Elo tried installing yesterday on Linux and didn't actually get it up and running either, and upon reflection everyone I know who's gotten it running locally has been using a mac. Linux should hopefully be easier to debug than Windows.

Meanwhile I'll update this post with the "step 0". I think once we've gotten a couple people who had troublesome installs to work properly I'll do some more documentation about best-practices-for-install-troubleshooting.

comment by philh · 2018-05-09T19:08:21.451Z · score: 4 (1 votes) · LW(p) · GW(p)

It didn't work for me on linux either.

npm install gave out a lot of warnings about outdated packages, I don't know if they were talking about my preexisting environment or about something in the LW2 repo.

npm start took over half an hour before I went away, and when I came back many hours later it seemed to be stuck in a loop of constantly retrying something that was throwing an error.

I will also take a moment to grump that prestart_lesswrong.sh installs meteor, which I would rather have done myself (and which I wouldn't have done through curl | bash). Still, I understand that having an easy setup makes people more likely to contribute, so I won't grump too much. I would have appreciated a warning though, or a separate npm install_deps script or something.

comment by gjm · 2018-05-10T14:48:29.738Z · score: 5 (1 votes) · LW(p) · GW(p)

Continuing attempts on WSL under Windows 10. I killed the bash prompt I had, and started a new one. This now seems to find the correct version of npm in preference to my Windows one, and npm install no longer fails instantly.

Unfortunately, it does still fail. After downloading a bunch of things, I get a bunch of errors associated with something called chromedriver. The first error message line says ERR! Linux-4.4.0-17134-Microsoft. Perhaps this means something has tried to figure out what sort of system it's running on and doesn't recognize what WSL reports? Possibly-informative later lines of the error message say ERR! code ELIFECYCLE and ERR! errno ENOENT. I can't find anything in the node_modules directory relating to chromedriver; perhaps when something fails to install npm destroys the evidence? Scrolling back, I see that there's an earlier sign of trouble: a line of the form > chromedriver@2.38.3 install [path to chromedriver module] (note: that path no longer exists in my filesystem) followed by a line of the form > node install.js followed by a blank line and then sh: 1: node: not found. After that comes a line beginning with "Vulcan" and then a big tree diagram of modules, then some dependency-related warnings, and then the errors I mentioned above.

[EDITED to add:] It turns out that on Ubuntu (which is the particular flavour of WSL I'm using) the nodejs package doesn't provide a node executable because there's another Ubuntu package that wants to call its executable node. So you need to install nodejs-legacy which I think just makes there be a symlink. And with that, npm install completes.

Now, of course, npm start fails. Meteor gets installed, we get a proxy started, but then three lots of Unexpected mongo exit code 100. Restarting. and some guesses at what Mongo might be unhappy about. And, actually, one of them looks super-plausible: this is all happening in an ordinary-Windows directory (via /mnt/c/... on WSL) and "MongoDB does not support filesystems like NFS that do not allow file locking." So, time to put the whole thing inside WSL-land and see if it fares any better, I guess.

comment by Raemon · 2018-05-10T21:08:16.186Z · score: 6 (1 votes) · LW(p) · GW(p)

When the npm install option results in some errors, what I find most helpful is to try to install those things individually (where you can focus on just the errors relevant to a single thing)

So my next step if this were my machine would be:

1. look up the version of chromedriver it's trying to install (in the package.json file)

2. I can see that it's chromedrive@r^2.34.1

3. Try npm install chromedriver@^2.34.1

4. See what errors come up, do a google search for those errors and see if you find anything useful on stack overflow

You can resolve individual failed-to-download packages individually, and once you've done that for each package throwing errors, npm start should work.

comment by philh · 2018-05-08T10:16:34.691Z · score: 9 (2 votes) · LW(p) · GW(p)

I'm in a similar place except for #2.

Another thing that would make me more likely to contribute would be having an easy-to-setup test database to work with, instead of having to create users, articles etc. manually.

comment by Raemon · 2018-05-09T02:04:45.756Z · score: 12 (2 votes) · LW(p) · GW(p)

Good to know – we actually used to have the default Vulcan seed database, but it periodically caused issues with our testing environment. But I can add it back as a command that you have the option of running once, and maybe tweak the database parameters to be more LW-relevant (i.e. including an admin user, a sunshine, a trustLevel1 user, a curated post, etc)

comment by Elo · 2018-05-07T20:36:49.148Z · score: 11 (2 votes) · LW(p) · GW(p)

I am interested and going to try to set up. Not sure how much coding capacity I have or how much help I can be.

comment by Raemon · 2018-05-07T20:44:11.416Z · score: 6 (1 votes) · LW(p) · GW(p)

Thanks! I think "get it set up on my local machine" is a pretty achieavable thing for most people and a good first step to at least remove one trivial inconvenience.

comment by crybx · 2018-05-09T20:51:07.840Z · score: 4 (1 votes) · LW(p) · GW(p)

Now that I have a copy of the code running on my local machine, I was thinking of grabbing an issue to work on. (I can't promise any commitment level beyond one issue yet.) I'm trying to be thorough reading what docs there are, and I've come across the contributing guidelines which says to check out a roadmap (a Trello board) and join a Slack channel before working on anything. The Trello board doesn't make much sense to me and I'm not sure if either of these instructions are still important to follow, or if I really should just claim an unassigned issue with a 'good first issue' or 'help wanted' tag.

comment by Raemon · 2018-05-09T21:37:11.091Z · score: 6 (1 votes) · LW(p) · GW(p)

Ah, that's out of date. For now, consider this blogpost more authoritative than anything else until I fix some of the older docs (will try to do that soon).

Here are the 'good first issues' :


comment by Raemon · 2018-05-09T21:56:15.696Z · score: 6 (1 votes) · LW(p) · GW(p)

Also, feel free to ask any questions about any of the issues – I'm happy to spend a bunch of time helping people orient around the issues themselves and the codebase in general.

comment by Elo · 2018-05-09T21:09:34.803Z · score: 5 (1 votes) · LW(p) · GW(p)

As far as I know, go claim an issue.

comment by crybx · 2018-05-08T19:36:57.802Z · score: 10 (2 votes) · LW(p) · GW(p)

I use Windows and intend to try to get this running on my machine.

Also, fyi, the first link to the GitHub repo in the tl;dr doesn't point to the right place.

comment by crybx · 2018-05-09T01:04:17.075Z · score: 17 (4 votes) · LW(p) · GW(p)

I got further than gjm reported.

I needed to:

  • Install Node (That link goes to the exact version listed in .nvmrc)
  • Install Python 2.7.whatever since I only had Python 3 before this.
  • Install Visual C++ 2015 Tools for Windows Desktop - This was the weirdest one, but after this, npm install works without error.
  • Install meteor

I learned that npm start is unnecessary. It runs a bash script that

1. checks meteor is installed

2. creates the file settings.json if it doesn't exist by copying from sample_settings.json

3. runs command meteor --settings settings.json

If you have meteor installed and create settings.json yourself, you can skip npm start and just run

meteor --settings settings.json

I verified that I was able to successfully build and run the VulcanJS starter example by doing:

  • cd Vulcan-Starter code repo
  • run npm install
  • manually copy sample_settings.json to settings.json
  • run meteor --settings settings.json
  • I then had a website running at http://localhost:3000/

Additional info in the VulcanJS docs includes:

(A note for windows user: While running npm install you might get error regarding node-gyp and bcrypt package installation. This is because you need windows-build-tool for node-gyp installation which is required for bcrypt installation.)

This is what the C++ dev tools fixed. I don't know if something like npm install -g windows-build-tool would have fixed this or not. I didn't read this until afterwards.


Note that you can also start the app with:
meteor --settings sample_settings.json
All npm start does is run the above command, while also checking for the presence of settings.json first and creating it for you if missing.

But for LessWrong 2 I am stuck at an error that looks like https://pastebin.com/M1vqTMZd

I think I'm stumped for now. I can clearly get the Vulcan Starter project running, just not LessWrong 2.

comment by Raemon · 2018-05-09T01:45:34.991Z · score: 6 (1 votes) · LW(p) · GW(p)

Ah, there's actually a line missing from the sample-settings.json file, which I just added. If you git pull it should fix the "can't construct client" issue.

comment by crybx · 2018-05-09T01:52:00.034Z · score: 16 (3 votes) · LW(p) · GW(p)

Adding the missing line fixed it! I have Lesswrong2 running successfully on Windows at http://localhost:3000/ now.

comment by Raemon · 2018-05-09T02:03:25.297Z · score: 6 (1 votes) · LW(p) · GW(p)

Huge thanks for all your debugging work!

It's occurring to me we should probably have direct karma rewards for people who pitch in with the codebase. For now I just upvoted all your comments. :)

comment by crybx · 2018-05-09T01:18:51.744Z · score: 16 (3 votes) · LW(p) · GW(p)

This might be a good use case for someone to create a Docker image (or some other container) that has a development environment that just works for new users.

comment by jglamine · 2018-05-06T23:51:02.655Z · score: 7 (2 votes) · LW(p) · GW(p)

Good work on the guide! Are you planning on combining this with the readme or linking to it from there? There's a bit of redundancy between the two.

comment by Raemon · 2018-05-07T00:04:30.846Z · score: 8 (2 votes) · LW(p) · GW(p)

Yeah, my hope was for this to replace the readme file (which was sort of out of date, but figuring out exactly how to replace it was a bit more work than I had bandwidth for today and I wanted to get this thing moving)

comment by Raemon · 2018-05-10T21:04:06.759Z · score: 6 (1 votes) · LW(p) · GW(p)

FYI, this issue that just came up here is pretty much the easiest possible first-issue-to-handle, so anyone who's looking for a good get-their-feet-wet issue could check it out:


comment by roryokane · 2019-03-21T15:57:10.574Z · score: 5 (3 votes) · LW(p) · GW(p)

There’s a formatting error in this part of the post:

Clone the two repos:

git clone https://github.com/LessWrong2/Lesswrong2 git clone https://github.com/LessWrong2/Vulcan.git

It should be a code block so that the newline is preserved. That makes it easier to notice that you have to clone two repos:

Clone the two repos:

git clone https://github.com/LessWrong2/Lesswrong2
git clone https://github.com/LessWrong2/Vulcan.git

Also, I found this description confusing:

If you are creating a branch for an existing issue, use this naming schema: branchTitle[issueNumber]. For example, if addressing this issue, your branch might be defaultSettingsFix425.

The schema “branchTitle[issueNumber]” suggests that the branch title should be “defaultSettingsFix[425]” instead. It would be better to describe it as “[branchTitle][issueNumber]” or “branchTitle + issueNumber”, or just say “append the issue number to the branch name”.

comment by Raemon · 2019-03-21T20:05:16.872Z · score: 6 (3 votes) · LW(p) · GW(p)

Thanks. I've added a newline for the first one, and just deleted the entire paragraph about the branch naming because we never really stuck to it.

comment by crybx · 2018-05-10T22:53:09.780Z · score: 4 (1 votes) · LW(p) · GW(p)

please assign the issue to yourself in github so people know someone is working on it.

It doesn't look like users can assign issues to themselves without being invited to be a contributor.


comment by Raemon · 2018-05-10T23:20:28.388Z · score: 6 (1 votes) · LW(p) · GW(p)

Ah, whoops. I'm actually not sure if I can change people's contributor status or change this policy (habryka might have some additional admin powers that I don't have). But for now, if you'd like to start working on an issue, just comment on it, and then I'll assign it to you (or maybe we'll just make do without officially having assignees).