GraphQL tutorial for LessWrong and Effective Altruism Forum

post by riceissa · 2018-12-08T19:51:59.514Z · LW · GW · 5 comments

Contents

  General steps for writing a query
  Examples
  Tips
  Acknowledgments
None
5 comments

This post is a tutorial on using GraphQL to query for information about LessWrong and the Effective Altruism Forum. It's mostly intended for people who have wanted to explore LW/EA Forum data but have found GraphQL intimidating (this was the case for myself until several weeks ago).

General steps for writing a query

(This section will make more sense if you have seen some example queries; see next section.)

For the queries that I know how to do, here is the general outline of steps:

  1. Go to https://www.lesswrong.com/graphiql [? · GW] or https://forum.effectivealtruism.org/graphiql [? · GW] depending on which forum you want to query data for.

  2. Figure out what the output type should be (e.g. comments, comment, posts, post).

  3. Type {output_type(input)} into GraphiQL and hover over input.

    Here is what it looks like for the comment output type:

    Here is what it looks like for the comments output type:

  4. Click on the type that appears after input (e.g. MultiCommentInput, SingleCommentInput). A column on the right should appear (if it was not there already). Depending on the fields listed in that column, there will now be two ways to proceed. (Generally, it seems like singular output types (e.g. comment) will have selector and plural output types (e.g. comments) will have terms.)

    Here is what it looks like for the comment output type. In the image, I have already clicked on SingleCommentInput so you can see selector under the documentation (rightmost) column.

    Here is what it looks like for the comments output type. Again, in this image, I have already clicked on MultiCommentInput so you can see terms under the documentation (rightmost) column.

    In the fields listed, if there is selector (e.g. for comment):

    • Click on the selector type (e.g. CommentSelectorUniqueInput). Use one of the fields (e.g. _id) to pick out the specific item you want.

      Here is what you should click on:

      What it looks like after you have clicked:

    If there is terms (e.g. comments):

    • Go to the collections directory in the LessWrong 2.0 codebase, and find the views.js file for your output type. For example, if your output type is comments, then the corresponding views.js file is located at collections/comments/views.js.

    • Look through the various "views" in the views.js file to see if there is a relevant view. (There is also a default view if you don't select any view.) The main things to pay attention to are the selector block (which controls how the results will be filtered) and the options block (which mainly controls how the results are sorted).

    • Pass in parameters for that view using keys in the terms block.

  5. Start a results block, and select the fields you want to see for this output type. (If you don't select any fields, it will default to all fields, so you can do that once and delete the fields you don't need.)

Examples

I've built a sample interface for both LessWrong and EA Forum that allows an easy way to access the queries used to generate pages:

By passing format=queries in the URL to any page, you can view the GraphQL queries that were made to generate that page. Rather than showing many examples in this post, I will just show one example in this post, and let you explore the reader.

As an example, consider the page https://eaforum.issarice.com/?view=top. Clicking on "Queries" at the top of the page takes you to the page https://eaforum.issarice.com/?view=top&offset=0&before=&after=&format=queries Here you will see the following:

    {
      posts(input: {
        terms: {
          view: "top"
          limit: 50
          meta: null  # this seems to get both meta and non-meta posts



        }
      }) {
        results {
          _id
          title
          slug
          pageUrl
          postedAt
          baseScore
          voteCount
          commentsCount
          meta
          question
          url
          user {
            username
            slug
          }
        }
      }
    }

Run this query



    {
      comments(input: {
        terms: {
          view: "recentComments"
          limit: 10
        }
      }) {
        results {
          _id
          post {
            _id
            title
            slug
          }
          user {
            _id
            slug
          }
          plaintextExcerpt
          htmlHighlight
          postId
          pageUrl
        }
      }
    }

Run this query

Clicking on "Run this query" (not linked in this tutorial, but linked in the actual page) below each query will take you to the GraphiQL page with the query preloaded. There, you can click on the "Execute Query" button (which looks like a play button) to actually run the query and see the result.

I should note that my reader implementation is optimized for my own (probably unusual) consumption and learning. For article-reading and commenting purposes (i.e. not for learning how to use GraphQL), most users will probably prefer to use the official versions of the forums or the GreaterWrong counterparts.

Tips

Acknowledgments

Thanks to:

5 comments

Comments sorted by top scores.

comment by namespace (ingres) · 2018-12-09T06:34:05.702Z · LW(p) · GW(p)

The official LessWrong 2 server is pretty heavy, so running it locally might be a problem for some people.

Whistling Lobsters 2.0 uses a clone of the LW 2 API called Accordius as its backend. Accordius is, with some minor differences, nearly an exact copy of the pre-October LW 2 API. It was developed with the goal that you could put the GreaterWrong software in front of it and it would function without changes. Unfortunately due to some implementation disagreements between Graphene and the reference GraphQL library in JavaScript, it's only about 95% compatible at the time of cloning.

Still, this thing will run on a potato (or more specifically, my years-old Intel atom based netbook) with GreaterWrong running on the same box as the front end. That makes it a pretty good option for anyone who's looking to understand GraphQL and the LW 2 API. This implementation does not take into account the changes made in the big API update in October. As a consequence, it may be more useful at this point for learning GraphQL than the LW 2 API specifically.

(Note to future readers: The GraphQL API is considered legacy for Accordius in the long term, so if you're reading this many months or even years from now, you may have to go back to the first alpha releases to get the functionality described here. Pre 1.0 perhaps.)

comment by NunoSempere (Radamantis) · 2021-01-06T11:14:47.555Z · LW(p) · GW(p)

I've come back to this occasionally, thanks. Here are two more snippets:

To get one post 

{
        post(
            input: {  
            selector: {
                _id: "Here goes the id"
            }      
            }) 
        {
            result {
            _id
            title
            slug
            pageUrl
            postedAt
            baseScore
            voteCount
            commentCount
            meta
            question
            url
            user {
                username
                slug
                karma
                maxPostCount
                commentCount
            }
            }
        }
}

or, as a JavaScript/node function:

let graphQLendpoint = 'https://forum.effectivealtruism.org/graphql' // or https://www.lesswrong.com/graphql. Note that this is not the same as the graph*i*ql visual interface talked about in the post. 

async function fetchPost(id){ 
  // note the async
  let response  = await fetch(graphQLendpoint, ({
    method: 'POST',
    headers: ({ 'Content-Type': 'application/json' }),
    body: JSON.stringify(({ query: `
       {
        post(
            input: {  
            selector: {
                _id: "${id}"
            }      
            }) 
        {
            result {
            _id
            title
            slug
            pageUrl
            postedAt
            baseScore
            voteCount
            commentCount
            meta
            question
            url
            user {
                username
                slug
                karma
                maxPostCount
                commentCount
            }
            }
        }
}`
})),
  }))
  .then(res => res.json())
  .then(res => res.data.post? res.data.post.result : undefined)  
  return response
}

 

To get a user

{
  user(input: {
    selector: {
      slug: "heregoestheslug"
    }
  }){
    result{
      username
      pageUrl
      karma
      maxPostCount
      commentCount
    }
  }
  
}

Or, as a JavaScript function

let graphQLendpoint = 'https://forum.effectivealtruism.org/graphql' // or https://www.lesswrong.com/graphql. Note that this is not the same as the graph*i*ql visual interface talked about in the post. 

async function fetchAuthor(slug){
  // note the async
  let response  = await fetch(graphQLendpoint, ({
    method: 'POST',
    headers: ({ 'Content-Type': 'application/json' }),
    body: JSON.stringify(({ query: `
       {
  user(input: {
    selector: {
      slug: "${slug}"
    }
  }){
    result{
      username
      pageUrl
      karma
      maxPostCount
      commentCount
    }
  }
  
}`
})),
  }))
  .then(res => res.json())
  .then(res => res.data.user? res.data.user.result : undefined)  
  return response
}
comment by Raemon · 2018-12-08T20:40:36.285Z · LW(p) · GW(p)

Thanks for writing this up! I've added it to the LW Open Source sequence [? · GW].

comment by sepremento · 2023-09-21T08:36:24.199Z · LW(p) · GW(p)

Can please someone post an example on how to find comments with negative agreement but positive approval?

comment by SimonM · 2022-12-28T22:22:33.713Z · LW(p) · GW(p)

Just in case anyone is struggling to find the relevant bits of the the codebase, my best guess is the link for the collections folder in github is now here.

You are looking in "views.ts" eg .../collections/comments/views.ts

The best thing to search for (I found) was ".addView(" and see what fits your requirements