[Site Feature] Link Previews

The basic idea behind the popups is that at the Markdown->HTML compile time, I do a lookup in a hashmap of (URL, (Title, Author, Date, DOI, Summary)), and if there is an entry, it gets inlined into the HTML as some quiet metadata; if there isn't, various scripts are called to try to generate metadata, and if nothing works, it falls back to a headless web browser taking a screenshot and saving it to a file. Then at runtime in the user browser, some JS checks every link for the metadata and if it exists, and the user mouses-over, it popups a form with the metadata filled in. If there is no metadata, it tries to fetch SHA1(URL) from a gwern.net folder under the assumption there will be the fallback screenshot.

There are a lot of fiddly details in how exactly you scrape from Arxiv or Pubmed Central or use the WP REST API, unfortunately. So many links must be handled by hand-writing definitions.

The relevant files in call order:

• https://www.gwern.net/hakyll.hs

• https://www.gwern.net/LinkMetadata.hs

• https://www.gwern.net/linkAbstract.R

• https://www.gwern.net/linkScreenshot.sh

• https://www.gwern.net/static/js/popups.js

• the generated/hand-written 'database' files:

  • https://www.gwern.net/static/metadata/auto.hs
  • https://www.gwern.net/static/metadata/custom.hs
  • https://www.gwern.net/static/metadata/preview.hs

Yeah, I was happy when I saw you/gwern were also doing this. We've looked a bit at the your implementation and may ping you about the details once we get to our own external links.

One problem with link coloring is that link coloring is already used by the browser as semantic annotation and has been for decades by every web browser I am aware of: specifically, whether a link has been visited before or is novel. When I look at your screenshot, I can't read it as 'off vs on site', I can only read it as 'ah, Raemon has not yet read about foreign Sequences'. It's too ingrained, and the colors are fighting >20 years of browser conditioning. Adding more colors to overload coloring doesn't help this, as that makes even more to learn (what would it be, dark green for 'unread on-site', light green for 'read on-site' etc?). It is also somewhat difficult to tune them, and more obscurely, you have issues with color-blindness (depending on the colors you pick, some <10% amount of readers will struggle or be entirely unable to perceive the difference).

Link icons, on the other hand, are additions, rather than overloads, used at least somewhat occasionally online already, can be understood by anyone who isn't blind (assuming grayscale like mine), and relatively self-explanatory (assuming good choice of logos).

I’m a fan of link icons, and not a fan of color coding.

I liked the little "LW".

Though why does it appear at the end of the link?

Since all links begin in the same place but end in different places there's an annoying micro-suspense before I reach the end of the link and figure out whether it's LW or not.

I've thought about things in that space but not that particular implementation, which is interesting. One problem is that an author might intend a different use of a given jargon term (esp. since some jargon is actually just "regular english words", such as "focusing.")

A couple other options are:

• The author receives a prompt to consider adding a link to a jargon word as they use it (which they can ignore)
• At the bottom of the post, there's a collection of jargon-links that the author can decide whether they match the author's use.

Both of those run the risk of being annoying for the writer.