How Beliefs Change What We See in Starlight

post by nixtaken · 2020-08-04T08:31:43.501Z · score: -1 (11 votes) · LW · GW · 27 comments

I was advised that the reason the articles I've posted here were not getting a good reception was that they were too long and they discussed epistemological concepts in physics in ways that seemed unfamiliar to this audience.

To back up and try to catch the people for whom this was the case, I've put together this brief introduction.

.................

When you believe in a theory that predicts the existence of gigantic, gravitational black holes that eat light, will you see evidence of these black holes all over the night sky? Will you construct pretty, artistic representations of the things in which you believe.

When you believe in a theory that predicts black hole collisions that release waves which will be detected on Earth as tiny changes in very sensitive measurement devices, will you see evidence of these collisions everywhere you look? Will you construct simulations of what you think these collisions would look like?

When the data from a single measurement device is so noisy that it shows you nothing, is it possible to combine data from multiple measurement devices to see something, or is there the risk that by picking and choosing which measurements to combine and look at, you might see something that isn’t really there? For example, if you took photographs every midnight of a dark, creepy hallway and none of them showed anything unusual, but when you superimposed all of the images together, all of the dust particles accumulated to form the shape of a ghost, can you believe that measurement?

There is always the risk that you might accidentally measure something other than what you intended, for example, instead of measuring a black hole shadow in the starry night sky, you might end up measuring light leaking around your earthly measurement device.

The purpose of the scientific method is to avoid fooling ourselves about what causes what we see. How can we avoid fooling ourselves about what we see in the stars? There are, after all, many ideas that are consistent with our theories of nature. Dragons and unicorns are perfectly consistent with the theory of evolution, yet they do not exist.

In a laboratory experiment, you have a closed system in which changing one variable should produce a predictable effect in another, but in astronomy, we can’t do these sorts of controlled experiments. At best, we can describe what we see. A cosmologist might see the results of a big bang explosion, a crystallographer might see a diffraction pattern, and a materials scientist might see a pattern of localized light in an inhomogeneous medium. When it comes to astronomy, there can never be one, definitively true description because we can never create a controlled experiment with the stars.

Some people build experiments to detect small changes in gravity and they attribute these changes to invisible things that are happening in the stars, but they can’t be sure that the stars are truly the cause of the changes. The changes could be caused by something much closer to Earth. They also can’t be sure that they haven’t made a mistake in how they have interpreted and filtered their data.

To avoid accidentally seeing something in noisy data that isn’t there, scientists will take multiple, independent measurements of something happening. If ten people all independently observe a dragon fighting a unicorn or a gravitational wave from colliding black holes in a distant galaxy, they were probably not all hallucinating. However, if ten people are all looking at a blurry, filtered, and enhanced image and they all agree that it is probably a picture of a dragon fighting a unicorn, they might be fooling themselves and looking at something much more mundane.

In the case of gravitational waves, back in the 1970s, hundreds of independent research groups constructed simple devices to measure them and they all compared their results. Each research group thought that it had measured gravitational waves, but when the results were combined, they all had to conclude that no one had been measuring gravitational waves. They had all been measuring different sources of noise.

Today, we have a new sort of gravitational wave detector that is very expensive and there are only three of them in existence. They all believe that they are measuring gravitational waves, but it is possible that they are all measuring different sources of noise because it is difficult to get enough results to compare with only three, independent measurement devices.

When evaluating a scientific result, it is important to remember that raw measurements should always be believed, but the interpretation we give to those measurements should always be treated skeptically because you might be measuring something you hadn’t intended to measure. It is also important to be wary of those who construct a result by combining biased, filtered, or calibrated measurements.

In conclusion, it is a good idea to be wary of scientists’ conclusions but to trust in the raw data. The conclusions might be biased by beliefs, even if the result was ‘peer-reviewed’. Groups of scientists can be just as unaware of their blind-spots and biases as individual scientists.

The scientific method itself is the source of the power of science. This power is not contained within the peer-review system or within the community, especially when that community is motivated to see things that are not there.

27 comments

Comments sorted by top scores.

comment by Dustin · 2020-08-04T23:50:28.715Z · score: 32 (15 votes) · LW(p) · GW(p)

I don't think you're saying anything here that longtime community members do not understand. Most here have discussed the basic human biases you're describing ad nauseum. The pushback you've received is not because we do not understand the biases you're describing. The pushback you've received is sourced in disagreements that scientists are doing the things that your analogies imply they are doing.

In this post you're just reasserting the things that people have disagreed with you about. I recommend directly addressing the points that people have brought up rather than ignoring them and restating your analogies. A brief perusal of what people have commented on your posts seems to show remarkably little effort by you to address any particular feedback other than to hand wave it away.

This is particularly the case when most people's priors are that the person disagreeing with the scientific establishment is the one who has a very strong burden of proof.

comment by nixtaken · 2020-08-05T05:51:23.573Z · score: -2 (6 votes) · LW(p) · GW(p)

The voting system here makes it look like there is far more community consensus than there is. At first glance, it looks like 11 people upvoted your comment, but upon closer inspection, it was only 3, and it may have been fewer if anyone has a sock puppet. I am always suspicious of people who claim to speak for a community.

comment by Ben Pace (Benito) · 2020-08-05T08:02:02.150Z · score: 20 (10 votes) · LW(p) · GW(p)

Nixtaken, I think it would be great if you responded to the substance of Dustin's point rather than replied with meta-commentary about how many people agreed with Dustin. For example, you could give a brief summary of the counterargument you've received, and explain where you think it's mistaken. I would upvote that.

[Insofar as it's relevant, I'm one of the people who works full-time on building this site and supporting this community.]

comment by nixtaken · 2020-08-05T09:32:30.167Z · score: 3 (2 votes) · LW(p) · GW(p)

Okay, if you'd like, I can respond to the criticism I've received:

After my first two posts, starting with the frontpage story, The Ghost of Joseph Weber, the response was a series of gish gallops by gjm in which he argued that organizing random data according to a criteria called 'weirdness' was scientific. (It is not.)

He was supported by dustin and by the moderator who removed the 'scientific methods and philosophy' tag from the posts, even though that was clearly the topic of the posts.

Several posts later, after my Karma had plummeted to -87, I was told that readers here don't like what I write because it is too long and that I should make shorter posts that are more focused, rather than drawing from a range of specific examples.

So I did that in this post, but then I was told by dustin that I've written something too glaringly obvious yet clearly incorrect and controversial. Instead of using specific examples from published papers, I used simple examples from everyday life and I was criticized for that as well.

There seems to be no winning with a crowd that doesn't like to see their favorite team (science) painted in a less than glowing light. I think that some of the elders of your community might be needed to help teach these people that science isn't about team loyalty, it is about a method, and that is what I've been writing about.

Your community has demonstrated the very problem that I claim is corrupting the physics community.

comment by Dustin · 2020-08-05T22:56:30.643Z · score: 17 (9 votes) · LW(p) · GW(p)

o I did that in this post, but then I was told by dustin that I've written something too glaringly obvious yet clearly incorrect and controversial.

No, I'm not qualified to gauge whether you are clearly incorrect. I am qualified to comment on whether you're making a convincing argument. Your arguments are not convincing largely because you do not really engage with people who question you.

The Ghost of Joseph Weber, the response was a series of gish gallops by gjm in which he argued that organizing random data according to a criteria called 'weirdness' was scientific. (It is not.)

And this is the problem. You could, for example, have a good and through discussion with gjm about this specific point. But you won't, and I find it disappointing.

Look, here's the deal for me:

  1. Bringing up that human bias could be the cause of a scientific result is not sufficient nor necessary to negate that result...the bias is beside the point of whether they are right or not. You have to engage the results.
  2. Most people, no matter how smart, do not have the background, time, or energy to engage on specific points of the technical subjects you have raised in your series of posts. (Of note, this is why you would do better to focus on single, specific technical points rather than shotgunning a non-physics-expert audience with every single technical thing you think is wrong with advanced physics experiments.) (This is also why, to most observers you are the one who started out with a gish gallop.)
  3. These technical points are the only thing you have to hang your hat on.
  4. gjm, to all appearances, seems to actually have the background to engage you on these points.
  5. Instead of engaging on any point gjm raised, you basically just dismissed all of them out of hand.
  6. Because of this, to an outsider of the field, you are now the one who looks like the one who has succumbed to unknown-to-us biases.
  7. As far as any outsider can tell there are a lot of plausible explanations for your position, and only one of them has to do with you being right...and you lowered my priors in the "this person is right about all of this physics stuff" explanation for your posts by rejecting engagement with the main person trying to engage you on a technical level.
  8. gjm could be full of shit. I don't know, but I do know that it doesn't seem like he's full of shit. I do know that a few of the factual things he brought up that I do have the background to check on...like him saying you were misquoting others seemed spot on. Add on to that your refusal to engage, and you're obviously going to be in the position you're in now.
  9. You may very well be correct but you're doing us all a disservice by arguing your points poorly.
comment by nixtaken · 2020-08-06T07:33:01.577Z · score: -2 (5 votes) · LW(p) · GW(p)

I encourage people like gjm who want to make comments that are longer than the post itself to make their own posts. In https://www.lesswrong.com/posts/oAsHa6xYMTBJWJGX6/the-new-scientific-method, [LW · GW] he claimed that, like EHT, he could organize random data according to a 'weirdness criterion' and that the result was scientific. Such a seemingly magical result would surely be worth publishing far and wide, if true, but if all of his points rest on such a false premise, his whole argument must fall like a house of cards and it is pointless to go through each individual point. I don't like making unnecessary noise.

comment by Dustin · 2020-08-06T17:08:17.900Z · score: 9 (4 votes) · LW(p) · GW(p)

gjm gave specific definitions of what he meant by "weirdness". I've yet to see you seriously engage on what he meant using the principle of charity and trying to figure out why you two were so far apart on this issue. That would be great to read and an effective way of convincing other people of your righteousness!

This willingness to engage is the core of good content on this site. Newcomers often have a hard time adjusting to this not-normal way of discussing issues.

As has been your wont in these threads you almost immediately fall back to accusing whomever you're arguing with to being biased in some way and saying "nuh-uh".

comment by lsusr · 2020-08-05T15:29:11.115Z · score: 9 (6 votes) · LW(p) · GW(p)

…a series of gish gallops by gjm…

What you, nixtaken, wrote [LW · GW] was a Gish gallop. What gjm wrote [LW(p) · GW(p)] was a refutation. You have ignored gjm's comment [LW(p) · GW(p)] about how you misapply the word "Gish gallop". This shows "remarkably little effort by you to address any particular feedback other than to hand wave it away."

So I did that in this post, but then I was told by dustin [sic] that I've written something too glaringly obvious yet clearly incorrect and controversial. Instead of using specific examples from published papers, I used analogies and I was criticized for that as well.

You have not refuted Dustin. You have not even failed to refute Dustin. You misrepresented Dustin's comment and then did not even contradict your self-contradictory misrepresentation of Dustin's comment that your post is "glaringly obvious yet clearly incorrect". In this way, your writing continues to demonstrate "remarkably little effort by you to address any particular feedback other than to hand wave it away."

There seems to be no winning with a crowd that doesn't like to see their favorite team (science) painted in a less than glowing light. I think that some of the elders of your community might be needed to help teach these people that science isn't about team loyalty, it is about a method, and that is what I've been writing about. [emphasis mine]

Thank you for not making personal [LW(p) · GW(p)] attacks [LW(p) · GW(p)] this time. Your overly-derogatory tone continues to violate the social norms of this website.

comment by nixtaken · 2020-08-05T16:10:51.182Z · score: -3 (3 votes) · LW(p) · GW(p)

When someone makes several comments that are longer than the post itself, and when the reasoning is demonstrably fallacious (weirdness criterion!?), I think it is fair to call the comment a gish gallop when that is the most economical way to express what happened.

I like economical language and tongue in cheek humor and that is why I condensed what dustin wrote to [your post is glaringly obvious yet clearly incorrect]. If I rewrite it in a less condensed form, this is what it would look like:

["You just wrote a bunch of stuff that is obvious to all of us and it was too long. You went on and on ad nauseum about things that people do, but we don't agree that it is right because we (rational scientists) don't do that sort of thing."]

I'll now copy and paste what dustin actually wrote so that you can compare, contrast, and complain about discrepancies.

" don't think you're saying anything here that longtime community members do not understand. Most here have discussed the basic human biases you're describing ad nauseum. The pushback you've received is not because we do not understand the biases you're describing. The pushback you've received is sourced in disagreements that scientists are doing the things that your analogies imply they are doing. "

comment by Dustin · 2020-08-05T23:15:57.072Z · score: 12 (6 votes) · LW(p) · GW(p)

When someone makes several comments that are longer than the post itself, and when the reasoning is demonstrably fallacious (weirdness criterion!?), I think it is fair to call the comment a gish gallop when that is the most economical way to express what happened.

You could have engaged on whether this was "demonstrably fallacious". That would have been interesting to read and I would've upvoted a good comment of this sort.

Again, you are the one who seems to be arguing in bad faith. It is very frustrating because LW has a long history of criticizing the practice of science, and it'd be interesting to see another good discussion in that vein.

comment by lsusr · 2020-08-05T16:24:38.801Z · score: 8 (5 votes) · LW(p) · GW(p)

When someone makes several comments that are longer than the post itself, and when the reasoning is demonstrably fallacious (weirdness criterion!?), I think it is fair to call the comment a gish gallop when that is the most economical way to express what happened.

You still have not refuted gjm's comment [LW(p) · GW(p)], which remains childless. I am not referring to the comment you handwaved away with the term "Gish gallop". I am referring to the comment where gjm explains why you are Gish galloping and ze is not. Once again, this demonstrates "remarkably little effort by you to address any particular feedback other than to hand wave it away."

…we (rational scientists)…

You continue to misrepresent the statements of others. Dustin did not claim to be a scientist nor member of the scientific establishment.

You have not refuted Dustin.

You have not even contested my statement "You have not refuted Dustin." This corroborates Dustin's claim "A brief perusal of what people have commented on your posts seems to show remarkably little effort by you to address any particular feedback other than to hand wave it away."

comment by Dustin · 2020-08-06T16:47:10.825Z · score: 5 (3 votes) · LW(p) · GW(p)

Just to make it clear and explicit. I am not a scientist nor am I a member of the scientific establishment.

comment by nixtaken · 2020-08-05T19:12:21.099Z · score: -3 (4 votes) · LW(p) · GW(p)

Online rule number one: never engage a gish galloper and never engage a person who insists that you 'refute' someone.

I broke it when I replied to gjm and you, but I can learn from my mistakes. Please excuse me from further conversation.

comment by Dustin · 2020-08-06T16:53:02.356Z · score: 2 (2 votes) · LW(p) · GW(p)

When someone makes several comments that are longer than the post itself, and when the reasoning is demonstrably fallacious

By this criterion, your original post is a gish gallop since it also included demonstrably fallacious statements.

On the other hand, we could take the charitable reading and say "maybe I don't understand the point they're trying to make and we should discuss it".

comment by lsusr · 2020-08-05T16:30:59.116Z · score: 2 (1 votes) · LW(p) · GW(p)

Your comments would benefit from the use of hyperlinks and blockquotes. If WISIWYG is a problem you can enable the markdown editor from your "Edit Account" [? · GW] page.

comment by Ben Pace (Benito) · 2020-08-06T04:35:18.144Z · score: 6 (3 votes) · LW(p) · GW(p)

I've not taken the time to read your posts, but it looks like you're trying to make some arguments about the feasibility of certain physics research programmes relating to gravitational waves, and that sounds cool. Physics is a difficult but very important (and just plain fascinating) area, and I myself am pretty disappointed with a lot of modern academia – including the physicists, and the string theorists, and so on. We discuss physics regularly and in depth on the site (e.g. the quantum mechanics sequence [LW · GW], the physics tag [? · GW]) and I appreciate you joining in humanity's effort on this front by being another person who takes the time to write up and explain their opinions on the subject, it's a virtuous effort – and it's nice that you came to our corner of the internet to talk with us :)

Overall, you've been downvoted a bunch, and gjm, dustin and Isusr have been pretty critical of your arguments, and you talked in intercom with Habryka where he also took Gjm's side in a specific technical disagreement. For me, these in combination mean I'm not going to set aside the 45+ minutes it would take for me to read your posts to double check whether the critiques are accurate.

I don't expect the reception will suddenly improve if you just change a little thing, so overall I'd suggest you cease posting, and lurk a bunch for at least a couple of months to get a better sense of how to engage with the site. If you like! You're not obligated to keep talking with us, especially if you don't find it fruitful.

comment by nixtaken · 2020-08-06T06:59:48.007Z · score: 3 (2 votes) · LW(p) · GW(p)

Thank you for your very diplomatic response. I understand that disruptions must be carried out slowly and with caution. I'm also beginning to understand that from certain perspectives, the purpose of physics education is to keep certain types of people busy, so that they don't cause any trouble. I find it unfortunate that many non-troublesome people get trapped in this net alongside the people who need more institutional support. This was a theme in a novel I wrote and I think that literature is a great way to help unbalanced people begin to see more sides of the issues that concern them.

After having already posted 6 articles here laying out the problem of black hole astronomy in both technical and colloquial terms, I did have two more articles that put the material into a larger context.

https://kirstenhacker.wordpress.com/2019/08/27/neutron-nonsense/

https://kirstenhacker.wordpress.com/2019/08/14/newtons-bad-apples/

But they have a more energetic tone and perhaps now is not the right time for that. Sometimes ideas need time to sink in.

Thank you for the suggestion about the quantum and physics forums. I might be able to contribute there in a less controversial manner.

comment by Ben Pace (Benito) · 2020-08-06T07:21:11.036Z · score: 3 (2 votes) · LW(p) · GW(p)

Thanks for the nice reply :)

I somewhat share your perspective that parts of physics academia are 'to keep certain types of people busy'. I think I agree, though I might put it in different words. Here's a transcript of a podcast from Eric Weinstein and Peter Thiel [LW · GW] that talks about some of these things, that I think is a valuable perspective. I do expect that further major advances in physics will probably come from outside of physics academia and education, inside of which things seem kind of stuck and unproductive.

Goodbye, and good luck with your future.

comment by Dustin · 2020-08-06T16:56:52.188Z · score: 11 (5 votes) · LW(p) · GW(p)

All in all, I find myself really disheartened by this whole saga since, 1) I find it, in the abstract, plausible that there are areas of modern science that have went down the wrong road because the practitioners have misled themselves, 2) some of the best content for me on LW over the many years has been of the type that highlights such deficiencies, and 3) I can see no progress being made on resolving our disagreements here.

As such, I'm not sure how much more value we can get out continuing these discussions. That really makes me sad since being willing to continually engage until disagreements are resolved is something I often enjoy.

comment by lsusr · 2020-08-06T18:39:02.841Z · score: 5 (2 votes) · LW(p) · GW(p)

I myself also tend to enjoy posts about science going down the wrong road. If you are comfortable sharing and it is is not too inconvenient to you, I am curious which specific posts you have enjoyed on this topic.

comment by Dustin · 2020-08-06T19:51:10.632Z · score: 7 (3 votes) · LW(p) · GW(p)

I know the vagueness of this is going to be irritating, and I sincerely apologize up front. I'm not a very "hygienic" reader...aka, I don't do a good job of physically or mentally organizing the information I've consumed to easily reference it in the future.

I can't actually think of any exact posts or comments, but when I ask myself "what do I like about LW?", one of the answers I give myself is something along the lines of "not willing to just accept science or scientific conventional wisdom at face value". (It's also possible that the impression I've built over the past 10+ years is just confused...probably stemming from the aforementioned bad information hygiene.)

Eliezer posted at least once on something at least tangentially related...about how science can't save you or something like that. There's been posts or comment threads about vitamins and I think other health-related "stuff". Over the years, Scott Alexander has written bucking-the-science-establishment-on-X posts as well.

As I give it more thought, I also think of posts that were written from the standpoint where the poster was seemingly prepared to accept that science was wrong or even thought ahead of time that science was wrong, but after investigation found out that, yep, science was probably right. IIRC, the vitamins post I mentioned above was in that vein.

comment by gjm · 2020-08-05T07:53:48.443Z · score: 7 (5 votes) · LW(p) · GW(p)

You write that

In the case of gravitational waves, back in the 1970s, hundreds of independent research groups constructed simple devices to measure them and they all compared their results. Each research group thought that it had measured gravitational waves, but when the results were combined, they all had to conclude that no one had been measuring gravitational waves. They had all been measuring different sources of noise.

So far as I can tell, this is not in fact true. E.g., this article about Joseph Weber says that Weber claimed to have measured gravitational waves, lots of other labs tried to replicate his results, and they all said they hadn't been able to. So does this one. The latter cites this paper in Physical Review Letters in which one pair of collaborators say exactly that. Here's another article saying the same. I haven't found anything that claims that everyone thought they had found gravitational waves, or that combining everyone's results made it plain that that wasn't so.

(I also remark that the story you tell in this paragraph is awfully similar to the one you want to debunk. Lots of scientific observations so noisy that you can't get anything reliable from any of them, but when you put them together you get something usable. This is exactly what you poke fun at at, e.g., the start of your "Weber's ghost" post. Surely you can't consider that approach valid when it says "no gravitational waves here" but invalid when it says "gravitational waves here"...)

comment by nixtaken · 2020-08-05T12:02:53.152Z · score: 3 (2 votes) · LW(p) · GW(p)

In response to what you wrote in parentheses:

If you combine noisy data and get a signal, you might be fooling yourself if you only select the data that gives you the signal.

If you combine noisy data and see randomness, you are probably not missing out on finding some secret signal that is hiding in the noise.

If you manipulate your data to see a secret signal hiding in noise, you are probably fooling yourself.

There isn't anything tricky here. You seem to see a trick when the logic is reversed. This isn't a trick. This is a natural consequence of entropy and the flow of space and time.

Smart people tend to make errors when they execute logical shortcuts. It seems like it makes sense to invert a concept until you think about the details.

comment by nixtaken · 2020-08-05T11:51:52.435Z · score: -3 (3 votes) · LW(p) · GW(p)

Weber says, "I have measured a cosmic gravitational wave with this device. The source of the signal is not of this world."

A hundred other researchers say, "We have used the same device to measure gravitational waves and found that it is too sensitive to local sources of noise (gravitational waves of undetermined origin) to be able to conclude that anything it measures is cosmic in origin."

Weber and the others were all measuring gravitational waves, but they disagreed about the origin of the waves. Some of the people thought that they would measure gravitational waves of cosmic origins and others thought that they wouldn't, but that wasn't relevant to their experiment, it was just their pre-existing bias, something that should not affect a well-designed experiment. Because an experiment is only well designed when it is conducted in many ways by many groups, when more people compared results, the more they would be able to see whether or not the statistics were random, which they were. That is the nature of the scientific method and my words reflect that, even though you seem unable to extract that context from the articles you read. Everything you read must be read within its proper context and experience with experimental research gives a reader context with more depth. As in, as you age, you get better at reading between the lines.

Without knowing the context you are carrying to this forum, it seems to me that you are deliberately misunderstanding what I wrote and that you are exploiting a lay-reader's confusion about a very narrow definition of the term 'gravitational waves'.

Weber's wife was Virginia Trimble and he lost NSF support after publishing his measurements. She would be the person to ask for the inside details. The narrative I described was passed to me in a classroom and it is consistent with the articles you linked to.

comment by lsusr · 2020-08-05T15:53:38.758Z · score: 4 (2 votes) · LW(p) · GW(p)

In the case of gravitational waves, back in the 1970s, hundreds of independent research groups constructed simple devices to measure them and they all compared their results. Each research group thought that it had measured gravitational waves, but when the results were combined, they all had to conclude that no one had been measuring gravitational waves. They had all been measuring different sources of noise.

A hundred other researchers say, "We have used the same device to measure gravitational waves and found that it is too sensitive to local sources of noise to be able to conclude that anything it measures is cosmic in origin."

Please provide a citation for this claim.

comment by nixtaken · 2020-08-05T19:03:08.279Z · score: -2 (3 votes) · LW(p) · GW(p)

Let me dig up that list of a hundred US and Russian papers from the 1970s...

I wonder why the physics community never published a widely circulated paper about that time that a large fraction of the community did something really dumb. Maybe its memetic immune system acted up - as it has in response to my articles.

The story of Weber and the gravitational wave debacle of the 1970s is the sort of lore that gets passed down within physics departments, but it is described in more oblique terms within the pages of Nature and Science. You have to be a party to the oral tradition of physics and be able to read between the lines to be able to understand and pass the stories on. I am giving this story to you. You are welcome. Analyze it for self-consistency with other things you know and do what you will with it. Write your thesis on it, for goodness sake.

comment by lsusr · 2020-08-05T19:16:38.060Z · score: 6 (5 votes) · LW(p) · GW(p)

That is not a citation.

Let me dig up that list of a hundred US and Russian papers from the 1970s...

Please do. A single citation pointing to that list ought to suffice.