The substrate

Is there any conceivable way of telling whether we're in a simulation

Yes, with high probability at least. We create lots of simulations of observers who have the same subjective beliefs as we do. Not that I'm in favor of this, but you did use the word conceivable.

Three tips for reasoning about the real world, for people who might be "living in a simulation" or otherwise inhabiting a systematic illusion:

(1) If: every world in some ensemble of possible worlds exists, and you know enough about this ensemble; then you can reason about the properties of the real world, by applying standard probabilistic inference to the set of all entities, occurring in this multiverse, which are subjectively identical to you. For example, if half of your subjective duplicates across the multiverse inhabit worlds made of superstrings, then the probability is 50% that your real world is made of superstrings.

This is both a no-brainer and arguably useless advice, since you just may not be in a position to know the full nature of the multiverse, or to calculate the demography of simulations and false realities throughout it. However, it is worth understanding that if we could know enough, then answering these questions really would be a matter of deduction and calculation.

(2) In the absence of a justified, top-down way of answering questions about the real world, you can at least aim for a systematic approach by collecting and generalizing hypotheses. Nancy has a small list above ("dreams, daydreams,..."); this could be expanded.

There are lists of proposed answers to the Fermi paradox (1, 2), perhaps it's time to start work on assembling comparably comprehensive lists of simulation scenarios.

(3) One might also wish to discreetly make a list of ways that making a list of simulation hypotheses could induce feedback from the laws or the lords of the matrix. There are simulation scenarios in which you are not supposed to know, or in which trying to learn the nature of the simulation has prearranged consequences. These could be called "simulation-fights-back" scenarios.

Edited to add: I should think that being a simulation is an existential risk.

Just being is an existential risk. Counterfactuals are much safer.

If we're part of a simulation, how likely is it that whatever it's running on is using the same sort of atoms we've discovered?

Even if we right now had the ability to run really good simulations we will probably run a larger variety of simulations with different types of physics with different building blocks than the building blocks of our reality (a crude example - Conway's game of life does not simulate atoms) so I'd reason that its more likely that the real physics of the real world are not very similar to our physics. And even if they are similar, our physics are more probably just an approximation of the real thing.

This is also the reason why I don't think we are very likely to be in an ancestral simulation - the intelligence(s) in the real universe that has/have the capability of running such an elaborate simulation of our world are likely to have ran an enormously large numbers of simulations of universes not like its/their own and the number of those simulations is probably way way way higher then the number of ancestral-type simulations that it/they will have run.

If anything the likeliest type of simulations to be data-mined (as in to be ran in the largest number of the possible varieties) which are in accord with whats going on in our world would be simulations of universe with intelligent life that leads to a singularity. In fact even ancestor simulations suggest that we are in a simulation that reaches a singularity (how else are they running a large number of such complex simulations). My reasoning for this is that data-mining of such simulations is likely done by a sort of superintelligence and it is reasonable to argue that a superintelligence will be searching for/find useful data from a post-singularity world and not so much data of what some stupid apes do.

So onwards to practical questions. Is there any conceivable way of telling whether we're in a simulation and if so, learning something about its nature? Is it worth trying to get out of the Big Box?

Every test can be faked by the lords of the matrix in a way that always gives the results that we will get in the 'real' universe, so there might not be a way to tell if we live in a simulation. However the more it seems that we will be able to run complex simulations with life that experiences qualia the more likely it is that there are a large number of such simulations per real universe which makes it more likely that we live in a simulation. If the evidence gets good enough we might one day know that the chances are only 10^100 to 1 for us to be in the real world (a made up number just to illustrate my point).

I agree that full-universe ancestor simulations are unlikely. Limited simulations focused on interesting historical events and people seem much more likely (alongside more exotic simulations, of course). Your probability that you're a part of such a simulation depends on the interestingness of people and events proximal to you. Robin Hanson wrote a paper on how to behave in such a simulation.

FWIW, I don't have a high probability of being in a simulation like that; I'm not close enough to anyone or anything historically interesting to be simulated in full, conscious detail. But someone like Robin Hanson, EY, etc., must have a much higher probability.