What's the subject?

What are the different engagement metrics you're planning on using?

Have you considered doing something like a Q&A?

I think it's worth taking a look [LW · GW] at what's out there:

• SpanBERT
  • Uses random spans to do masked pre-training
  • Seems to indicate that using longer spans is essentially difficult
• Distillation of BERT Models
  • BERT embeddings are hierarchical

Markov and general next-token generators work well when conditioned with text. While some models, such as Bert, are able to predict masked tokens I'm not aware of models that are able to generate the most likely sentence that would sit between a given start/end prompt.

It's worth working in the Markov setting to get a grounding for what we're looking for. The core of Markov model is the transition matrix $P_{ij}$ which tells us the conditional likelihood of the token $j$ following immediately after the token $i$. The rules of conditional probability allow us to write,

$$p(k|j,i)=\frac{p(j,k|i)}{p\left(j|i\right)}=\frac{p\left(j|k\right)p\left(k|i\right)}{p\left(j|i\right)}$$

This gives us the probability of a token $k$ occurring immediately between the start/end prompts. In general we're interested in what happens if we 'travel' from the starting token $i$ to the ending token $j$ over $T$ time steps. Say we want to see the distribution of tokens at time step $t<T$. Then we'd write,

$$p^t(k|j,i)=\frac{p^{T-t}\left(j|k\right)p^t\left(k|i\right)}{p^T\left(j|i\right)}=\frac{\left(e_j{P}^{T-t}{e}_k\right)\left(e_k{P}^t{e}_i\right)}{e_j{P}^T{e}_i}$$

This shows us that we can break up the conditional generation process into a calculation over transition probabilities. We could write this out for an arbitrary sequence of separated words. From this perspective we'd be training a model to perform a regression over the words being generated. This is the sense in which we already use outlines to effectively create regression data-sets to model arguments.

What would be ideal is to find a way to generalize this to a non-Markovian, preferably deep-learning, setting. This is where I'm stuck at the moment. I'd want to understand where the SOTA is on this. The only options that immediately come to mind seem to be tree-search over tokens or RL. From the regression point of view, it seems like you'd want to try fitting the 'training data' such that the likelihood for the result is as high as possible.

I don't know if this is already known, but you might be interested in the fact that you can currently use start prompts for GPT-2.