The two types of language model

There are two types of language model. Masked language models and autoregressive language models.

Masked language models are good at guessing missing items given the context around the missing words, so it would be good at guessing the blank in cases like this: “The hamster had a ___ in its cage”. Whereas autoregressive language models on the other hand predict the next token “The hamster had a wheel in its ___”. Type a sentence into the little widget to see how each language model type would handle predictions.

Interactive explanation

What can the model see?

Write a sentence, then pause to see it through either model.