I think the only cool LLM application I'd like to see is a thing where you type in a sentence and it tells you, based on the predicted perplexity of that sentence and a fermi estimate, how many times it might've been spoken in the past

I think it's really cool how we can accurately measure the information content of a sentence given surrounding content. everything else fucking sucks

there's so many questions in NLP that a highly accurate statistical model could help answer but it feels like everyone's lost interest in that

can we accurately decompose a sentence into its semantic information and its grammatical information? "six comes before seven" and "six precedes seven" have the same semantic content but make different grammatical choices. so some number of bits in their perplexity must be taken up by the grammar choice and not the actual semantics. can we quantify that?

when LLM-based "assistants" create responses they sample from the LLM's distribution semi-randomly, sharpening the distribution so it picks more likely options more. because it's sampling randomly, it's both making arbitrary decisions about grammatical structure (which I doubt many care about so much) but *also* arbitrary decisions about the semantic content. it seems to me like it would be useful to quantify how much of the semantic content of a response us up to chance? but nobody seems to care? what the fuck?