LLMs are programmed to provide an answer (cannot remain silent when prompted) and cannot say: ‘Don’t know.’ There is no silence, no space, no doubt.
Any system forced to perform fluency and/ or coherence, particularly when backed into a corner, will lead us a merry dance when there is no alternative.