Human feedback training may incentivize providing any answer -- even wrong ones.

When a research team led by Amrit Kirpalani, a medical educator at Western University in Ontario, Canada, evaluated ChatGPT's performance in diagnosing medical cases back in August 2024, one of the things that surprised them was the AI's propensity to give well-structured, eloquent but blatantly wrong answers.

Now, in a study recently published in Nature, a different group of researchers tried to explain why ChatGPT and other large language models tend to do this. "To speak confidently about things we do not know is a problem of humanity in a lot of ways. And large language models are imitations of humans," says Wout Schellaert, an AI researcher at the University of Valencia, Spain, and co-author of the paper.

Early large language models like GPT-3 had a hard time answering simple questions about geography or science. They even struggled with performing simple math such as "how much is 20 +183." But in most cases where they couldn't identify the correct answer, they did what an honest human being would do: They avoided answering the question.

The problem with the non-answers is that large language models were intended to be question-answering machines. For commercial companies like Open AI or Meta that were developing advanced LLMs, a question-answering machine that answered "I don't know" more than half the time was simply a bad product. So, they got busy solving this problem.

The first thing they did was scale the models up. "Scaling up refers to two aspects of model development. One is increasing the size of the training data set, usually a collection of text from websites and books. The other is increasing the number of language parameters," says Schellaert. When you think about an LLM as a neural network, the number of parameters can be compared to the number of synapses connecting its neurons. LLMs like GPT-3 used absurd amounts of text data, exceeding 45 terabytes, for training. The number of parameters used by GPT-3 was north of 175 billion.