Page 12 - AI Vol 1: Foundations of AI
P. 12
recognizing patterns in how words are used to 3. EMERGENT PROPERTIES IN LLMs
convey different meanings. For instance, the Emergent properties in LLMs refer to
model discerns whether the word “bank” refers behaviors, capabilities, or phenomena that
to a financial institution or a riverbank based arise unexpectedly as these models process and
on the surrounding context. Ultimately, LLMs synthesize vast amounts of data. These properties
function through probability, predicting the next are not explicitly programmed into the AI but
word in a sequence relying on patterns the model develop naturally from the complex interactions
learned through its training data. For example, if within the model’s architecture and its exposure
the input text is about finance, the word “bank” to diverse training data. Spanning beyond mere
would have a higher probability of following language generation, emergent properties can
words like “money” or “transaction” than “river” include aspects like creativity, logic, and task-
or “stream.” Through this training process, LLMs specific learning abilities that the model was not
not only learn common phrases and terminology directly trained to perform.
but also pick up on subtler aspects like idiomatic
expressions, cultural references, and even humor. A central debate in the study of emergent
As we will discuss in more detail in the next properties is whether these behaviors represent
part related to emergent properties, LLMs may “actual” understanding and reasoning or are
also learn the underlying meanings, concepts, sophisticated simulations thereof. For example,
and logic of their training data, enabling them one recent study suggested that emergent
to develop human-like capabilities, including properties might sometimes be illusions shaped
creativity and logic.
by how researchers interpret and evaluate
models, rather than actual abilities. This
distinction is crucial yet nuanced. If an LLM
consistently performs tasks with apparent logical
abilities, the practical distinction for end users
might be minimal. However, this does not negate
the importance of understanding the underlying
mechanisms, especially when considering the
ethical and safety implications of AI.
As examples, an early research paper demonstrated
that GPT-4 is able to understand mathematical
concepts to a certain degree, suggesting that it did
not simply “learn” the order of certain numerals
in relation to computation symbols (e.g., 4+4 =
8) but that it learned the underlying mathematical
concepts and was able to apply them to unique
questions not present in its training set. In
12 | VOLUME 1 FOUNDATIONS OF AI | LOZANOSMITH.COM
