AI Glossary: LLMs and Text Generation

_INTRODUCTION

Large language models have become central to business AI strategy, powering applications from customer support automation to content generation and creative problem-solving. However, organizations must understand the technical parameters governing how these models work to evaluate their capabilities, predict outcomes, and use them effectively. This glossary explains nine fundamental concepts that shape how language models generate text and determine their suitability for specific business applications.

Token

A token is the smallest unit of text that an AI model processes; it may be a character, word, or subword depending on the model's design. For decision-makers, understanding tokens is essential because token count directly impacts processing speed and cost; longer documents require more tokens and thus more computational resources. Organizations should recognize that token limitations constrain how much text a model can process in a single request, affecting feasibility for document analysis and content summarization tasks.

Context Window

The context window is the maximum length of text a model can consider when generating a response, measured in tokens. This parameter directly determines a model's ability to understand long documents, maintain conversation continuity, and reference earlier information. For businesses, larger context windows enable more sophisticated applications like multi-page contract analysis and extended customer conversations; this capability has become a critical differentiator between competing language model solutions.

Completion

A completion is the text that a language model generates in response to a user prompt or instruction. Understanding completions as the model's "answer" or "output" helps organizations set realistic quality expectations and understand how model outputs require human review for critical applications. The quality of completions depends on prompt clarity, model capability, and generation parameters; organizations must invest in prompt engineering to maximize output quality.

Temperature

Temperature is a parameter controlling the randomness or predictability of model outputs; higher temperatures produce more creative and varied responses while lower temperatures yield more consistent and focused results. For business applications, this parameter enables critical tuning for different use cases: low temperatures for precise tasks like data extraction, higher temperatures for creative work like brainstorming. Understanding temperature settings is fundamental for optimizing model behavior for specific business objectives.

Top-p

Top-p (nucleus sampling) is an alternative parameter that controls output diversity by limiting the model to choose from the most probable next words that collectively account for a specified percentage of probability. This approach often produces more coherent and contextually appropriate outputs than temperature alone. Organizations should recognize top-p as a sophisticated control mechanism enabling fine-tuned output quality; combining top-p with temperature settings provides precise control over model behavior.

Max Tokens

Max tokens is the maximum length constraint on model-generated responses, limiting how long a completion can be. This parameter is critical for controlling costs, as longer generations consume more tokens and compute resources, and for preventing models from generating excessively verbose outputs. Organizations must balance business requirements for thorough responses against budget constraints by strategically setting max token limits.

Stop Sequence

A stop sequence is a text string that signals a model to halt generation upon encountering it, useful for ensuring model outputs follow specific formats or end at appropriate boundaries. For structured applications like API response generation or formatted document creation, stop sequences enable reliable output control. Organizations can use stop sequences to enforce compliance with required response structures without requiring post-generation processing.

Latency

Latency is the time required for a model to generate a response from receiving a prompt, typically measured in milliseconds or seconds. For customer-facing applications, latency directly impacts user experience; real-time applications require sub-second latency while background processing tasks tolerate higher latency. Organizations must evaluate model providers' latency guarantees alongside accuracy requirements when selecting AI solutions for time-sensitive applications.

Sampling

Sampling refers to the model's process of selecting the next token based on probability distributions during text generation. Different sampling strategies (including temperature, top-p, and top-k) determine whether the model chooses the most likely token or explores alternatives. Understanding sampling helps organizations recognize that language model outputs are fundamentally probabilistic rather than deterministic; this stochasticity explains why identical prompts may produce slightly different responses.