Chain-of-Thought
Also known as: CoT, chain of thought prompting, step-by-step reasoning, scratchpad reasoning
Chain-of-thought (CoT) prompting is a technique for improving the reasoning quality of large language models by asking them to generate intermediate reasoning steps before producing a final answer, rather than jumping directly to the answer.
When an LLM works through a problem step by step ('first I need to identify X, then calculate Y, therefore Z') it consistently produces more accurate answers on tasks requiring multi-step reasoning, mathematics, logic, and complex problem-solving. Chain-of-thought was introduced as a research finding in 2022 and quickly became a standard technique in LLM applications.
Chain-of-Thought, explained at three levels
At home
Chain-of-thought is about asking an AI to 'show its work' before giving an answer, just like a teacher might ask a student to show their working in a maths test. If you ask an AI 'If a train travels at 60 mph for 2.5 hours, how far does it go?', it might immediately answer '150 miles'. With chain-of-thought, you ask it to reason through: 'First I calculate: 60 mph × 2.5 hours = 150 miles. The answer is 150 miles.' The step-by-step reasoning makes it less likely to make careless errors.
At work
Chain-of-thought prompting is used in business AI applications to improve accuracy on tasks requiring logic, calculation, or multi-step reasoning. Customer service AI that needs to calculate whether a customer is eligible for a refund, code generation AI working through a complex algorithm, and contract analysis AI identifying which clauses apply to a specific situation all benefit from chain-of-thought reasoning. The technique is most useful for tasks with a clear correct answer that requires intermediate steps to reach.
At home
Zero-shot chain-of-thought works by simply adding 'Let's think step by step' to a prompt: no examples are needed. Few-shot chain-of-thought provides examples of the desired reasoning format: a worked example showing the problem, the step-by-step reasoning, and the correct answer, before presenting the actual question. Few-shot CoT generally produces better reasoning than zero-shot CoT for complex tasks, because the examples show the model what format and depth of reasoning is expected.
At work
Chain-of-thought has an important limitation: a longer, more elaborate reasoning chain is not always more accurate. LLMs can produce confident-looking step-by-step reasoning that leads to an incorrect conclusion: a phenomenon sometimes called 'reasoning hallucination'. A chain-of-thought output should not be trusted simply because it looks plausible: the intermediate steps should be verified for tasks where accuracy is critical. CoT improves average performance across a distribution of tasks; it does not guarantee correctness for any specific output.
At home
Modern reasoning models (OpenAI o1, Anthropic Claude with extended thinking) extend chain-of-thought into a first-class model capability by training models to generate extensive internal reasoning (often called a 'scratchpad' or 'thinking' process) before producing a response. These models perform significantly better on complex reasoning benchmarks than base models prompted with simple CoT, because the reasoning process itself is trained rather than just prompted. However, the extended thinking uses more tokens and incurs higher inference costs.
At work
From a practitioner standpoint, chain-of-thought prompting should be considered for any application requiring multi-step reasoning, and the reasoning chain should be surfaced to users or reviewers for tasks where verifiability matters. For high-stakes applications (medical diagnosis support, legal analysis, financial calculation), ask the AI to explain its reasoning and have a human verify the reasoning chain, not just the final answer. The reasoning chain also provides debugging information when outputs are wrong: identifying where the reasoning went astray is the first step to correcting the prompt or identifying the task's limitations.
Real-world examples
At home
Marcus was using an AI tool to help him calculate whether he met the income requirements for a housing benefit. When he asked directly, the AI gave him a number without explanation. He then added 'Please show your reasoning step by step' and the AI showed each calculation, allowing Marcus to spot that the AI had used the wrong income threshold for his region and correct the calculation himself.
At work
A legal analytics company added chain-of-thought prompting to their contract review tool, requiring the AI to explain which specific clause triggered each flagged issue and why it potentially violated the company's standard terms. Lawyers found the reasoning chains dramatically more useful than bare flags: they could review and accept or override the AI's reasoning rather than treating its outputs as a black box.
Common confusions
- Chain-of-thought does not make AI outputs reliable enough for unsupervised use in critical decisions. A convincing-looking reasoning chain can still reach an incorrect conclusion. CoT improves accuracy in aggregate but does not guarantee correctness per output. Human review of the reasoning chain remains necessary for high-stakes applications.
- The length of a reasoning chain is not a reliable indicator of its quality. A model can generate a very long, detailed-looking reasoning chain that contains fundamental errors in the first step that propagate through the rest of the reasoning. Evaluating chain-of-thought outputs requires checking the reasoning logic, not just the conclusion.
- Chain-of-thought prompting is not the same as extended thinking or reasoning models. Adding 'think step by step' to a prompt produces chain-of-thought in a standard model. Reasoning models (like o1) are specifically trained to generate extended internal reasoning before responding: the reasoning process is part of the model's trained behaviour, not just a prompting technique. The two produce similar outputs but through different mechanisms, and reasoning models typically perform better on complex tasks.
For analysts
In BA artefacts, chain-of-thought requirements appear when specifying AI features that perform multi-step analysis or decision support. Require reasoning chain output as a functional requirement for auditability: 'For each AI-generated recommendation, the system must produce a human-readable explanation of the reasoning steps that led to the recommendation, not just the recommendation itself.' This supports both user trust and error investigation. For regulated contexts (financial advice, medical support), the reasoning chain may be required as part of the audit trail.
History & origin
Chain-of-thought prompting was introduced in a 2022 paper by Wei et al. at Google ('Chain-of-Thought Prompting Elicits Reasoning in Large Language Models'). The researchers found that including worked examples with step-by-step reasoning in a prompt dramatically improved LLM performance on arithmetic and reasoning tasks. A follow-up 2022 paper (Kojima et al.) showed that simply adding 'Let's think step by step' let models produce their own reasoning chains (zero-shot CoT), and later work found that longer, more detailed chains improved accuracy on complex tasks. This finding influenced the design of models trained to reason explicitly (like OpenAI's o1/o3 and Anthropic's Claude extended thinking), which generate extended internal reasoning before producing a final output.