What Are Large Reasoning Models (LRMs)? Smarter AI Beyond LLMs

IBM Technology

21,335 views • 8 months ago Save 3 min 5 min read

Video Summary

Large Reasoning Models (LRMs) represent an advancement beyond standard Large Language Models (LLMs) by incorporating a planning and verification phase before generating output. Unlike LLMs that predict the next token statistically, LRMs first sketch a plan, weigh options, and double-check calculations in a sandbox. This "thinking" process, though more computationally intensive, allows LRMs to tackle complex tasks like debugging code or tracing financial flows more effectively than LLMs. The development of an LRM involves massive pre-training, followed by specialized reasoning-focused tuning with curated datasets of logic puzzles and multi-step problems, and further refinement through reinforcement learning. Interestingly, the most intelligent AI models currently scoring highest on benchmarks are LRMs.

Short Highlights

LRMs predict the next token after planning, weighing options, and double-checking calculations, unlike LLMs which solely predict statistically.
LRMs are beneficial for complex tasks such as debugging gnarly stack traces or tracing cash flow through shell companies.
Building an LRM involves massive pre-training on vast datasets, followed by specialized reasoning-focused tuning with curated logic puzzles and math problems.
Reinforcement learning, including RLHF or using smaller process reward models, is used to train LRMs by rewarding logical coherence.
LRMs offer complex reasoning, improved decision-making, and require less prompt engineering, but come with higher computational costs and increased latency.

Key Details

The Distinction Between LLMs and LRMs [00:00]

LLMs predict the next token in a sequence using statistical pattern matching to generate humanlike text.
LRMs, or Large Reasoning Models, go a step further by thinking before they talk, planning and verifying before responding.
An LLM, given a prompt, will predict the statistically fitting next word sequentially.
LRMs, while also outputting tokens, first sketch out a plan, weigh options, and double-check calculations in a sandbox before building their response.
This internal chain of thought allows LRMs to test hypotheses and discard dead ends, leading to reasoned answers.

"And those extra steps, they can matter."

The Trade-offs of LRM Functionality [01:42]

The advanced capabilities of LRMs come at a cost of inference time and GPU dollars due to extra passes through the network, self-checks, and search branches.
This adds latency and processing time, meaning LRMs offer deeper reasoning at the expense of a longer, pricier "think" phase.
LRMs are particularly effective for complex tasks like debugging stack traces or tracing intricate financial flows where simple statistical prediction is insufficient.
The internal chain of thought in LRMs enables them to test hypotheses, discard dead ends, and arrive at a reasoned answer rather than just a statistically likely pattern.

"So LRM, they buy you deeper reasoning at the cost of a longer, pricier think."

Building a Large Reasoning Model [02:03]

LRMs are typically built upon existing LLMs that have undergone massive pre-training on billions of web pages, books, and code repositories to gain language skills and broad knowledge.
Following pre-training, LRMs undergo specialized reasoning-focused tuning.
This tuning involves feeding the model curated datasets of logic puzzles, multi-step math problems, and tricky coding tasks, each with a full chain-of-thought answer key, teaching the model to "show its work."
The model learns to start with a problem, develop a plan, execute that plan in multiple steps, and ultimately arrive at a solution.
Reinforcement learning is then applied, using a reward system where either humans (RLHF) or smaller "process reward models" judge each step of the reasoning chain as good or bad.

"So we're now going to finetune the model specifically to provide reasoning capabilities."

Advanced Training Methods and Runtime Inference [04:34]

Other training methods, such as distillation, can be used where a larger "teacher" model generates reasoning traces, which are then used to train a smaller or newer model.
The result is a model capable of planning, verifying, and explaining, ready for complex tasks.
At runtime, the amount of "thinking time" or compute allowance given to an LRM can be varied based on the question's complexity.
For instance, debugging a stack trace might receive ample compute, while writing a caption might use a "budget version" with fewer passes.
During extended inference, an LRM may run multiple chains of thought, vote on the best one, backtrack using tree search, or call external tools like calculators or databases.

"And each extra pass through the model, well, it comes at a cost."

Benefits and Drawbacks of LRMs [06:33]

LRMs offer complex reasoning capabilities for tasks requiring multi-step logic, planning, or abstract thought, and improved decision-making due to internal verification.
They often require less prompt engineering, as the model inherently thinks step-by-step.
However, LRMs incur higher computational costs, demanding more VRAM and energy, leading to higher cloud provider invoices.
There's also an increase in latency, resulting in slower replies while the model processes its thoughts.
Despite the trade-offs, LRMs are the leading models in AI benchmarks, demonstrating a shift towards AI that takes time to think through responses rather than just predicting the next word.

"So, so look, with LRM, AI models are no longer just spewing language out at you as fast as they can predict the next word in a sentence."