“Modern AI can recite oceans of facts, yet it often stumbles on the shallow stream of logic that connects them. Intelligence is not only knowing many things, but knowing how one truth must lead to the next.” – MJ Martin
Artificial intelligence platforms have advanced rapidly in recent years, from tools that complete sentences to systems that generate essays, code, and creative text. Yet despite these breakthroughs, one persistent weakness stands out. Many large language models fail unexpectedly on what should be simple reasoning tasks that require a chain of logical steps. These failures occur even when the questions involved information that could be retrieved with a basic web search. The problem lies not in the quantity of data these systems have ingested, but in how they internally represent and reason about relationships across a sequence of facts. This phenomenon is often referred to as the multi-hop reasoning problem.
The phrase multi-hop reasoning refers to a class of tasks that require connecting several discrete pieces of information to arrive at a conclusion. A simple example close to everyday experience would be asking a human to find the answer to a question that involves locating one fact, then using that fact to find another, and so on. In educational settings, students tackle these problems routinely. Teachers assign word problems in mathematics that require multiple steps, science tasks that link cause and effect, and history assignments that require connecting events to outcomes. For humans, these tasks may be nontrivial, but they are well within the capabilities of a curious high-school student equipped with search tools and critical thinking skills.
In contrast, many advanced AI systems struggle with these questions. A typical failure mode occurs when an AI is asked to synthesize a correct response based on several linked facts drawn from separate sources. While the model may repeat individual facts with high accuracy, it can fail to combine them in a logical sequence. As a result, responses may be internally inconsistent, incomplete, or simply wrong in ways that would be obvious to a human reader.
This investigative paper explores why these failures occur. It examines both the technical foundations of large language models and the nature of multi-hop tasks. It also investigates why these issues persist even as AI systems grow larger and more sophisticated, and why current mitigation strategies only partially solve the problem.
An Example of the Challenge for AI
A classic example of a multi-hop reasoning problem that exposes weaknesses in AI platforms would be the following query: List all hotels in the greater Vancouver area designed before 1980, that have highly rated restaurants by Michelin Guide standards, hold a four star or higher overall rating, and offer a diverse menu selection for lunch.
On the surface, this appears to be a straightforward search task, yet it requires multiple sequential reasoning steps.
First, the system must identify what qualifies as the greater Vancouver area and compile a comprehensive list of hotels within that geographic boundary.
Second, it must determine which of those properties were originally designed or constructed before 1980, which often requires consulting architectural or historical records rather than simple marketing descriptions.
Third, it must verify whether the hotel contains an in-house restaurant that has been evaluated by the Michelin Guide, and if so, confirm the rating status according to Michelin’s standards.
Fourth, it must cross-reference the hotel’s overall star classification from recognized hospitality rating agencies.
Finally, it must evaluate whether the restaurant offers a diverse lunch menu, a qualitative assessment that may require analyzing menus or reviews. Each step depends on the accurate completion of the previous one, and the information needed resides across multiple independent data sources.
While a human researcher could complete these steps methodically using search tools and verification, AI systems frequently struggle to chain these requirements reliably, leading to incomplete lists, conflated ratings, outdated information, or fabricated details.
Understanding Large Language Models
Large language models, including those developed by major research labs, are built on neural network architectures trained on vast amounts of text. During training, these systems encounter words, phrases, and entire passages from books, articles, websites, and other digital sources. Through a process called self-supervised learning, the model learns statistical patterns that let it predict the next word given preceding context. In effect, the model builds a statistical representation of language that encodes probable word sequences.
This approach is powerful for generating coherent text, summarizing content, and completing sentences in plausible ways. However, it does not instantiate a formal, symbolic understanding of knowledge. Instead of manipulating explicit facts, rules, and logical relationships, the model relies on patterns in the training data to produce likely continuations. That is why these systems excel at stylistic mimicry but falter when a correct answer depends on symbolic chaining across multiple intermediate facts.
In a multi-hop task, the model must recall one fact, then use that fact as the basis for another retrieval, repeating this process until the final answer is assembled. Because the model does not have an explicit structure for chaining facts in this manner, it is forced to approximate multi-step logic through the patterns it learned. When the reasoning path diverges from typical patterns in training data, performance degrades quickly.
The Anatomy of a Multi-Hop Query
Most multi-hop queries involve a sequence of linked sub-questions. For example, a question might ask for the name of a politician who succeeded another in office, then inquire about an event associated with that successor. To answer correctly, one must identify the intermediate relationship and then apply it to resolve the final question.
Humans handle this by internally storing discrete facts and reasoning over them. A student can use a search engine to find the sequence of facts, then consciously combine them. The student might even write them down to track intermediate steps. Language models, however, do not have discrete representations accessible to an external reasoning process. They instead encode relationships implicitly in high-dimensional vector spaces that correlate words and contexts. Extracting the right answer therefore becomes a matter of pattern recall, rather than intentional logical deliberation.
To illustrate concretely, imagine a question that requires linking the establishment year of a company, the name of its founder, and the birthplace of that founder. A human might search for the company’s history page to find its founder, then search for biographical information about that person. Each step yields an explicit fact that feeds into the next. In a language model, each step is an implicit statistical activation within neural weights. The model does not “remember” past steps in the same explicit way, so it is vulnerable to conflating facts or skipping essential intermediate connections.
Why High-School Search Tasks Are Hard for AI
High-school students typically answer multi-hop questions by using search tools to locate and connect facts. A student might open several browser tabs, read source material, evaluate trustworthiness, and synthesize the result. They engage in a controlled process of reasoning, checking and cross-checking. Modern AI platforms attempt shortcuts. They try to surface answers directly from learned patterns without explicitly consulting external sources or breaking down the task into distinct reasoning phases.
Many AI architectures embed everything they know into gigantic parameter matrices. This design contributes to impressive fluency, but it obscures individual facts. Because the underlying data is compressed, the system cannot always access the precise information needed. When a question requires two or more linked facts, the errors in approximation can compound. One mistake in an early linkage can propagate, causing the final answer to be incorrect or incoherent.
Researchers have attempted to address this problem using specialized training data, fine-tuning on reasoning benchmarks, and incorporating retrieval mechanisms that allow the model to consult external knowledge sources. Some systems integrate search engines or knowledge bases to ground responses in factual text. While these approaches improve performance on certain tasks, they are not perfect. Errors still occur because the model must still determine how to integrate the retrieved information coherently.
Persistent Challenges and Misconceptions
One misconception about AI is that bigger models are inherently smarter. Increasing the number of parameters improves pattern recognition but does not confer a symbolic reasoning engine. Without a representation of discrete logic that can enforce explicit reasoning chains, errors will persist. Another common assumption is that fine-tuning on reasoning datasets will eliminate the problem. However, because these datasets are limited examples of structured logic, training on them does not guarantee generalization to all multi-hop scenarios. The model may learn shortcuts that work for benchmark tests but fail in real-world contexts.
Another persistent challenge is the evaluation of correctness. Multi-hop tasks often require precise answers, but language models can produce output that appears plausible while being factually wrong. In educational or investigative settings, this can mislead users into trusting incorrect responses. The inability to provide explanations for intermediate reasoning steps further undermines confidence in the model’s output.
Paths Forward
To address the multi-hop reasoning problem, researchers are exploring hybrid architectures that combine neural networks with symbolic reasoning modules. These systems aim to bridge pattern recognition with structured logic processing. Another promising direction is improving retrieval-augmented generation (RAG), where models retrieve and cite external sources before synthesizing an answer. This can help ground responses in verifiable information rather than internal approximations.
Advances in interpretability research also aim to make the reasoning process more transparent, enabling models to provide step-by-step rationales. If successful, this could make it easier to diagnose where mistakes occur and to correct them systematically. Finally, continued benchmark development that emphasizes reasoning over pattern matching can push the field toward more robust solutions.
Summary
The multi-hop reasoning problem remains a fundamental challenge for modern AI systems. Although these platforms have made remarkable progress in generating fluent text and answering simple questions, they frequently fail on tasks that require chaining multiple facts logically. These failures highlight limitations in current architectures, which excel at pattern recall but struggle with explicit reasoning. Addressing this gap will require new approaches that combine the strengths of neural language models with structured reasoning and retrieval mechanisms. Only then will AI systems begin to approach the level of logical consistency that even a high-school student exhibits when solving multi-step problems.
About the Author:
Michael Martin is the Vice President of Technology with Metercor Inc., a Smart Meter, IoT, and Smart City systems integrator based in Canada. He has more than 40 years of experience in systems design for applications that use broadband networks, optical fibre, wireless, and digital communications technologies. He is a business and technology consultant. He was a senior executive consultant for 15 years with IBM, where he worked in the GBS Global Center of Competency for Energy and Utilities and the GTS Global Center of Excellence for Energy and Utilities. He is a founding partner and President of MICAN Communications and before that was President of Comlink Systems Limited and Ensat Broadcast Services, Inc., both divisions of Cygnal Technologies Corporation (CYN: TSX).
Martin served on the Board of Directors for TeraGo Inc (TGO: TSX) and on the Board of Directors for Avante Logixx Inc. (XX: TSX.V). He has served as a Member, SCC ISO-IEC JTC 1/SC-41 – Internet of Things and related technologies, ISO – International Organization for Standardization, and as a member of the NIST SP 500-325 Fog Computing Conceptual Model, National Institute of Standards and Technology. He served on the Board of Governors of the University of Ontario Institute of Technology (UOIT) [now Ontario Tech University] and on the Board of Advisers of five different Colleges in Ontario – Centennial College, Humber College, George Brown College, Durham College, Ryerson Polytechnic University [now Toronto Metropolitan University]. For 16 years he served on the Board of the Society of Motion Picture and Television Engineers (SMPTE), Toronto Section.
He holds three master’s degrees, in business (MBA), communication (MA), and education (MEd). As well, he has three undergraduate diplomas and seven certifications in business, computer programming, internetworking, project management, media, photography, and communication technology. He has completed over 60 next generation MOOC (Massive Open Online Courses) continuous education in a wide variety of topics, including: Economics, Python Programming, Internet of Things, Cloud, Artificial Intelligence and Cognitive systems, Blockchain, Agile, Big Data, Design Thinking, Security, Indigenous Canada awareness, and more.