Automating Systematic Literature Reviews With Web Research Agents

Automating systematic literature reviews with web research agents is a pipeline design problem. The useful approach splits retrieval, screening, extraction, and synthesis into specialised stages, feeds each one clean, structured data, and places human checks where errors do the most damage.

Why systematic literature reviews are still a manual bottleneck despite decades of tooling

Systematic literature reviews ask researchers to find, screen, and synthesise thousands of studies using clear inclusion and exclusion criteria. Each stage feeds the next, so a delay in retrieving full texts can hold up extraction and synthesis. AgentSLR at ICML 2026 examined epidemiological reviews of nine priority pathogens designated by an international health organisation and found that a manual review consumed roughly seven weeks of calendar time.

Adding an LLM to one task does not remove that dependency. Existing LLM-assisted workflows still face false positives, false negatives, prompt sensitivity, and failures in coordinating stages. Those are distinct problems from whether a model can understand a paper. As publication volumes grow, isolated shortcuts leave researchers with an expanding backlog and a faster way to reach the next bottleneck.

What a fully agentic SLR pipeline looks like end to end

An end-to-end pipeline assigns separate jobs to separate agents. Retrieval finds candidate articles, title and abstract screening filters them, and full-text screening applies stricter criteria. Extraction puts the evidence that remains into defined fields, which synthesis uses to build a report. Each stage needs its own inputs and can cope with different kinds of errors.

AgentSLR makes that division concrete. It starts with Boolean searches against bibliographic databases, screens abstracts with language reasoning models, and converts PDFs to Markdown through OCR. Full-text screening then precedes tool-calling extraction with schema validation, followed by report generation and iterative refinement. Conversion is an explicit bridge to extraction.

The ISPOR Europe 2026 platform also keeps first-pass screening separate from field-level extraction. Santos and Nascimento (2026) demonstrate a modular screening architecture using n8n and a locally run language model. The pattern does not depend on a high-compute setting. The pattern works well because each task has a clear contract.

What the retrieval stage actually needs from the web layer

A relevant article that retrieval misses never reaches screening. Retrieval therefore sets the upper bound on recall, so choosing which sources to cover is a key part of the design. A database such as PubMed supplies candidate records through domain-specific Boolean searches, while open-access PDFs must be located and fetched separately.

The handoff needs complete, deduplicated records in clean Markdown or JSON. AgentSLR processed thousands of articles at abstract screening, and asking the model to read raw HTML at that scale makes it do conversion work over and over. Navigation text and parsing noise compete with the title and abstract that determine inclusion.

Access failures can damage coverage just as quietly. Cloudflare now blocks AI Training and Agent bots by default on ad-supported pages for new domains. A retrieval layer must show when access is blocked so that missing results do not look like a search that found nothing.

How title/abstract screening agents make decisions and where they fail

Santos and Nascimento found high recall and much less need for manual screening, while the ISPOR platform checked first-pass decisions against outcomes that humans had reviewed. These results support automation at the screening stage. They do not make every wrong decision equally acceptable. For pipeline design, the distinction between over-inclusion and a critical miss is more useful than a single accuracy score.

An over-included article can be rejected during full-text screening. A relevant article excluded early is absent from every later stage, so downstream precision cannot repair the loss. The design implication is to favour higher recall at first pass and recover precision later.

Screening agents apply inclusion and exclusion rules to titles and abstracts, but that reasoning is not their only vulnerability. AgentSLR identified sensitivity to prompt formulation and orchestration failures at stage transitions. Evaluating the model alone therefore leaves part of the decision system untested.

Turning full-text PDFs into structured data the extraction agent can use

PDF conversion helps explain why pipelines with the same screening agent can get different extraction results. Full-text reasoning needs machine-readable text as well as the file itself. A successful download is only the beginning. Conversion errors can pass into extracted fields and then into synthesis.

The demands increase after abstract screening. AgentSLR’s full-text stage processed roughly an eighth of the abstract-screening volume, but needed methodology, population, and outcome details in addition to topic relevance. Preserving section structure gives the agent the context needed to apply stricter criteria. Text that looks broadly readable may still be inadequate for a context-dependent field.

The ISPOR platform found that most extraction errors were in complex or context-dependent variables, while accuracy was high for core epidemiologic, study-design, and population characteristics. That pattern makes input quality especially important, though it does not isolate conversion as the cause. For articles hosted on the web, direct HTML-to-Markdown conversion cuts out a separate cleanup stage and leaves fewer chances for conversion error.

What schema-constrained extraction means in practice and why it beats free-text output

Schema-constrained extraction requires output to match a predefined field structure before acceptance. AgentSLR uses specialised tool calls and field-level validation to extract epidemiological parameters, transmission models, and concluded outbreaks. Validated structured output organised consistently across papers makes programmatic report generation possible.

The ISPOR platform had 91.9% extraction accuracy across 27 structured variables. Defined fields made that accuracy measurable in the first place. Without a shared schema, free-text summaries require human normalisation before aggregation, giving the saved time back with interest. A schema makes extraction testable, though having the right structure does not prove that the content is right.

Model selection needs to reflect that job. AgentSLR’s model ablations found that distinctive model capabilities drove performance more than raw size or inference cost. Developers should therefore evaluate schema adherence and tool-calling reliability. The input still needs to be clean because validation cannot bring back context lost at an earlier stage.

Where hallucination and citation error enter the pipeline and how to design against them

The most dangerous error is one that gains authority as it travels. A fabricated finding or citation can pass from one agent to another until synthesis treats it as established evidence. The GUEST guidelines (2026) identify plausible but unverifiable text as a fundamental threat to the rigour and transparency of systematic reviews. Fluent output is not an audit trail.

This risk grows when systems draw from the same biased or partly fabricated corpus and treat earlier AI outputs as independent support. The NSA’s AI Security Center identifies a related vulnerability in agent systems using MCP, where one agent implicitly trusts another’s output. Agreement between agents does not necessarily mean independent confirmation.

The architectural response is to treat each output as untrusted input. Schema validation catches malformed extraction records, while citation verification against live sources addresses fabricated references. Grounding synthesis in retrievable evidence makes its claims auditable. Human review remains necessary because neither valid JSON nor a live URL establishes that a conclusion is justified.

Where to place human-in-the-loop checkpoints and what each one must verify

Human checkpoints should follow confidence and consequence. Santos and Nascimento state that “the researcher remains responsible for the final decision in ambiguous cases”. That makes uncertainty a trigger for review. It avoids turning automation into a second employee whose entire workload someone must repeat.

At retrieval, human review should compare the returned corpus size and source distribution with expectations, looking for missing databases or access blocks. At title and abstract screening, attention belongs on the low-confidence queue near the inclusion threshold. Full-text screening needs a different check. Articles excluded despite matching the earlier inclusion criteria form a critical-miss risk set, because they will otherwise disappear before extraction.

At extraction, spot-checks should compare records with source PDFs, with mandatory review for context-dependent fields where the ISPOR platform found concentrated errors. Synthesis needs humans to review all conclusions before they are used in policy or published. The GUEST guidelines report only moderate agreement between AI-assisted risk-of-bias assessments and expert reviewers, which makes that final gate more than a formality.

How MCP and emerging interoperability standards change what agents can retrieve

The Model Context Protocol gives agents structured tool interfaces beyond page scraping. For retrieval, that opens a route to bibliographic data supplied in a consistent schema. Less parsing means fewer transformations between a source record and the screening input.

WebMCP opens that route to tools that websites make available, but is still an early preview. Where a publisher exposes a structured tool, an agent could receive bibliographic fields directly. That possibility is not a guarantee of publisher coverage. By April 2026, the Agent-to-Agent protocol had reached broad adoption and shipped in Azure AI Foundry, Copilot Studio, and Amazon Bedrock AgentCore Runtime, supporting standard handoffs across hosted services.

The NSA’s MCP guidance warns about implicit trust relationships between agents. A standard protocol improves connectivity, but does not certify the evidence passing through it.

Monitoring the literature continuously rather than running point-in-time reviews

Even a faster review can go out of date. AgentSLR includes living systematic review generation, with report production designed for iterative updates as new evidence appears. That requires monitoring beyond the initial search. The pipeline needs to detect new matching publications, corrections, retractions, and changes in access to previously retrieved papers.

Watching for publications that match a Boolean query is a lightweight continuous process, while rerunning the entire search is a heavier scheduled operation. The architecture should support both so that each update can happen without starting a fresh review. Otherwise, “living” becomes a polite name for repeatedly starting over.

The GUEST guidelines call for prospective evaluation in real-world operational settings. A continuously running review provides that setting, where source availability and the evidence base change over time. Monitoring therefore belongs in the operating design.

What synthesis agents can and cannot yet do reliably

Synthesis is the least defensible place to confuse speed with reliability. AgentSLR combines programmatic descriptive generation with iterative writing, critique, and evidence grounding, completing report generation in under five minutes per pathogen. That is rapid report production. It does not, by itself, establish expert-level synthesis.

The MedEvidence benchmark (arXiv 2025) found that frontier LLMs failed to match systematic-review conclusions in a substantial proportion of evaluated cases with inputs held constant. The supported role is therefore bounded. Agents can reliably perform programmatic aggregation of validated records, including counts, ranges, and distributions. Inferential judgements about effect direction, heterogeneity, or policy implications remain a different task.

The GUEST guidelines require experts to review synthesis outputs before they are used. That is a governance requirement for current tools. Automation can get the evidence ready and draft the account while leaving the final interpretation to humans.

Assembling the complete pipeline and its web data requirements

The full architecture links stages through data contracts that stay consistent. Retrieval supplies structured bibliographic results across sources and exposes access failures. Screening receives consistently encoded titles and abstracts without HTML artefacts. Full-text processing preserves methods, results, and conclusions through high-fidelity OCR or direct Markdown conversion, giving extraction usable context.

Extraction then maps that text into validated fields, while synthesis remains grounded in live sources and subject to expert review. Monitoring feeds new publications and retractions into the pipeline’s update logic. Each handoff has a specific obligation. A unified interface for search, scrape, crawl, and monitor reduces the normalisation overhead that separate tools can introduce at those boundaries.

The measure of success is correct, usable evidence. AgentSLR reports a 58× reduction in calendar time, but those gains depend on avoiding manual cleanup that consumes the time automation saves. Human checkpoints, schema validation, and source grounding make the pipeline fit to produce a systematic review.