Building an AI-Assisted Candidate Brief From Public Web Sources

Most recruiting teams build a candidate brief the same way: paste a LinkedIn URL into a chat window and ask the model to "summarize this person." It works, until it doesn't, and the failure mode is quiet. The page it was fed was half-rendered JavaScript scaffolding instead of an actual profile, so the model writes a confident, polished, occasionally wrong paragraph about someone's job history.

LLMs are capable synthesizers. Clean, structured, current information about a candidate lets it write a sharper brief than most humans could in the same ten minutes. The bottleneck sits earlier, at retrieval: getting clean, current, structured data out of public sources that are JavaScript-rendered, geo-gated, or actively fingerprinting the traffic hitting them. A brief is only as good as what made it into the prompt, and what makes it into the prompt depends on five separate engineering stages: source identification, structured extraction, format normalization, LLM synthesis, and output.

Treating this as a pipeline instead of a prompt-engineering exercise changes where effort goes. A developer who thinks "I need a better prompt" will spend a week tuning instructions against bad data and wonder why the output still hallucinates a job title. A developer who thinks "I need a better pipeline" will trace the problem to the second stage, fix the extraction, and watch the exact same prompt start producing accurate briefs. The rest of this piece walks that pipeline stage by stage, starting with the sources themselves.

Mapping the public sources that feed a candidate brief

Before any code gets written, the sources need mapping. Not every public page about a candidate is worth the same engineering investment, and a brief built for one hiring purpose will lean on a different mix of sources than a brief built for another.

A typical candidate brief draws from seven source types:

• LinkedIn profile: current role, tenure, career history, education, skills, and endorsements. It's also the hardest source to retrieve, since it renders client-side and requires session handling, so a raw HTTP fetch returns an empty shell.
• Current employer's About, Team, or Press pages: confirms current role, company context, funding stage, and product focus. Usually static or lightly dynamic HTML, which makes it one of the easier sources to pull.
• Personal site or portfolio: work samples, self-description, stated expertise, and contact context. These range wildly in build, from a static one-pager to a full React single-page app, so the retrieval difficulty here is genuinely a coin flip until you look.
• Published writing on blogs, Substack, Medium, or industry publications: shows depth of thinking and communication style. Generally static HTML, though pulling a full archive may mean following pagination across multiple pages.
• Conference and podcast appearances: speaker bios and talk topics, usually sitting on event sites with inconsistent structure from one conference to the next.
• News and press mentions: third-party validation, funding announcements, award mentions, and direct quotes. News HTML is typically the most accessible source type in the entire list.
• GitHub or other public code repositories: language fluency, project types, activity recency, and collaboration patterns, available as structured JSON through a public API.

Lining those seven up by retrieval difficulty produces a spectrum. GitHub sits at the easy end with a clean REST API. News articles are usually static HTML, not far behind. Personal sites vary. LinkedIn and some company pages sit at the hard end, because content on them only appears after JavaScript runs, and the traffic requesting that content gets fingerprinted at the network level, a pattern documented across 2026 agent pipeline benchmarks.

Scope the source list to the job before touching any of this. A brief for a senior engineering hire should lean on GitHub activity and published papers. A brief for a sales executive should lean on LinkedIn career history and news mentions. The map is a decision, made once per brief type, that determines where the rest of the pipeline spends its energy.

Retrieving live data from public web sources

A plain HTTP fetch covers maybe half of a candidate brief's sources, and that's being generous. The gap between what a raw request returns and what a real browser renders is exactly where pipelines fail silently, returning a seemingly successful response on a page that contains nothing but a loading spinner and a cookie banner.

The fix is to match retrieval method to page type instead of using one method everywhere:

• Static or lightly dynamic pages, which cover most news articles and plenty of personal sites, only need a well-configured HTTP client with the right headers. It's fast and cheap, and there's no reason to reach for anything heavier.
• JavaScript-heavy or session-gated pages, like LinkedIn, some company team pages, and single-page apps, need a headless or cloud browser that renders the full page and holds a session the way a logged-in human would.
• Sources with a public API, GitHub chief among them, should skip HTML retrieval. The API returns structured JSON directly, with no rendering step and explicit rate limits.

The architecture that makes this affordable is selective escalation: default to the lightweight fetch, and only route a URL through the full browser when it actually needs one. Running every request through a headless browser by default burns compute and time on pages that never needed it. A simple routing check handles the decision:

def fetch_candidate_source(url):
    if is_api_available(url):
        return call_api(url)
    elif requires_js_rendering(url):
        return render_with_headless_browser(url)
    else:
        return fetch_with_http_client(url)

requires_js_rendering can be as simple as a domain lookup table (LinkedIn, known SPA platforms) or something more adaptive that checks for an empty body on first pass and escalates automatically.

Retrieval and extraction are two different jobs, and the pipelines that blur them tend to break the moment a site redesigns its layout. Fetching the rendered page is step one. Deciding what to pull out of that page, which field maps to which part of the DOM, is step two, and it deserves its own stage separate from the fetch logic.

Most teams end up building a small internal platform by accident here, with a proxy pool, a headless browser cluster, retry logic, CAPTCHA handling, and selector maintenance scattered across scripts that nobody wants to touch. A unified web data API, one that handles rendering, session management, proxy routing, and structured output through a single interface, replaces that entire stack. Zyte's 2026 Web Scraping Industry Report describes exactly this shift: developers moving away from managing infrastructure components piece by piece and toward managing data outcomes through full-stack APIs instead.

Structuring extracted content into LLM-ready formats

Raw HTML pulled from a rendered page is packed with <div> wrappers, inline styles, tracking scripts, and navigation markup that have nothing to do with a candidate's career history, and every one of those tokens costs money and attention without adding information, so a model should never have to read it directly.

A two-layer structure solves this cleanly for a candidate brief:

• Structured fields, things like name, current role, employer, education, and tenure, get extracted as JSON directly from each source. These become the brief's factual skeleton.
• Prose sections, like a summary of published work, recurring themes in someone's writing, or descriptions of notable projects, get passed as Markdown. These become the brief's qualitative depth.

As raw HTML, a single job entry in the same LinkedIn experience section can run several hundred tokens of nested spans, data attributes, and empty anchor tags before you reach the sentence that describes what the person did. As clean Markdown, the same entry is three lines: a role, a company, a date range, and a short description. Same information, a fraction of the tokens, and a format the model can actually parse without getting distracted by markup noise.

Filtering has to happen before any of this reaches the model. Smart extraction separates the main content, the actual bio, job history, or article text, from the boilerplate around it: navigation bars, footers, ad units, cookie banners. Feeding that boilerplate into the prompt inflates token count for no benefit and can actively confuse the model when it's trying to pull a specific field out of a page that's mostly junk.

Building the extraction and normalization layer across heterogeneous sources

Seven source types means seven different structures, and none of them agree on how to describe the same fact. That disagreement is the next problem the pipeline has to solve, and it's a bigger one than it sounds.

Picture one candidate with three sources open side by side. LinkedIn lists the role as "Head of Product." The company's team page lists the same person as "VP Product." A news article from a funding announcement calls them the "product lead." All three are describing one job. Without a normalization step, a model sees three conflicting facts and has to guess which one is right, and guessing is another word for hallucinating.

The fix is defining a canonical schema before extraction starts, so every source gets mapped into the same shape. A minimal version of this schema looks like this:

candidate_schema = {
    "current_employer": None,
    "current_title": None,
    "tenure_start": None,
    "education": [],
    "prior_employers": [],
    "publications": [],
}

def merge_sources(linkedin_data, company_page_data):
    merged = candidate_schema.copy()
    for field in merged:
        linkedin_value = linkedin_data.get(field)
        company_value = company_page_data.get(field)
        if linkedin_value and company_value and linkedin_value != company_value:
            merged[field] = {
                "conflict": True,
                "linkedin": linkedin_value,
                "company_page": company_value,
            }
        else:
            merged[field] = linkedin_value or company_value
    return merged

Source priority matters here too. LinkedIn is usually the most current record for role and tenure, since people update it themselves. Company pages confirm employer details independently. News articles add third-party validation that neither of the other two can offer. Personal sites fill in self-description context that's useful but least authoritative on hard facts.

Where two sources disagree, resist the urge to silently pick a winner. Pass both values into the model with their source attached; the brief then states the discrepancy. A hiring manager reading "Head of Product per LinkedIn, VP Product per company page" learns something real. A hiring manager reading a confidently wrong single title learns nothing until it costs them later.

Deduplication belongs in this same layer. The same job, the same published article, and the same conference talk often appear across two or three sources at once, and without deduplication that repetition inflates the token count the model sees and can make it overweight whatever got mentioned three times instead of once.

At scale, add validation on top of the schema: required fields, type checking, allowed values for things like education level. A missing field caught here produces a clean structured error. A missing field caught nowhere becomes, three weeks later, an embarrassing gap in a brief someone already sent to a hiring committee.

Feeding the normalized data to an LLM for synthesis

Everything up to this point exists to make this step boring, in the best sense. A prompt built on clean, normalized, source-attributed data doesn't need clever phrasing to produce a good brief. It needs structure.

A production-grade synthesis prompt for a candidate brief has three components:

SYSTEM:
You are producing a factual candidate brief for a recruiting team.
Use only the information provided below. Do not infer or add
information not present in the sources. Where sources conflict,
flag the conflict explicitly rather than choosing one value.
If a field is absent from all sources, mark it "not found in
public sources."

STRUCTURED DATA (ground truth, do not contradict):
{normalized_json_fields}

PROSE CONTEXT (supporting evidence for qualitative sections):
{markdown_content_from_publications_news_and_other_sources}

TASK:
Produce a JSON-formatted brief with these sections: summary,
career_history, key_themes, notable_work, flags_and_discrepancies.

The system instruction sets purpose and constraints up front: who the brief is for, and what the model is explicitly barred from doing, which is inventing. The structured data block carries the facts the model must not contradict. The prose block carries supporting color, the kind of material that makes a brief read like it was written by someone who actually read the candidate's work.

Grounding discipline earns its keep here. Instructing the model to attribute specific claims, writing "according to their LinkedIn profile" or "as reported in [publication]," does two things at once: it cuts the chance of invented detail, and it makes the brief auditable, so a recruiter can trace any line back to where it came from.

Output format matters as much as content. Structuring the result as JSON with fixed sections, summary, career history, key themes, notable work, and flags or discrepancies, keeps the brief readable by a human and parseable by a machine. That same JSON can render straight into a document or feed directly into an applicant tracking system downstream.

Gaps deserve explicit handling too. When a field never appears in any source, the instruction should tell the model to mark it "not found in public sources. Inference from absence is one of the most common ways a factual extraction task quietly turns into a hallucination task.

Running this pipeline at scale: batching, monitoring, and keeping briefs current

A pipeline built for one candidate turns into an infrastructure project the moment a recruiting team points it at a few hundred people at once. Batching, error handling, and freshness checks stop being optional at that point.

On the intake side, validate every URL before retrieval starts: flag dead links, paywalled pages, and login-walled profiles when the batch is submitted, before a run burns an hour of compute on them. Retrieval itself should run in parallel at the source level, pulling every URL tied to one candidate concurrently, while rate-limiting per domain so the pipeline doesn't trip bot detection on LinkedIn or anywhere else that's watching for exactly this kind of traffic pattern.

For large batches, a queue-and-webhook setup holds up better than synchronous polling: submit a batch of candidate URL sets, then receive structured results back asynchronously through a webhook as each one finishes. Agent-native APIs increasingly expose this pattern as a default rather than an afterthought, which fits how batch recruiting work actually runs.

Briefs also go stale, because people keep living their careers after the brief gets written. A new role, a published article, a funding announcement, a conference talk, any of these can materially change what a brief should say weeks after it was generated. The monitoring pattern that handles this defines the set of URLs tied to each candidate, sets a re-crawl schedule, diffs the newly normalized output against the stored version, and triggers a refresh only when the diff turns up a substantive change. That keeps the pipeline from regenerating briefs on every trivial page edit, and it makes sure nobody's handed a brief that describes a job the candidate left two months ago.