Automating University Recruiting Pipelines With Public Graduation Data

By the time a new graduate posts a résumé to a job board, the best recruiters have already found them somewhere else. That's the entire argument of this piece: public university data lets employers identify strong candidates before they ever touch an applicant tracking system, and building the pipeline to do that is now a solvable engineering problem, not a research project.

The pressure behind this comes down to simple math. LinkedIn research cited by CNBC found that most recruiters planned to increase their use of AI pre-screening in 2026, and that squeezes the time between a job posting going live and the role getting filled. Candidates aren't just competing against other candidates anymore. A recruiter sifting through five hundred near-identical cover letters isn't finding candidates; they're panning for gold in a river someone dumped glitter into.

Meanwhile, real differentiation exists in the graduate pool and gets buried. Handshake's analysis found that seniors graduating in 2026 mentioned AI skills on their résumés at twice the rate of the class of 2022, and nearly three-quarters of those mentions traced back to actual projects they had built. The problem is that it only becomes visible once a résumé lands in a stack, and most of those résumés never make it that far. Federal Reserve Bank of New York data puts the underemployment rate for recent college graduates well above a third, so capable, qualified people are landing in jobs beneath their skill level, partly because the employers who'd value them never crossed paths with them in the first place.

None of this is a screening problem. It's a timing problem, and better filters on the inbound pile don't fix it. So the fix is to source candidates before the pile exists, directly from the open web, before the application stage even starts. The data is already public. The engineering challenge is building a reliable pipeline to collect and structure it.

Public graduation data: formats and publication sources

Before writing a line of crawler code, it helps to know exactly what's sitting out there waiting to be read. Universities publish several distinct flavors of graduation data on the open web, and each one behaves differently once you try to scrape it.

Commencement programs are the backbone. These are printed-program-turned-PDF or HTML pages listing graduates by department and degree, published every year around the May, August, and December ceremony windows. They're the primary source for name, degree, and graduation cohort, and most institutions treat them as a given, predictable publication.

Department honor rolls and award announcements go further. These usually show up as news posts or standalone pages maintained by individual schools, and they often include GPA distinctions like summa or magna cum laude, thesis titles, the name of a faculty advisor, and specific named awards. That's a richer vein of signal than a bare commencement list, since it starts to describe what a graduate actually did, not just that they finished.

Machine-readability varies by how well-resourced the institution is, not by how good the school is. Large research universities tend to publish clean, structured HTML commencement pages. Smaller institutions frequently publish nothing but PDFs, which means a different extraction path has to exist in the pipeline from day one. This is a scoping detail, not a judgment call on any particular school.

Source data also comes with built-in mess. At the May 2026 commencement at Glendale Community College, an AI system reading names mixed up or skipped hundreds of them, and College President Tiffany Hernandez had to acknowledge the error mid-ceremony. It starts at the source, in how the data was entered in the first place, and it travels downstream into any pipeline that ingests it unchecked. That means a cleaning and normalization step isn't optional. It's load-bearing.

Mapping the source landscape before writing a single crawler

A crawler built without a map first will break constantly, miss the pages worth the most, and burn compute on sources that return almost nothing useful.

The seed list doesn't get built by hand. Use a web search API to run queries like site:edu "commencement" "class of 2026" filetype:html or site:edu "honor roll" "graduation" department, which surfaces the exact pages a crawler will process later instead of anyone manually clicking through hundreds of university websites. For each domain that turns up, run a site-mapping pass to enumerate the commencement and alumni subdirectory structure before crawling goes any deeper, so the pipeline doesn't waste requests poking around gated or irrelevant sections of a site.

While mapping, a handful of questions determine how each page gets handled later: is it HTML or PDF? Every answer routes that page to a different crawler configuration downstream.

Mapping also catches staleness. A commencement page with a last-modified header from six months back is worth crawling now. One that hasn't changed in three years is low priority for a current model, though it matters for building a historical cohort model.

Crawling commencement pages at scale: architecture decisions that determine reliability

Fetching a page is the easy part. Modern infrastructure handles that without much thought. What the crawling layer actually has to do is stay reliable across hundreds of institutions, each running a different tech stack, publishing on a different schedule, and reacting to automated traffic differently.

A few architectural choices decide whether this holds up. Rendering strategy comes first: static HTML pages, which cover most commencement lists, can be fetched directly, while JavaScript-rendered department sites or alumni portals need a headless browser path. When you mix both inside one pipeline, routing logic has to live at the URL level, deciding which path a given page takes before it's fetched.

Rate limiting matters more than it sounds like it should. Per-domain delays and respect for robots.txt crawl-delay directives keep the pipeline welcome on the sites it crawls.

PDFs need their own lane. If commencement programs come as PDFs, they need a conversion step, text extraction from the PDF, before any structured extraction can even start. Flagging these at the mapping stage means the pipeline routes them correctly the first time.

Bot detection deserves real attention, even on public university pages. Automated traffic now makes up a majority of all internet requests, and university IT departments run the same defenses as any commercial site: user-agent filtering, rate-based blocking, CAPTCHA on some alumni directories. A pipeline that handles these by default, rather than requiring a custom workaround for every single institution, is the difference between something that runs itself and something that needs a human on call every week.

Extracting structured candidate records from unstructured graduation pages

Getting the page is one problem. Getting usable data out of it is a bigger problem, because the real challenge is building extraction logic that generalizes across hundreds of institution-specific layouts without writing a custom selector for each one.

A minimal candidate record coming out of this stage needs: full name as published, degree type (B.S., M.S., Ph.D.), major or field of study, institution and department, graduation term (something like "May 2026"), any listed distinction (summa cum laude, a named department award, a fellowship), and the source URL plus crawl timestamp for provenance and later deduplication.

Two extraction strategies cover most of the ground, and each fits a different kind of page. CSS and XPath selectors work well on institutions with consistent, well-structured HTML, think table-based commencement lists or tagged department pages. If you write one good prompt, it generalizes across layout differences that would otherwise take dozens of hand-built rules to cover.

Names get mangled or dropped during extraction from these pages just as they were at Glendale. If you skip that step, deduplication breaks quietly, and matching against GitHub or other enrichment sources downstream starts producing false negatives that are hard to spot until the candidate list looks thinner than it should.

Enriching candidate records by cross-referencing public professional signals

A name, a degree, and a school is a lead. It's not a candidate profile yet, and enrichment is what turns it into one without ever requiring the person to have applied for anything.

A few public sources carry high-value signal for new graduates. Public GitHub repositories linked to a university email domain or named institution show commit history and project descriptions that speak directly to technical skill and the kind of work someone's actually done, lining up with the Handshake finding that 74% of AI-skill mentions in 2026 graduate résumés traced back to real-world projects. University research indexes matter too: thesis repositories, departmental publication pages, and preprint servers like arXiv and SSRN. Personal and project sites round things out, since many graduates link a portfolio from their GitHub profile or a department page, surfacing project evidence a commencement page would never contain on its own.

This enrichment step runs as a second pass, not a parallel build. It starts from the candidate record the extraction stage already produced, runs targeted lookups for that specific person, and appends new fields to the existing record. So if enrichment fails, it can't corrupt the base record underneath it.

Matching across sources takes fuzzy logic, since the same graduate might appear as "J. Anchoring each match to institution name and graduation year cuts down on false positives considerably.

Storing, deduplicating, and refreshing candidate records across graduation cohorts

A pipeline that re-crawls without deduplication or cohort awareness ends up producing duplicate outreach, stale records, and eventually degrades the exact signal it was built to improve.

Deduplication works in layers. Secondary deduplication handles URL variants: the same page fetched over HTTP and HTTPS, with or without a trailing slash, through a canonical link versus an alias. Enrichment-level deduplication handles people, not pages: a GitHub profile or portfolio site matched to two separate commencement records (a dual-degree graduate, for instance) should merge those records into one candidate, not create two entries pointing at the same human being.

Cohort tracking needs its own dedicated field. Graduation term should be stored directly, not inferred later from other data, so the pipeline can tell a May 2025 graduate apart from a December 2025 graduate and from a May 2026 graduate. Cohort age ends up being one of the most common filters in any recruiting workflow built on top of this.

Freshness logic differs by source type. If you schedule light re-enrichment passes at 30 and 90 days after graduation, you catch that later activity without re-running the whole pipeline from scratch.

Structuring candidate records as AI-ready data for downstream recruiting workflows

A candidate record sitting in a messy, inconsistent format isn't useful to anything downstream, whether that's a CRM, an LLM-powered agent, or a Slack webhook. It needs to be structured, schema-consistent JSON that those systems can consume without extra cleanup first.

Being "AI-ready" means a few concrete things. Field names and types stay consistent across every record, regardless of which institution or page format it came from: a field called graduation_term is always a normalized date string, never free text like "Spring '26." And a signal_score or tag field encodes the recruiting-relevant distinctions directly, honors, named awards, published research, a public project in a specific technology, as filterable attributes rather than buried free text, so a recruiter can query "May 2026 computer science graduates with a public ML project" without running every single record through a model first.

That structure is what makes the downstream workflows actually work. CRM integration means posting each new candidate record into an ATS or CRM like Greenhouse or Lever as a sourced candidate, already tagged with cohort and signal metadata, so the recruiter opens a pre-built pipeline instead of a to-do list of names to go research manually. Webhook-triggered outreach means a new record matching a saved search, "PhD in materials science, May 2026, any institution," can kick off a notification or outreach sequence immediately, with the structured record itself acting as the trigger condition.

The most forward-looking pattern is the AI agent hand-off. The agent is reasoning about a clean record, not scraping meaning out of a raw web page on the fly. That's a meaningfully different, and more reliable, starting point, and it raises the next question immediately: what are the limits on reaching out to someone this way?

Legal and ethical boundaries on public graduation data

A pipeline built on public commencement pages sits on legally solid ground, as long as it stays clearly on the public side of the line FERPA draws between open web publications and records a university controls privately.

The line matters because it determines what the pipeline can legally touch. The right design choice is detecting authenticated endpoints and excluding them outright, not finding a clever way around them.

State privacy law doesn't stop applying just because a student graduated. Processing that data for commercial recruiting purposes may trigger consent and use-limitation rules that vary by state, so the pipeline should be built to collect only what it actually needs and to support deletion requests cleanly when they come in.

Robots.txt deserves the same respect as any other instruction from the site owner. If you ignore robots.txt, you create real friction with the institutions a recruiting team wants to stay on good terms with, and in some jurisdictions it also carries exposure under computer access statutes.

Outreach at the far end of the pipeline needs its own guardrails. If you skip those elements in bulk outreach, you run into exposure under CAN-SPAM and similar regulations, depending on the channel and jurisdiction involved.

Built this way, the pipeline sources people from information they or their institution chose to make public, and treats each one as a person worth reaching out to thoughtfully, not a record to be processed and discarded.

Putting the pipeline together: a reference architecture from discovery to outreach trigger

Everything above collapses into five layers, each with a clear input, process, and output, and each one swappable on its own without tearing apart the rest of the system.

Layer 1, Discovery

Process: web search API queries surface commencement, honor roll, and department news URLs; a site-map crawl enumerates the subdirectory structure underneath each one.

Layer 2, Crawling

Layer 3, Extraction

Process: CSS and XPath selectors handle structured pages; LLM-assisted extraction against a defined JSON schema handles unstructured sources; a name normalization pass runs on everything.

Layer 4, Enrichment and deduplication

Layer 5, Delivery and monitoring

Process: match against saved searches or role criteria; post to a CRM or trigger a webhook; queue an AI agent to draft outreach; schedule re-enrichment at 30 and 90 days; monitor source URLs for changes that should trigger a re-crawl.

What this whole system buys is time. The pipeline is just the infrastructure that finally makes it usable.