Benchmarking Compensation Ranges From Public Job Postings

Pay transparency laws are turning job postings into a dataset, not just a bulletin board of openings. That shift is the whole subject of this piece: how to take a pile of disclosed salary ranges and turn it into something a comp team can actually trust.

The legal push behind pay transparency laws is accelerating. Virginia's salary-range disclosure requirement took effect July 1, 2026. Maine's HB 18, signed April 24, 2026, covers any employer with ten or more people on staff. Connecticut's HB 5003, signed May 11, 2026, goes a step further and requires benefit descriptions alongside the wage range itself. None of these laws were written with data engineers in mind, but that's effectively who they're feeding. Every posting a law forces into the open is one more row in a table that didn't used to exist.

JobsPikr's 2026 AI salary benchmark report states that most salary reports are still built on surveys and self-reported numbers, both of which carry built-in lag. The report argues that the disclosed-posting corpus has grown large enough to build benchmarks from directly. That's a genuine shift in what's possible, but it comes with a catch. A legally mandated, publicly posted, constantly updating pile of salary data is only useful if something can reliably read it. That something is an engineering pipeline, and building one well is the rest of this piece.

Why survey-based benchmarking lags a fast-moving market

Traditional comp surveys break down exactly when a company needs them most: during periods when the market is moving fast. A survey is a photograph. It freezes pay at the moment someone answered a questionnaire, and that photograph ages the entire time it takes to commission the survey, collect responses, review them, publish a report, and get that report into an HR team's hands. JobsPikr's 2026 report calls this lag the core problem with how most organizations benchmark AI salaries, and the label fits. A year-old number isn't a stale coupon. In a fast-moving segment, it can be wrong enough to cost a company its next hire.

Nowhere is that lag more exposed than in AI hiring. Lightcast's AI Sector data shows AI job postings growing several-fold over in under five years. Employers broadly report that their salary structures haven't caught up with what those AI roles now demand, and payscale's own numbers back that up: 49% of employers admit their pay structures haven't kept pace with AI skills. Call it a coin flip. Half the market is pricing AI talent against a map that's already out of date.

Small sample sizes make the lag worse. Surveys pull from a limited pool of respondents, and when the talent in question is concentrated in a handful of cities and a handful of roles, that pool gets thin fast. Thin samples swing wildly from one reporting cycle to the next because a dozen new responses happened to tilt the average, not because pay actually moved that much.

Job titles are the sneakiest failure mode of the three. An "AI Engineer" at a fintech and an "AI Engineer" at a healthcare startup can share a job title and nothing else: different skill sets, different seniority, different risk tolerance for the role. JobsPikr's 2026 report notes that running both through the same survey bucket produces an average that describes neither job accurately. It's like averaging the price of a studio apartment and a beach house because the listing called both of them "homes."

Lag, small samples, and title fragmentation together produce a benchmark that looks confident on a slide deck and falls apart on contact with an actual offer letter. Job descriptions are getting rewritten faster than comp teams can price them, and more than half of employers say current market benchmarking doesn't reflect what those roles actually require now. That's the gap a pipeline built on live postings is meant to close.

Requirements for a usable compensation dataset

Before anyone calculates a single percentile, the dataset needs a spec. Every usable compensation record should carry the same core fields: role title, seniority, location, remote status (as both a string and a boolean), currency, pay period, pay minimum, pay maximum, and a pay-disclosed flag. That's the schema a build guide for salary benchmarking lays out, and it holds up because it treats "no salary listed" as information worth keeping. A posting with no disclosed pay still tells someone something: maybe about the employer, maybe about the state it's hiring in. Dropping that record or guessing a number destroys the signal.

A benchmark is worth trusting only when enough comparable postings sit behind each number and those postings are fresh. A salary band built from six stale listings is noise dressed up as a statistic. A few hundred postings refreshed every week is a real signal. Job postings don't stay accurate for long after they're published, and a stale listing can quietly inflate open-role counts and throw off every number downstream of it. Freshness is a condition for the data being usable at all, not a bonus feature.

Zyte's job posting data page lists a similar core set of fields, title, company, description, location, salary, employment type, skills, and metadata, and makes the same point from a different angle: the value isn't in collecting the fields, it's in normalizing them into one clean record per job regardless of where that job came from. A record from a company's own career page and a record scraped from an aggregator need to end up looking identical once they're in the dataset. That normalization step is covered later in this piece, but the schema decided here is what makes it possible.

Choosing where to get the data: ATS public APIs before scraping career pages

Where the data comes from decides almost everything else about the pipeline: cost, legal exposure, and how much maintenance the thing needs six months from now. The most overlooked source is also the easiest one: applicant tracking systems. Platforms like Greenhouse, Lever, Ashby, Workday, iCIMS, Recruitee, and Workable each power thousands of company career pages, and several of them offer public APIs or structured feeds that hand over clean posting data without anyone writing a scraper. Ashby, for instance, lets a caller add a query parameter to include compensation data in the API response (it isn't returned by default, so that parameter has to be requested on purpose).

Pulling from a public API instead of scraping a rendered page changes the cost and risk profile in three ways at once: no proxy infrastructure to pay for, lower legal exposure because these endpoints are built to be called, and data that shows up already structured without needing to be pried out of HTML. That's about as close to a free lunch as data engineering gets, and it's worth treating as the first stop rather than a nice-to-have.

Company career pages themselves are the original source of truth, every posting starts there, but they're the hardest tier to scale. Millions of companies means millions of slightly different HTML layouts, ATS widgets embedded in a dozen configurations, and JavaScript-rendered content that a plain HTTP request can't see. Practitioners call the work of handling all those variations "job wrapping," and it's as tedious as it sounds: one company's listing at a time, multiplied by every company that doesn't expose a public feed.

Major aggregators, Indeed, LinkedIn, Glassdoor, sit at the far end of the spectrum. They carry the largest volume of postings anywhere, and they defend that volume aggressively: rate limiting, bot detection, JavaScript rendering requirements, and active legal enforcement against large-scale scraping. They're the highest-risk target in the entire source hierarchy, and treating them as a last resort, approached carefully and only when the ATS and career-page tiers can't supply what's needed, is the practical read on where they fit in a pipeline.

Crawling at scale: the failure modes that matter for benchmark quality

The crawl layer is where a dataset quietly loses the postings nobody notices are missing, and that's a worse problem than a crawl that fails loudly. A build guide for salary benchmarking names two things that erode a benchmark faster than anything else: the per-page cost that caps how wide a sample can get, and missed postings that shrink the sample without telling anyone they're gone.

Zyte's job posting data page names the main mechanism behind that silent loss: job boards update their layouts constantly, and HTML parsers built against the old layout keep running without complaint, just returning empty or wrong values. Salary fields tend to be among the first things to break when that happens, since they're often buried in page structures that change the most between redesigns.

Three failure modes do most of the damage. Site redesigns can break a parser overnight with zero warning. Listings expire in the middle of a crawl, so a page that existed an hour ago now returns a dead link. And JavaScript-rendering requirements mean a plain HTTP request comes back with an empty shell of a page, technically a successful response, practically nothing inside it. Each one removes postings from the sample without raising any kind of alarm.

The fix on the operational side depends on what the pipeline already has. For a pipeline that already holds a list of posting URLs, submitting them in batches is more efficient than fetching one at a time. For a pipeline that has to go find those URLs first, list-crawling patterns are needed to surface them before any individual posting can be scraped.

A crawl that drops a quarter of its postings without logging the failure doesn't look broken. It looks complete. Percentile calculations run on that sample will skew toward whichever postings happened to render successfully, and nothing in the output will flag that the input was biased. Anti-bot countermeasures, CAPTCHAs, rate limiting, IP blocks, make this worse at any real scale, since platforms are actively defending against the kind of high-volume crawling a benchmark needs. The practical lesson is to log failures as aggressively as successes, because an invisible gap in the sample is far more dangerous than a visible one.

Extracting pay fields from unstructured postings using LLM-based parsing

Salary numbers almost never arrive in a clean labeled field. They show up as free text, buried three paragraphs into a job description, written in a dozen different formats: "$120k to $150k," "£45,000 per annum," "$55/hr," sometimes spelled out in words. Turning that mess into structured data used to mean writing regular expressions for every format anyone had ever seen and hoping the next posting didn't invent a new one. LLM-based extraction changes that math. A model can read the sentence the way a person would and map it onto a schema, rather than needing a hand-built pattern for every possible phrasing.

The practical version of this looks like sending the posting content to a scrape endpoint with a JSON schema attached, and letting the model return typed fields that match it. A build guide for salary benchmarking describes this as a managed feature: the model runs server-side, and there's nothing to host or operate on the pipeline's end. The schema being filled in is the same one defined back at the design stage: role title, seniority, location, remote status, currency, pay period, pay minimum, pay maximum, pay disclosed. That schema is the contract the extraction step has to satisfy.

That contract needs a gate, not just a hope. LLM-based extraction can hallucinate a field mapping or quietly skip a field it should have caught, even without anyone trying to trick it. A record that fails schema validation shouldn't get dropped silently, and it shouldn't get shoved into the dataset half-finished either. It should trigger a retry. Treating a failed validation as a signal to try again, rather than a reason to throw the record away, is what keeps the extraction stage from reintroducing the same silent data loss the crawl layer was just built to avoid.

Normalizing pay periods, currencies, and job titles into comparable units

Extraction hands over typed fields.

Three problems have to get solved on purpose, not left to chance. Pay period is the first: hourly and monthly figures need converting to one annualized base, using a consistent assumption such as a standard number of annual hours for hourly rates, and that assumption has to be stored next to the record so it can be recalculated later if the assumption changes. Currency is the second: every figure needs converting to a single reporting currency using an FX rate tied to a specific date and stored with the record, so that when the rate moves, the pipeline can recompute from the stored source data instead of re-crawling the entire posting all over again. A build guide for salary benchmarking lays out exactly this pattern. Keeping the raw extracted values next to the normalized ones is what makes that recompute possible, and it's both cheaper and easier to audit than re-running the whole pipeline from scratch.

Job title is the third problem, and it's the hardest one. Free-text titles need snapping onto a small, controlled taxonomy, and "AI Engineer," "Machine Learning Engineer," and "Applied AI Engineer" might describe the same job or three different ones. The pipeline has to make that call explicitly, rather than leaving three near-identical titles to quietly blur into each other once the aggregation step starts averaging them.

The normalization stage earns its keep here, because the strongest objection to benchmarking off of job postings is title ambiguity: the same job title can describe a mid-level engineer at a growth-stage startup and a staff-level engineer at a major AI lab, and without a seniority dimension, both land in the same bucket. A wide salary band under one normalized title is usually a sign the taxonomy needs a seniority split, the same way a suspiciously wide shoe size chart probably means someone forgot to separate men's and women's sizing.

Handled with that kind of discipline, title ambiguity flips from a weakness into an advantage over the old survey method. Job postings carry something most surveys never capture: the actual skills a role requires, written right there in the listing. That makes it possible to benchmark at the skills level instead of stopping at the title level, which is a resolution no self-reported survey response could ever offer. The objection that job titles are messy turns out to be the reason posting-based benchmarks can outperform the surveys they're replacing, provided the normalization stage does its job before the aggregation stage ever sees a number.