From Polling Loop to Event-Driven Feed Inside a Live Agentic Workflow

An event-driven web scraping API with webhook support should turn the page changes it detects into structured events, then send signed callbacks when the data is ready. The reference path is monitor → webhook → queue → agent, with an event log for recovery. Moving a live agentic workflow to this model means replacing scheduled checking with signals that agents can rely on, so agents spend tokens deciding what to do.

Why polling breaks inside agentic workflows specifically

An agent wakes up, spends tokens, checks for updates, finds nothing, and goes back to sleep. Then it does it again, with the dedication of someone refreshing a parcel-tracking page. Nothing changed, but the meter still ran. When the agent reasons on every scheduled check, idle time becomes an LLM bill rather than merely an empty response from a service.

The cost compounds when agents depend on one another. SAP describes business flows that move through order, payment, inventory, shipping, and email, where each scheduled handoff can add delay. An upstream agent may finish its work while its downstream partner keeps checking because it does not know the work is ready. By the time the next agent acts, its inputs may already be stale. As more agents and sources are added, those requests multiply even when nothing in the business has changed.

What event-driven architecture actually means for an agent

With event-driven architecture, agents move from ask-wait-ask to subscribe-decide-act. A meaningful state change becomes an immutable, timestamped record describing what happened. The agent subscribes to relevant event types and decides how to respond when one arrives. Until then, it spends no tokens checking. That is zero idle token cost, with infrastructure costs still to account for.

Three roles make the inversion work in practice. Producers, including external APIs, file systems, and web scrapers, emit events when state changes. The event bus sends those records to the agents that subscribe, and neither side needs to know how the other is built. Each consumer has a non-blocking local event loop so it can receive the next event even when processing is slow. A pipeline can therefore emit pipeline.completed when it finishes, rather than leave a downstream agent checking every 30 seconds.

Event schemas: defining the contract that agents reason over

The first implementation artifact is the event schema. Each event needs a name, timestamp, source, payload fields, and version so the receiving agent knows what happened and what context it has. For example, data.quality.alert can include an asset ID, quality score, and affected columns. That gives an agent enough information to assess severity and route the alert without querying the source again. The payload is part of the reasoning interface.

As producers change, the interface stays usable if it stays consistent. An agent should not have to guess whether fields described in different ways mean the same thing from one event to the next. Explicit versions make changes visible, while Confluent Schema Registry or AWS Glue Schema Registry can enforce contracts and prevent incompatible changes from reaching consumers. Full payloads further reduce ambiguity and follow-up work by including the whole resource. Thin payloads containing only an ID save context for later, but later means another fetch and more latency.

The event bus: decoupling agents from data sources at scale

The bus makes adding an agent a subscription change, leaving producers unchanged. A source emits the event once, and each interested consumer receives it on its own. This separation is the point of the infrastructure. If every producer still needs to know the details of every agent, the old coupling has just moved to a new address.

Broker choice depends on the workload. Apache Kafka fits high-throughput, durable workflows where replay and ordering guarantees are important. Apache Pulsar suits distributed enterprise deployments needing multi-tenancy and geo-replication, while AWS EventBridge offers a serverless option with lower operational overhead for teams already on AWS. Direct HTTP POST webhooks can be enough for straightforward workflows with manageable volume.

Delivery needs retries, deduplication, and burst handling, plus dead-letter queues and replay before the pipeline is production-safe. These provisions introduce operational work that a simple polling loop did not require. Distributed failures are also harder to follow, so event lag monitoring, delivery tracking, and consumer lag dashboards belong in the architecture from the start.

Cursor-based pull and webhook push: the two delivery modes and when to use each

Svix’s POKE architecture, short for Publish Once, Kickoff Everywhere, lets push and pull work from the same base. Tom Hacohen introduced it at Supabase Select 2026. Every relevant change is written once to an append-only event log per customer, and consumers either read from a cursor or receive signed webhooks. The transport can change without changing the event. That makes the two modes complementary rather than rival designs.

A cursor identifies the last event a consumer read. On reconnect, the consumer requests everything after it through one indexed range query. POKE contrasts this with checking an email status endpoint dozens of times, mostly receiving “nothing changed” and still discovering delivery late. A cursor request instead returns every change since the previous read, and full payloads provide the email context without another GET. This suits local, ephemeral, or intermittently connected agents.

Always-online agents can receive a signed HTTP POST as soon as an event enters the log. POKE uses the Standard Webhooks convention so consumers can check that a webhook is genuine. If a push is missed, the agent catches up from its cursor using the same event IDs and ordering. There is one important Postgres trap. A bigserial ID is assigned before commit, so concurrent writers can leave a late-committing row behind an advanced cursor unless inserts are serialized per customer with an advisory lock or IDs come from a single writer.

Four coordination patterns for multi-agent event-driven systems

Sequential work fits event chaining, where one agent’s output triggers the next. A research agent emits research.complete, a writing agent responds and emits draft.complete, and a review agent follows. Each handoff becomes discrete and auditable. When the tasks can run on their own, fan-out lets one asset.updated event trigger quality-check, lineage-refresh, and notification agents at the same time. The choice is whether work depends on the previous result or merely shares the same trigger.

Audit-critical work calls for event sourcing. An ordered event log preserves each state change so past state can be reconstructed through replay. Keeping a record of what agents decided in that history helps with governance and provides training data from past runs. It also gives an investigation something firmer than the latest database value and a hopeful shrug.

Long-running workflows fit saga orchestration when partial completion needs explicit handling. A coordinator emits sequential command events and uses compensating events to undo partial work after failures. This works for multi-agent operations that take seconds to hours, such as schema scanning, classification, and lineage work. Choreography is the decentralized variant, where agents react to one another’s outputs without a controller. It removes the need for central coordination but makes it harder to trace what the system does as a whole.

Migrating a live polling loop to an event-driven feed without a full rewrite

For a team migrating a running system, the practical path is incremental. Start with one resource and identify the state changes consumers currently poll for. Those changes become the event types. Define each schema with its name, timestamp, source, payload, and version, while leaving the existing polling endpoints available. Agents can then move one subscription at a time.

Next, add an append-only event log for each customer alongside the API already in place. A Postgres table is a valid starting point. Write each event in the same database transaction as the change it describes, so the resource update and its event are atomic. Before production traffic moves, address the cursor race with a per-customer advisory lock or a single writer for ID assignment. A log that quietly skips committed changes is not a migration milestone.

Expose a cursor endpoint using GET /events?after=:last_id, with filtering by event type. SSE or long-polling can serve low-latency consumers, while signed webhooks provide push delivery for always-online agents. Add bootstrap events for existing state next. Synthetic contact.created events, for example, let a new consumer build its view through the same stream, which also serves as its bulk-load path.

Before deprecating polling, add dead-letter handling, replay, and consumer lag monitoring. Track which agents consume which event types, and record their decisions as audit events. These controls keep the growing system legible. Unified.to describes a composition in which triggers detect changes, pass-through APIs supply current data, RAG retrieves knowledge, tool calling executes actions, and orchestration coordinates the workflow.

How do webpage changes trigger scraping and webhook notifications?

The web becomes an event source when a monitoring layer detects meaningful changes and emits structured records such as page.price_changed or page.content_updated. The agent subscribes to those records instead of fetching a competitor’s pricing page over and over. Detection still needs a monitoring mechanism. The architectural gain is that checking and extraction no longer require an agent to reason on every pass, and downstream work begins only when there is a signal.

A callback saying that requested data is ready does not, by itself, establish that a service watches pages for changes. Buyers should look for meaningful-change monitoring, structured payloads, signed webhook delivery, and a way to recover missed events. A monitor → webhook → queue → agent design separates those responsibilities. The monitor detects, the callback delivers, the queue absorbs bursts, and the agent decides what the change warrants.

Payload quality determines how much work remains after delivery. Raw HTML forces the agent to parse and interpret before acting, whereas clean Markdown, semantically structured HTML, or JSON gives it more usable context. A hybrid scraper can use an LLM to write deterministic extraction code once, run that code cheaply, and bring an agent back in when the site changes and the scraper needs to change with it. Schema.org JSON-LD can supply structured data directly. But redesigns and moves to client-side rendering can break extraction without warning, so values need to be checked against the expected types and ranges before downstream events are emitted.

What genuinely reactive agents look like in production

A reactive manufacturing workflow starts when a streaming pipeline detects a pressure sensor that is drifting. The pipeline assembles device history, maintenance schedules, and correlated sensor readings before invoking the agent. The agent opens a maintenance work order and notifies the on-call engineer within seconds. Context arrives with the trigger rather than becoming a scavenger hunt across APIs. Similarly, a customer-query agent with context already loaded through streaming change data capture can respond in under two seconds instead of first assembling it through separate calls.

The same model applies to business decisions. SAP describes how a large customer order triggers an inventory agent that checks stock, existing commitments, and replenishment schedules and may use demand forecasts and logistics data to decide whether and where to reorder. In competitive intelligence, the trigger can instead be a meaningful change in a competitor’s positioning, delivered with a structured brief. The agent receives a confirmed signal. Recording the resulting decisions as events also leaves a replayable trail for business stakeholders and regulators.

The extra machinery is justified by the reaction time the workflow needs. If a 30-second polling interval is fast enough for the work, distributed event flows may bring a debugging bill the team could avoid. Where timely action is essential, the completed migration produces a different operating model. An agent receives a specific change with context, acts, and emits the result for whichever consumer needs it next. Its work starts because something happened.