Autonomous Competitor Tracking: Build an AI Monitoring Agent

TL;DR / Quick Summary & Key Takeaways

Manually checking competitor pricing pages every week doesn’t scale past two or three competitors — and by the time a human notices a price drop, the sales conversation is already stale. This guide builds a fully autonomous monitoring agent: it scrapes target pages on a schedule, diffs both the raw HTML/text and a rendered screenshot against the last snapshot, uses an LLM to summarize what actually changed and why it matters (not just a raw diff), and pushes a digestible weekly report straight to Slack or Discord.

• Three-layer detection: text diff (cheap, catches copy/pricing changes) + visual diff (catches layout/design changes text-diffing misses) + LLM synthesis (turns raw diffs into a human-readable “what changed and why it matters” summary).
• Store snapshots, not just diffs — you need historical state to diff against, and a snapshot archive lets you answer “when did they last change pricing” months later.
• Use Claude for the synthesis step, not the scraping step — LLMs are expensive and unnecessary for raw HTML extraction; reserve them for the semantic comparison where judgment actually matters.
• Respect robots.txt and rate limits. This is a monitoring tool, not a scraping-at-scale operation — target 5-20 competitor pages, checked daily or weekly, not thousands of pages hammered continuously.
• Visual diffing catches what text diffing misses: a repositioned CTA button, a new badge (“Now with AI!”), or a redesigned pricing table often carries more competitive signal than the text itself.
• Weekly digest beats real-time alerts for most use cases — a Slack channel that pings on every whitespace change trains your team to ignore it. Batch, synthesize, then notify.

---

Core Technical / Conceptual Deep Dive

The three-layer change-detection architecture

┌──────────────┐     ┌───────────────┐     ┌──────────────────┐
│ 1. Scrape    │ --> │ 2. Diff        │ --> │ 3. Synthesize     │
│ (Playwright) │     │ (text + visual)│     │ (Claude API)      │
└──────────────┘     └───────────────┘     └──────────────────┘
       │                     │                       │
   HTML + PNG          Raw diff output          Human-readable
   snapshot            (line changes,            summary + severity
                        pixel diff %)             score

Each layer has a distinct job. The scraper’s only responsibility is capturing state reliably — page text, key selectors (pricing blocks, headlines), and a full-page screenshot. The diff layer is pure computation: no LLM involved, just comparing this week’s snapshot to last week’s. The synthesis layer is where an LLM earns its cost — turning “line 47 changed from ‘$49/mo’ to ‘$39/mo’” into “Competitor X dropped their Pro tier price by 20%, likely a response to the recent influx of budget-tier reviews.”

Why not just diff raw HTML?

Raw HTML diffs are noisy — a change to a Google Analytics tracking ID or a rotating testimonial carousel produces diff noise identical in size to an actual pricing change. The fix is targeting specific selectors (pricing blocks, headline text, feature lists) rather than diffing entire page source, plus a visual diff as a catch-all for anything selector-based extraction misses.

---

Practical Tutorial / Step-by-Step Implementation

Step 1: Define your competitor watchlist and target selectors

# watchlist.py
WATCHLIST = [
    {
        "name": "Competitor A",
        "url": "https://competitor-a.com/pricing",
        "selectors": {
            "pricing": ".pricing-table",
            "headline": "h1",
        },
    },
    {
        "name": "Competitor B",
        "url": "https://competitor-b.com/pricing",
        "selectors": {
            "pricing": "[data-testid='pricing-grid']",
            "headline": "h1",
        },
    },
]

Step 2: Capture a snapshot (text + screenshot) with Playwright

from playwright.sync_api import sync_playwright
import hashlib
import json
from datetime import datetime, timezone
from pathlib import Path

SNAPSHOT_DIR = Path("snapshots")
SNAPSHOT_DIR.mkdir(exist_ok=True)

def capture_snapshot(target: dict) -> dict:
    with sync_playwright() as p:
        browser = p.chromium.launch()
        page = browser.new_page(viewport={"width": 1440, "height": 900})
        page.goto(target["url"], wait_until="networkidle", timeout=30000)

        extracted = {}
        for key, selector in target["selectors"].items():
            try:
                extracted[key] = page.locator(selector).inner_text(timeout=5000)
            except Exception:
                extracted[key] = None

        timestamp = datetime.now(timezone.utc).strftime("%Y%m%d_%H%M%S")
        screenshot_path = SNAPSHOT_DIR / f"{target['name']}_{timestamp}.png"
        page.screenshot(path=str(screenshot_path), full_page=True)

        browser.close()

    return {
        "name": target["name"],
        "url": target["url"],
        "timestamp": timestamp,
        "extracted_text": extracted,
        "screenshot_path": str(screenshot_path),
        "content_hash": hashlib.sha256(
            json.dumps(extracted, sort_keys=True).encode()
        ).hexdigest(),
    }

Step 3: Compare against the last stored snapshot

def load_last_snapshot(name: str) -> dict | None:
    history_file = SNAPSHOT_DIR / f"{name}_history.json"
    if not history_file.exists():
        return None
    entries = json.loads(history_file.read_text())
    return entries[-1] if entries else None

def save_snapshot(snapshot: dict):
    history_file = SNAPSHOT_DIR / f"{snapshot['name']}_history.json"
    entries = json.loads(history_file.read_text()) if history_file.exists() else []
    entries.append(snapshot)
    history_file.write_text(json.dumps(entries, indent=2))

def text_changed(previous: dict, current: dict) -> bool:
    return previous["content_hash"] != current["content_hash"]

Step 4: Visual diff with Pillow (pixel-level comparison)

from PIL import Image, ImageChops

def visual_diff_percent(old_path: str, new_path: str) -> float:
    old_img = Image.open(old_path).convert("RGB")
    new_img = Image.open(new_path).convert("RGB")

    # Resize to match if pages have slightly different heights
    if old_img.size != new_img.size:
        new_img = new_img.resize(old_img.size)

    diff = ImageChops.difference(old_img, new_img)
    diff_pixels = sum(1 for px in diff.getdata() if px != (0, 0, 0))
    total_pixels = old_img.size[0] * old_img.size[1]
    return round((diff_pixels / total_pixels) * 100, 2)

Step 5: Synthesize the change with Claude

This is the step that turns raw diff data into an actionable insight — only run it when a change was actually detected, to control cost.

import anthropic

client = anthropic.Anthropic()

def synthesize_change(name: str, old_text: dict, new_text: dict, visual_diff_pct: float) -> str:
    prompt = f"""You are a competitive intelligence analyst. Compare the following
before/after page extracts for {name} and summarize what changed and why it
might matter for a sales/marketing team. Be concise (3-4 sentences max).
If the change looks cosmetic/irrelevant, say so directly.

BEFORE:
{json.dumps(old_text, indent=2)}

AFTER:
{json.dumps(new_text, indent=2)}

Visual difference: {visual_diff_pct}% of page pixels changed.
"""
    response = client.messages.create(
        model="claude-sonnet-5",
        max_tokens=300,
        messages=[{"role": "user", "content": prompt}],
    )
    return response.content[0].text

Step 6: Push the weekly digest to Slack

import requests

SLACK_WEBHOOK_URL = "https://hooks.slack.com/services/YOUR/WEBHOOK/URL"

def send_slack_digest(changes: list[dict]):
    if not changes:
        text = "No competitor changes detected this week."
    else:
        blocks = ["*Weekly Competitor Intelligence Digest*\n"]
        for c in changes:
            blocks.append(
                f"*{c['name']}* — {c['visual_diff_pct']}% visual change\n"
                f"{c['summary']}\n<{c['url']}|View page>\n"
            )
        text = "\n".join(blocks)

    requests.post(SLACK_WEBHOOK_URL, json={"text": text}, timeout=10)

Step 7: Orchestrate the full run and schedule it

def run_monitoring_cycle():
    changes_this_run = []

    for target in WATCHLIST:
        current = capture_snapshot(target)
        previous = load_last_snapshot(target["name"])

        if previous is None:
            save_snapshot(current)
            continue  # first run, nothing to diff against yet

        if text_changed(previous, current):
            visual_pct = visual_diff_percent(
                previous["screenshot_path"], current["screenshot_path"]
            )
            summary = synthesize_change(
                target["name"], previous["extracted_text"],
                current["extracted_text"], visual_pct
            )
            changes_this_run.append({
                "name": target["name"],
                "url": target["url"],
                "visual_diff_pct": visual_pct,
                "summary": summary,
            })

        save_snapshot(current)

    send_slack_digest(changes_this_run)

if __name__ == "__main__":
    run_monitoring_cycle()

Schedule with cron for a weekly digest:

# Run every Monday at 8am
0 8 * * 1 /usr/bin/python3 /path/to/monitoring_cycle.py >> /var/log/competitor-monitor.log 2>&1

---

Tool / Solution Comparison Table

Component	Recommended	Alternative	Notes
Page rendering/scraping	Playwright	Puppeteer, Selenium	Playwright handles JS-heavy pricing pages most reliably
Text diffing	Selector-targeted extraction + hash compare	Full-HTML diff (difflib)	Selector targeting avoids noise from ads/trackers/carousels
Visual diffing	Pillow (ImageChops)	pixelmatch, resemble.js	Pillow is sufficient for percentage-level change detection
Change synthesis	Claude API	GPT-4o	Prompt-tunable severity/tone; cheap since only run on detected changes
Notification	Slack Incoming Webhook	Discord Webhook, email	Both Slack/Discord webhooks are a single requests.post() call
Scheduling	Cron	GitHub Actions scheduled workflow, n8n	Cron is simplest for a single-server deployment

---

Actionable Checklist / Next Steps

• Start with 5-10 competitors max and specific, stable CSS selectors — broad full-page scraping produces too much diff noise to be useful.
• Check robots.txt and rate-limit your requests — this is a monitoring tool, not a scraper at scale; one visit per target per check cycle is enough.
• Store every snapshot, not just the latest — historical pricing/copy data becomes valuable for trend analysis months later.
• Only call the LLM when a change is actually detected — gating synthesis behind the diff check keeps costs proportional to actual signal.
• Tune your visual-diff threshold — a 100% pixel-identical page with 0.3% diff (font rendering noise) shouldn’t trigger a report; set a minimum threshold (e.g., 2%) before flagging.
• Route the weekly digest to a dedicated Slack channel, not a busy general channel — keeps the signal visible instead of buried.
• Revisit selectors quarterly — competitor site redesigns will silently break selector-based extraction; add a fallback alert if a selector returns empty for 2+ consecutive runs.