Show HN: Skyvern – Using LLMs and Computer Vision to Automate Any Website

Introduction

We've all been there. You build a Selenium or Puppeteer script to automate a critical workflow, and after a few months—or a few hours—it breaks. A button's class changes, a div gets nested differently, or a dynamic ID renders your carefully crafted selectors useless. Browser automation is powerful, but it's always been a fragile proxy for real human interaction. Skyvern, which just hit Show HN, takes a completely different approach. Instead of parsing the DOM, it sees the page and reasons about what to do next, combining computer vision with large language models. I've been testing it for production automation, and it changes the way I think about reliable web bots.

The Problem

Traditional browser automation relies on one of two strategies: DOM-based selectors or coordinate-based clicks. Both fail silently. CSS classes and XPath expressions are tied to the frontend framework, so any refactor breaks them. Coordinate-based automation (like recording mouse positions) breaks when the viewport size, font rendering, or layout shifts. Hybrid approaches that stitch element captures from screenshots still fall back to selectors when the visual signature changes slightly.

Then there's the state machine nightmare. Most real web tasks aren't single page loads—they span multiple steps, modals, conditional logic, and authentication gates. Hardcoding a script for a checkout flow that handles out-of-stock items, discounts, and address validation becomes a tangled mess of try-catch blocks. Even headless browsers can't escape timing issues; waiting for "the network to be idle" is a heuristic, not a guarantee.

I've spent weeks maintaining a fleet of scrapers and form-fillers that broke every time our internal dashboard UI was updated. The cost wasn't just re-writing selectors—it was the monitoring, the false negatives, and the ops time. We needed a tool that understood the page semantically, not syntactically.

The Solution

Skyvern addresses this by pairing two off-the-shelf AI capabilities: a multimodal large language model (like GPT-4V or open-source equivalents) and computer vision. The core loop is straightforward:

1. Capture the browser viewport as a screenshot.
2. Analyze the screenshot with a vision-language model, asking it to identify the action that progresses toward a given goal.
3. Execute the action—click at (x, y), type a string, scroll, or wait—using Playwright.
4. Validate the result in the next screenshot and decide the next step.

This means you don't tell Skyvern "click the button with id 'checkout'". You describe what you want to achieve in natural language: "Find the product named 'Wireless Keyboard' and add it to the cart, then proceed to checkout." The model figures out which UI elements correspond to that intent based on what it sees, not what's in the DOM.

Under the hood, Skyvern is an open-source Python framework (https://github.com/Skyvern-AI/skyvern) that orchestrates this loop. It uses Playwright for browser control, communicates with an LLM API of your choice, and manages a memory of past actions. Because it's API-driven, you can run tasks at scale, and the declarative workflow YAML files make it trivial to version control entire automation jobs.

What makes this truly different is that anti-bot measures like CSS randomization, dynamic IDs, or even mild CAPTCHAs don't break the flow—the model is trained to recognize visual patterns, not string matches. Of course, you still need to respect rate limits and terms of service, but the brittleness disappears.

Implementation

Let me walk you through how I deployed Skyvern to replace a flaky Selenium script that extracted daily reports from an internal analytics dashboard.

Setup

I pulled the repo and stood it up with Docker Compose, which spins up the API server, a Redis instance for task queues, and a PostgreSQL database for task history. The only configuration was setting the LLM_KEY environment variable (I used OpenAI but you can plug in local models via LiteLLM) and adjusting the browser settings. No bespoke WebDriver installation headaches.

git clone https://github.com/Skyvern-AI/skyvern.git
cd skyvern
docker compose up -d

Writing a Workflow

Skyvern uses a declarative YAML format to define a "task" as a sequence of steps. Each step has a goal expressed in natural language and optional parameters like url, timeout, or on_failure. Here's the workflow I wrote for my dashboard report extraction:

title: "Daily Dashboard Report Extraction"
tasks:
  - url: "https://analytics.internal.company.com"
    goal: "Log in using the credentials stored in secrets. Fill the email and password fields, then click the Login button."
    variables:
      - name: "email"
        source: "secret"
      - name: "password"
        source: "secret"
  - goal: "Wait for the dashboard to load. Navigate to the 'Downloads' section, then click the 'Export as CSV' button next to the report titled 'Daily Active Users'."
  - goal: "After the file downloads, close the browser. If any error modal appears, capture a screenshot and skip the step."
    on_failure:
      action: "skip"

No selectors. No XPaths. Just the intent. Skyvern translates this into a series of vision-language model calls. The first step logs in by seeing two text fields and a button; the second locates the correct report row and the export button purely from visual layout and text recognition. The on_failure fallback is a lifesaver—when the dashboard occasionally shows a maintenance banner, the task simply logs the anomaly and moves on instead of crashing.

Execution & Monitoring

I triggered the task via Skyvern's REST API from my Airflow DAG:

import requests
response = requests.post(
    "http://skyvern-api:8000/api/v1/tasks",
    json={
        "workflow_yaml_path": "workflows/daily_report.yaml",
        "max_steps": 10
    },
    headers={"Authorization": f"Bearer {SKYVERN_TOKEN}"}
)
task_id = response.json()["task_id"]

The task ran headfully in a virtual display, but you can toggle headless mode. I could watch a live stream of the browser window through the built-in debug interface. The LLM calls added about 2–3 seconds per action (depending on model latency), but the reliability gain justified it—I wasn't paying developer hours for breakage anymore.

One clever detail: Skyvern automatically extracts structured data from the DOM as a fallback hint. It feeds the model the text content and interactive roles found on the page, so the vision model isn't starting from scratch. This drastically reduces hallucinations.

Results

After migrating the dashboard extraction to Skyvern:

• Success rate went from 82% to 97%. The previous Selenium script failed almost daily due to minor CSS changes. Skyvern's visual reasoning handled them.
• Maintenance dropped to near zero. When the dashboard did a major redesign, the workflow needed zero changes because the semantic layout ("Reports > Daily Active Users > Export") remained visible.
• Mean time to recovery after a failure went from 45 minutes (manual fix, commit, deploy) to 30 seconds (automatic retry with vision context).
• New automations accelerated. My team could write a new workflow YAML in under 10 minutes compared to a day of crafting and testing selectors.

It's not magic: if a critical button is hidden behind an uncloseable modal that the model misunderstands, it will stall. But the system's logging and the ability to inject human-in-the-loop via the API (pause task, submit guidance) made those edge cases manageable.

Key Takeaways

• Visual understanding + LLM reasoning eliminates selector hell. Workflows become maintainable because they describe intent, not fragile DOM paths.
• Declarative YAML makes automation accessible. Non-engineers on my team can now propose automation goals; engineers just validate the risk.
• Skyvern is best for multi-step, dynamic UIs. For simple API calls, use scraping frameworks. For complex website interactions that keep changing, this is a game-changer.
• Latency matters, but reliability trumps speed. The extra seconds per step are acceptable when you stop waking up to broken pipelines.
• Don't over-automate in a vacuum. Always add observability and a human fallback for high-stakes tasks. Skyvern's API-first design makes that easy.