Skyvern: Browser Automation That Sees the Page Like You Do

Introduction

Most browser automation still feels like it’s stuck in 2015. You write selectors, you maintain them, and the moment a developer renames a CSS class or moves a button, your script breaks. I’ve been hearing about AI agents that can “see” the screen and click around for a while, but most were either closed demos or vaporware. Then Skyvern dropped as an open-source project that pairs browser automation with LLMs and computer vision. I spent a weekend putting it through its paces to see if it’s actually ready for real-world automation work.

The Problem

Traditional tools like Selenium and Playwright require you to define interactions using DOM selectors. That approach works until the page changes—and pages change constantly. Even modern “low-code” RPA tools eventually fall back on fragile element identifiers. AI‑powered agents that attempt to parse raw HTML and decide what to do next often choke on dynamically rendered UIs, shadow DOMs, or complex single‑page applications. They’re still drinking from the DOM firehose without any visual intuition.

What we need is something closer to how a human interacts with a webpage. Humans don’t inspect the DOM; we look at the screen, recognize buttons, text fields, and dropdowns, and then act. Skyvern tries to mimic that exact workflow. It sees the page, understands what it sees, and then drives the browser accordingly. That shift in approach solves the selector maintenance problem at its root.

The Solution

Skyvern is an open-source (Apache 2.0) browser automation engine that uses computer vision models and large language models to navigate web interfaces. Instead of relying on XPath or CSS selectors, it takes screenshots of the current viewport, sends them to a vision-capable LLM (like OpenAI’s GPT-4V or an open-source alternative via Ollama), and receives back a description of the relevant UI element along with its bounding box. It then converts that bounding box into click coordinates or keystrokes, all through a real browser controlled by Playwright.

Under the hood, Skyvern maintains a task plan and a memory of previous steps. It can correct itself if an action doesn’t produce the expected result—something you rarely get with static scripts. The entire system is self-hostable, which means you control where your screenshots go and which models you use. For automation engineers who handle sensitive internal tools, that’s a non‑negotiable advantage.

Implementation

Getting started is straightforward if you’re comfortable with Docker. The project provides a docker-compose.yml that spins up the web UI, the API, and a browser container. After cloning the repo, you set your OPENAI_API_KEY in the environment (or configure a local model endpoint) and run:

docker compose up -d

The UI comes up at http://localhost:8000. From there you can send tasks in plain English. For example:

Go to github.com, search for 'skyvern', open the repository, and star it.

Inside the box, Skyvern captures a viewport screenshot, sends it to the vision model, and gets back something like: “The search input is located at [x, y] with dimensions [w, h].” It then issues a Playwright click and types the query. Repeat for each subsequent step.

For integrations, there’s a REST API. Here’s a quick Python snippet that submits a task and polls for the result:

import requests, time
 
TASK = "Go to news.ycombinator.com and extract the top 5 article titles as a JSON list."
 
resp = requests.post(
    "http://localhost:8000/api/v1/tasks",
    json={"task": TASK, "max_steps": 10}
)
task_id = resp.json()["task_id"]
 
while True:
    status = requests.get(f"http://localhost:8000/api/v1/tasks/{task_id}").json()
    if status["status"] in ("completed", "failed"):
        break
    time.sleep(2)
 
print(status["output"])

That script reliably returned a clean JSON array of titles without me writing a single selector. Under the hood it took 8 steps, including scrolling and parsing the rendered list.

If you want to avoid sending screenshots to OpenAI, Skyvern supports Ollama and LLaVA-style local models. I tested both paths. Local models lower cost and keep data on‑prem, but their visual reasoning isn’t yet as sharp as GPT-4V on complex layouts. For internal admin panels, however, it’s often good enough.

Results

I threw three real-world scenarios at Skyvern:

1. Invoice download from a legacy supplier portal – This site uses a shadow-DOM-heavy React interface that broke my old Selenium script every other week. Skyvern logged in, navigated to the billing section, and downloaded the latest invoice without any element‑specific code. Time: 2 minutes 14 seconds. Cost: $0.32 using GPT-4V.

2. Hacker News data extraction – Asking it to grab top posts and their scores worked with 100% accuracy across five runs. The task completed in roughly one minute each time.

3. Captcha‑protected login – As expected, it couldn’t solve the captcha automatically. It did recognize the captcha element and paused, waiting for human intervention (which the API supports via a “wait for user” action). That’s a realistic feature, not a failure.

Throughput isn’t blazing fast. Every action takes 4–7 seconds because of the LLM inference round‑trip. That’s fine for tasks where correctness matters more than speed—like invoice processing or CI‑based end‑to‑end tests that run nightly. For high‑frequency scraping, I’d still reach for Playwright with robust selectors. But for the long‑tail of unpredictable, rarely‑automated chores, this is a game‑changer.

Key Takeaways

• Visual understanding kills selector maintenance – No more chasing CSS class changes. Skyvern locates elements the same way a human does, drastically reducing breakage over time.
• LLM overhead is the main trade‑off – Each decision costs tokens and adds latency. Use it where reliability trumps speed; combine it with traditional methods for hot paths.
• Self‑hosted and open‑source means data stays yours – You can run entirely on‑prem with local models, which is essential for finance, healthcare, or any regulated environment.
• It shines on complex, dynamic interfaces – Shadow DOM, canvas elements, and heavily nested components are no longer obstacles.
• Early, but already useful – LLM‑powered vision still stumbles on very crowded UIs or when actions require precise drag‑and‑drop. Expect to build custom “skill” fallbacks for edge cases, but the foundation is solid.