Back to all steps

Extract data

Step fields
{"container":".product-card","limit":100,"fields":[{"name":"Title","selector":"h3","attr":"text"},{"name":"Price","selector":".price","attr":"text"},{"name":"Link","selector":"a","attr":"href"}]} (text)
Collects structured data from the page: a list of similar cards (products, vacancies, listings) or a single object. The result appears on the “Data” tab of the run report, exports to CSV/JSON and is available over the API.
The parameter is JSON: {"container":".product-card","limit":100,"fields":[{"name":"Price","selector":".price","attr":"text"}]}
container is the selector of the repeating block, and field selectors are looked up inside it; an empty container collects a single object. If no card is found at all, the step fails - the selector is stale.
In the Visualizer you do not have to write the JSON by hand - use the “Pick fields on the page” button. The number of rows is capped by your plan. It spends no AI tokens.
The field picker, pagination ("append": true) and the limits are covered in the action help.
Usage example

Collect structured data from the page - a list of repeating cards or a single object. The result appears on the “Data” tab of the run report, can be exported to CSV/JSON and is available through the API. The parameter is JSON. The step spends no AI tokens or money.

A list of cards (the typical case):

{
  "container": ".product-card",
  "limit": 100,
  "fields": [
    { "name": "Title", "selector": "h3", "attr": "text" },
    { "name": "Price", "selector": ".price", "attr": "text" },
    { "name": "Link",  "selector": "a", "attr": "href" }
  ]
}

Field selectors are looked up inside the card, so they stay short (h3, not .product-card h3).

A single object (product page, order page):

{
  "fields": [
    { "name": "Product", "selector": "h1", "attr": "text" },
    { "name": "Price",   "selector": ".product-price", "attr": "text" },
    { "name": "In stock", "selector": ".stock-badge", "attr": "text" }
  ]
}

With no container, a single row is collected and selectors apply to the whole page.

How many rows you get (your plan). The row cap for one step is set by the plan: 10 rows on Free, Start and Pay-per-test (which is also plenty for single-object mode - there is only one row there), 100 on Pro, and no plan cap on the senior plans. The server trims the excess and writes the reason into the step log - “plan limit” - so a trimmed table does not read as a broken selector. Pagination is not a way around it: when accumulating, the cap applies to the total across all pages, not to a single one.

What you can take (attr):

  • text - visible text (default)
  • html - inner HTML of the element
  • href / src - link and image (absolute URL)
  • value - form field value
  • any attribute: data-id, title, alt

Behaviour: a field missing from a card stays empty and the card still goes into the dataset. If no card is found at all, the step fails - a sign that the selector is out of date.

Tip: put a “Wait for element” step with the card selector before this one if the listing is loaded by a script, otherwise the step captures an empty page.

The step also runs in preview/recorder, where it collects a shortened sample so you can see what will end up in the dataset.

In the Visualizer you do not have to write the JSON. Open the test in Visualizer mode and click “Pick fields on the page” inside the step:

  • click a product card - the step finds the container itself and shows how many such cards it found on the page;
  • clicks on the price, title and link build the field list; the column name is inferred from the content (Price, Title, Link);
  • the preview table shows what will actually land in the dataset, and columns are renamed right in its headers;
  • the JSON below the form updates itself - you never touch it.

Above the button sits a List / Single object switch. List - the page holds many identical blocks (a catalogue, a table, a feed), and the first click looks for the repeating block. Single object - you need one value or one card: no container is searched for, and every click adds a column straight away. Switch to Single object when auto-detection mistook a text highlight or a page-wide wrapper for a catalogue and reported dozens of “cards” holding a single value. Changing the mode resets the picking: in list mode field selectors are relative (inside the card), in single-object mode they are absolute.

The plain test editor has no such builder: there is no live page to click on there, so the parameter is entered as JSON.

Pagination: collect every page into one table. Add "append": true to the parameter and put the step inside a loop of a graph test:

Loop (while: “See element” .pagination .next, limit 50)
 └─ body:  Extract data  →  Click .pagination .next  →  Wait for .product-card
{
  "container": ".product-card",
  "append": true,
  "fields": [
    { "name": "Title", "selector": "h3", "attr": "text" },
    { "name": "Price", "selector": ".price", "attr": "text" }
  ]
}

With the flag, the snapshots of all iterations are merged into one dataset: one table in the report (labelled with the number of pages it was built from), one CSV export, one API response. Without the flag every iteration produces its own table - twenty pages arrive as twenty exports.

The overall row limit counts the accumulated total, and re-sending the same page after a network glitch does not duplicate rows. A failing step inside the loop stops the repeats, so a stale selector never turns into fifty empty iterations.

Change monitoring with pagination: put the “Compare with previous run” step AFTER the loop with an empty selector - it picks up the whole accumulated dataset. Inside the loop it would compare a page against the same-numbered page of the previous run, and a single new product would shift the entire listing by one position.