Content

Scrape domains / List

Returns all domains to scrape.

Route: /api/{tenant:minlength(2)}/v{version:apiVersion}/scrape_domains

Method: GET

Authorisation: Minimum role: USER

Name	Type	Source

Request body example

No input arguments

Response status: 200 (OK)

Other response statuses: 400 (BadRequest), 403 (Forbidden)

Response body example

Name	Type	ReadOnly	Description
created_by	DTO_reference_user	ReadOnly	Created by
last_updated_by	DTO_reference_user	ReadOnly	Last updated by
id	Int32	ReadOnly	Leave empty on input
domain_name	String		Domain name
status	ENUM scrape_domain_status	ReadOnly	Status
created	DateTime	ReadOnly	Created
scraped_count	Int32	ReadOnly	Scraped count
last_updated	DateTime	ReadOnly	Last updated
verified	DateTime (nullable)	ReadOnly	Verified
sitemap_url	String		Optional custom sitemap location. Either a path ("/sitemap_index.xml") or a full URL. Empty = /sitemap.xml
include_paths	String		Newline-separated path patterns to focus the crawl on (e.g. "/legal*"). Empty = whole site
exclude_paths	String		Newline-separated path patterns to skip (e.g. "/sv/*"). Wins over include_paths
primary_language	String		Primary content language to keep when a site publishes the same page in several languages. Empty = auto-detected from the site
detected_language	String	ReadOnly	Detected language
last_scraped	DateTime (nullable)	ReadOnly	Most recent successful page crawl on the domain
ignored_count	Int32	ReadOnly	Pages ignored as language duplicates or by path rules
error_count	Int32	ReadOnly	Error count
plan_pages_used	Int32	ReadOnly	Scraped pages counted against the tenant plan this month
plan_pages_max	Int32	ReadOnly	The tenant plan's monthly scraped-pages limit
pending_count	Int32	ReadOnly	Pages waiting in the crawl queue

Name	Type	ReadOnly	Description
profile_pic	String	ReadOnly	Profile pic
email	String	ReadOnly	Email
id	Int32		Id
label	Object	ReadOnly	Label

PUT Scrape domains / Update

GET Scrape domains / Get one