Scrape domains
Create
Registers a website domain whose pages are crawled and used as AI answer material. Any http:// or https:// prefix is stripped from domain_name, public email provider domains and already registered domains are refused, an absolute sitemap_url must stay on the same domain, and the plan's limit on the number of scrape domains applies. The domain is stored as verified without any ownership check, and crawling has not started when the response returns: this call only publishes the scrape init task that seeds the root page and the sitemap.
POST
/api/{tenant:minlength(2)}/v{version:apiVersion}/scrape_domains
Request arguments
| Name | Type | Source |
|---|---|---|
| input | DTO_scrape_domain | Body |
Request body example
Response
Response body example
DTO_scrape_domain
| Name | Type | ReadOnly | Description |
|---|---|---|---|
| created_by | DTO_reference_user | ReadOnly | Created by |
| last_updated_by | DTO_reference_user | ReadOnly | Last updated by |
| id | Int32 | ReadOnly | Leave empty on input |
| domain_name | String | Domain name | |
| status | ENUM scrape_domain_status | ReadOnly | Status |
| created | DateTime | ReadOnly | Created |
| scraped_count | Int32 | ReadOnly | Scraped count |
| last_updated | DateTime | ReadOnly | Last updated |
| verified | DateTime (nullable) | ReadOnly | Verified |
| sitemap_url | String | Optional custom sitemap location. Either a path ("/sitemap_index.xml") or a full URL. Empty = /sitemap.xml | |
| include_paths | String | Newline-separated path patterns to focus the crawl on (e.g. "/legal*"). Empty = whole site | |
| exclude_paths | String | Newline-separated path patterns to skip (e.g. "/sv/*"). Wins over include_paths | |
| primary_language | String | Primary content language to keep when a site publishes the same page in several languages. Empty = auto-detected from the site | |
| detected_language | String | ReadOnly | Detected language |
| last_scraped | DateTime (nullable) | ReadOnly | Most recent successful page crawl on the domain |
| ignored_count | Int32 | ReadOnly | Pages ignored as language duplicates or by path rules |
| error_count | Int32 | ReadOnly | Error count |
| plan_pages_used | Int32 | ReadOnly | Scraped pages counted against the tenant plan this month |
| plan_pages_max | Int32 | ReadOnly | The tenant plan's monthly scraped-pages limit |
| pending_count | Int32 | ReadOnly | Pages waiting in the crawl queue |
DTO_reference_user
| Name | Type | ReadOnly | Description |
|---|---|---|---|
| profile_pic | String | ReadOnly | Profile pic |
| String | ReadOnly | ||
| id | Int32 | Id | |
| label | Object | ReadOnly | Label |
ENUM scrape_domain_status
| Value | Name | Description |
|---|---|---|
| 0 | UNKNOWN | UNKNOWN |
| 1 | PENDING | PENDING |
| 2 | VERIFIED | VERIFIED |
Generated Aug 4, 2026 18:37