Scrape domains
Update
Updates the crawl configuration of one scrape domain (sitemap_url, include_paths, exclude_paths, primary_language). domain_name cannot be changed after creation because every scraped page is keyed on its URL, so sending a different one fails the call. Changing any of those crawl settings makes the domain eligible for a fresh seeding and queues background tasks that re-apply the path rules to the pages already scraped, none of which has happened when the response returns.
PUT
/api/{tenant:minlength(2)}/v{version:apiVersion}/scrape_domains/{id}
Request arguments
| Name | Type | Source |
|---|---|---|
| id | Int32 | |
| input | DTO_scrape_domain | Body |
Request body example
Response
Response body example
DTO_scrape_domain
| Name | Type | ReadOnly | Description |
|---|---|---|---|
| created_by | DTO_reference_user | ReadOnly | Created by |
| last_updated_by | DTO_reference_user | ReadOnly | Last updated by |
| id | Int32 | ReadOnly | Leave empty on input |
| domain_name | String | Domain name | |
| status | ENUM scrape_domain_status | ReadOnly | Status |
| created | DateTime | ReadOnly | Created |
| scraped_count | Int32 | ReadOnly | Scraped count |
| last_updated | DateTime | ReadOnly | Last updated |
| verified | DateTime (nullable) | ReadOnly | Verified |
| sitemap_url | String | Optional custom sitemap location. Either a path ("/sitemap_index.xml") or a full URL. Empty = /sitemap.xml | |
| include_paths | String | Newline-separated path patterns to focus the crawl on (e.g. "/legal*"). Empty = whole site | |
| exclude_paths | String | Newline-separated path patterns to skip (e.g. "/sv/*"). Wins over include_paths | |
| primary_language | String | Primary content language to keep when a site publishes the same page in several languages. Empty = auto-detected from the site | |
| detected_language | String | ReadOnly | Detected language |
| last_scraped | DateTime (nullable) | ReadOnly | Most recent successful page crawl on the domain |
| ignored_count | Int32 | ReadOnly | Pages ignored as language duplicates or by path rules |
| error_count | Int32 | ReadOnly | Error count |
| plan_pages_used | Int32 | ReadOnly | Scraped pages counted against the tenant plan this month |
| plan_pages_max | Int32 | ReadOnly | The tenant plan's monthly scraped-pages limit |
| pending_count | Int32 | ReadOnly | Pages waiting in the crawl queue |
DTO_reference_user
| Name | Type | ReadOnly | Description |
|---|---|---|---|
| profile_pic | String | ReadOnly | Profile pic |
| String | ReadOnly | ||
| id | Int32 | Id | |
| label | Object | ReadOnly | Label |
ENUM scrape_domain_status
| Value | Name | Description |
|---|---|---|
| 0 | UNKNOWN | UNKNOWN |
| 1 | PENDING | PENDING |
| 2 | VERIFIED | VERIFIED |
Generated Aug 4, 2026 18:37