Tools
/
Crawler
/
APIs
/
Configuration
Jan. 17, 2022
Crawler: RequestOptions
Type:
object
Parameter syntax
requestOptions: {
proxy: 'http://proxy'
timeout: 10000
retries: 2
headers: {
...
}
}
About this parameter
Modify all crawler’s requests behavior.
The Crawler only accepts headers from our list of allowed headers.
The Cookie header will be replaced by the cookie fetched in login.
Examples
Copy
1
2
3
4
5
6
7
8
9
10
11
{
requestOptions: {
proxy: 'http://algolia:password@1.1.1.1:8080',
timeout: 10000,
retries: 2,
headers: {
Authorization: 'Bearer Aerehdf==',
Cookie: 'session=1234;',
}
}
}
Parameters
| Parameter | Description |
|---|---|
proxy
|
type: string (URL)
Optional
Set a proxy for your crawler’s requests. |
timeout
|
type: number
default: 30000
Optional
Set a timeout (in milliseconds) to your crawler’s requests. |
retries
|
type: number
default: 3
Optional
Set a maximum retry to your crawler’s requests. |
headers
|
type: Object
Optional
Specify headers to be sent in the crawler’s requests. Supported headers:
|
Did you find this page helpful?
