Crawler: Login

About this parameter

This property defines how the crawler acquires a session to access protected content.

The crawler supports multiple ways to authenticate to protected websites, such as:

• Basic auth through an HTTP request
• Basic auth by visiting a login page through a web browser and sending the login form like a human would do
• OAuth 2.0

Basic auth

The crawler extracts the Set-Cookie response header from the login page, stores that cookie and sends it in a Cookie header when crawling all pages of the website defined in the configuration.

This cookie is only fetched at the beginning of each complete crawl. If it expires, we won’t renew it automatically.

There are two ways the crawler can interact with your login page:

• By doing a direct request with the credentials to your login endpoint, like a standard curl command.
• By emulating a web browser, loading your login page, entering the credentials and validating the login form.

OAuth 2.0

The crawler supports OAuth 2.0 Client Credentials Grant flow. It performs an Access Token Request using the provided credentials, stores the fetched token in an Authorization header and sends it when crawling all pages of the website that are defined in the configuration.

This token is only fetched at the beginning of each complete crawl. If it expires, it won’t be renewed automatically.

Client authentication is performed by passing the client credentials (client_id / client_secret) in the request-body as described in the RFC.

The following providers are supported. You can reach out if you need to add others.

• Azure AD v1.0

Examples

1
2
3
4
5
6
7
8
9
10
11
12
13
{
  login: {
    fetchRequest: {
      url: 'https://example.com/secure/login-with-post',
      requestOptions: {
        method: 'POST',
        headers: { 'Content-Type': 'application/x-www-form-urlencoded' },
        body: 'id=my-id&password=my-password',
        timeout: 5000 // in milliseconds
      }
    }
  }
}

1
2
3
4
5
6
7
8
9
  {
    login: {
      browserRequest: {
        url: 'https://example.com/secure/login-page',
        username: 'my-id',
        password: 'my-password',
      }
    }
  }

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
  {
    login: {
      oauthRequest: {
        accessTokenRequest: {
          url: 'https://example.com/oauth2/token',
          grant_type: 'client_credentials',
          client_id: 'my-client-id',
          client_secret: 'my-client-secret',
          extraParameters: {
            resource: 'https://protected.example.com/'
          }
        }
      }
    }
  }