Netlify FAQ

This is a general purpose FAQ. If you are looking for more technical answers about the Algolia Crawler, please refer to the Crawler documentation.

How to extract more information from pages?

While the Algolia Crawler offers extended record customization, the plugin provides a single, default extraction strategy.

If you think something is missing, or you’d like to have more control, please share your feedback in the Algolia Discourse forum.

How to build a custom user interface?

The easiest solution is to use InstantSearch, Algolia’s open source, production-ready family of UI libraries that let you build a search interface in your frontend app. InstantSearch is available for vanilla JS, React, Vue, and Angular.

How to get a notification when the crawl is done?

You can configure notifications in the account settings of your Crawler Admin Console.

There’s an issue with the crawl

The Monitoring tab in the Crawler UI offers information to help you improve the crawler behavior. Keep in mind that, by default, the Algolia Crawler respects most of the standard web rules: robots.txt, password protection, canonical URLs, redirects, etc.

You can help the crawler by:

• Allowing the Algolia user agent in your robots.txt file.
• Specifying relative URLs in your links and canonical URLs instead of absolute URLs that could conflict with your domain.

How to handle custom domains?

The plugin can automatically alias your custom domain:

• The Algolia Crawler automatically aliases custom domains that you configured in Netlify (Settings > Domain management > Custom domains).
• If this is impossible to configure in Netlify, a customDomain option is available in your netlify.toml file.

How to crawl a website not at the root level?

The plugin can automatically alias paths (for example, by removing a path prefix) by setting the option pathPrefix in your netlify.toml file.

Is displaying the Algolia logo required?

Yes, if you are on a Free plan. Please see the complete answer for more details.

As an Algolia paying user, can I use my own app?

Yes, but it requires manual intervention. Contact the Algolia support team to implement the change.