API reference / A/B Testing API

Mar. 01, 2023

A/B Testing API

A/B testing enables seamless relevance tuning experimentation by measuring the impact they have on the end users.

Note: each endpoint targets a single index.

Domain

The Analytics API and A/B test API can be reached from multiple domains, each specific to a region. You should use the domain that matches the region where your analytics data is stored and processed. View your analytics region.

The following domains are available:

United States: https://analytics.us.algolia.com
Europe (Germany): https://analytics.de.algolia.com

Note: https://analytics.algolia.com is an alias of https://analytics.us.algolia.com.

Common parameters

Every endpoint accepts the following set of parameters:

X-Algolia-Application-Id: The appID. Can also be passed through HTTP headers.
X-Algolia-API-Key: Auth token. Can also be passed through HTTP headers.

ACL

For adding, stopping, and deleting A/B tests, the X-Algolia-API-Key needs to have the editSettings ACL.

For getting and listing A/B tests, X-Algolia-API-Key needs to have the analytics ACL.

Usage notes

Some important considerations to keep in mind.

You will need to have both analytics and clickAnalytics enabled on your account.
Currently the feature is limited to 2 concurrent variants (A and B). You can’t target the same index in the 2 variants.
The engine will spread traffic for both variants depending on the A/B test configuration, so there is no modification to perform on the frontend.
The ordering in which variants are defined is important; the first variant index will be the “control index” meaning that when targeting this index, some traffic will be automatically redirected to the other variant index (how often is determined by the allotment percentage). However, when targeting the second variant index, no redirect will happen.
While running a test, the Analytics Dashboard will show the combined metrics for both indices (cf previous point). However, this doesn’t fully make sense for some metrics (like no results). We might iterate on this in the future.
We do not provide a way to validate that the UX will work with both variants for now so users should handle this on their side. We therefore recommend running your B index in your current UI before starting the A/B test
A/B test analytics will not provide a daily breakdown but just a global view. A daily breakdown is actually not very useful when comparing the variants.
The full A/B test history will be kept until the app is deleted or a test is explicitly deleted.
The ABTest status might be:
- active: the Analytics created the A/B test and performed a successful request to the engine
- stopped: the A/B test was stopped by a user: it was deleted from the engine but you have to keep the data for historical purposes
- expired: the A/B test reached its end date and was automatically stopped. It is removed from the engine but the metadata/metrics are kept
- failed: the A/B test creation failed

Validation error

In case of error, the API will return a payload in the following format:

Copy

1
2
3
4
{
  "status": 123,
  "message":"The error message"
}

A/B test endpoints

Quick reference

Verb	Path	Method
POST	/2/abtests	Add A/B test
GET	/2/abtests/{id}	Get A/B test
GET	/2/abtests	List A/B tests
POST	/2/abtests/{id}/stop	Stop A/B test
DELETE	/2/abtests/{id}	Delete A/B test

Add A/B test

Path: /2/abtests
HTTP Verb: POST
Required API Key: any key with the setSettings ACL

Description:

Creates a new A/B test with provided configuration.

You can set an A/B test on two different indices with different settings, or on the same index with different search parameters by providing a customSearchParameters setting on one of the variants.

Parameters:

Parameter Description


            name

type: string

Required

A/B test name.


            variants

type: object

Required

List of 2 variants for the A/B test.

All properties are required, except description.

Copy
[
  {
    "index": string // mandatory
    "trafficPercentage": int // mandatory
    "estimatedSampleSize": int // optional
    "description": string // optional
  },
  {
    "index": string // mandatory
    "trafficPercentage": int // mandatory
    "estimatedSampleSize": int // optional
    "description": string // optional
  }
]

To create an A/B test for search parameters only, you must provide customSearchParameters for one of the variants:

Copy
[
  {
    "index": string // mandatory
    "trafficPercentage": int // mandatory
    "description": string // optional
  },
  {
    "index": string // mandatory
    "trafficPercentage": int // mandatory
    "customSearchParameters": object // mandatory for one of the variants
    "description": string // optional
  }
]


            endAt

type: Date, UTC. Inclusive.

Required

End date for the A/B test expressed as YYYY-MM-DDThh:mm:ssZ.


            configuration

type: object

Optional

Object containing overall configuration settings for the A/B test. All properties are optional.

Copy
{
  "outliers": {
    "exclude": boolean
  } // optional
  "emptySearch": {
    "exclude": boolean
  } // optional
  "minimumDetectableEffect": {
    "size": float // mandatory
    "effect": string // mandatory
  } // optional
}

Errors:

400: Invalid signature or invalid request
401: AppID or API Key missing
402: Feature not enabled
403: Invalid credentials
404: index not found
422: Invalid payload or unprocessable entity
500: Internal error (possible e.g. if insertion query fails or engine query fails)

Example:

Copy
curl -X POST \
     -H "X-Algolia-API-Key: ${API_KEY}" \
     -H "X-Algolia-Application-Id: ${APPLICATION_ID}" \
     --data-binary '{
      "name": "Custom Ranking salesRank Test",
      "variants": [
        {
          "index": "atis-abtest-default",
          "description": "current production index",
          "estimatedSampleSize": 9800,
          "trafficPercentage": 70
        },
        {
          "index": "atis-abtest-salesRank",
          "estimatedSampleSize": 4500,
          "trafficPercentage": 30
        }
      ],
      "endAt": "2018-05-17T23:59:59Z"
    }' \
    "https://analytics.algolia.com/2/abtests"

Upon success, the response is 200 OK.

Copy
{
  "abTestID":78,
  "taskID":111887230,
  "index":"atis-abtest-default"
}

Create an A/B test with the second variant enabling personalization:

Copy
curl -X POST \
     -H "X-Algolia-API-Key: ${API_KEY}" \
     -H "X-Algolia-Application-Id: ${APPLICATION_ID}" \
     --data-binary '{
      "name": "Custom Ranking salesRank Test",
      "variants": [
        {
          "index": "atis",
          "trafficPercentage": 70
          "estimatedSampleSize": 9800,
        },
        {
          "index": "atis",
          "trafficPercentage": 30,
          "estimatedSampleSize": 4500,
          "customSearchParameters": {
              "enablePersonalization": true
          }
        }
      ],
      "endAt": "2018-05-17T23:59:59Z"
    }' \
    "https://analytics.algolia.com/2/abtests"

Get A/B test

Path: /2/abtests/{id}
HTTP Verb: GET
Required API Key: any key with the analytics ACL

Description:

Returns metadata and metrics for A/B test id.

Behaves in the same way as GET /2/abtests however the endpoint will return 403.

Parameters:

Parameter	Description
`id`	type: int Required A/B test ID.

Response

Parameter	Description
`abTestID`	type: integer ID of the A/B test.
`clickSignificance`	type: float A/B test significance based on click data. Should be > 0.95 to be considered significant (no matter which variant is winning).
`conversionSignificance`	type: float A/B test significance based on conversion data. Should be > 0.95 to be considered significant (no matter which variant is winning)
`purchaseSignificance`	type: float A/B test significance based on `purchase` conversions. Should be greater than 0.95 to be considered significant, no matter which variant is winning.
`addToCartSignificance`	type: float A/B test significance based on `addToCart` conversions. Should be greater than 0.95 to be considered significant, no matter which variant is winning.
`revenueSignificance`	type: object A/B test revenue significance for each currency. This object has currency codes as keys and significant scores as values.
`createdAt`	type: string Time at which the A/B test has been created. Y-m-d\TH:i:s\Z
`endAt`	type: string Time at which the A/B test will automatically stop. Y-m-d\TH:i:s\Z
`name`	type: string Name of the A/B test.
`status`	type: string Current status of the A/B test.
`variants`	type: list of variant List of 2 variants: The base index The index to test against
`configuration`	type: configuration object Configuration parameters for the A/B test. { "outliers": outliers object, "emptySearch": emptySearch object "minimumDetectableEffect": minimumDetectableEffect object }

variants ➔ variant

Parameter	Description
`averageClickPosition`	type: int Average click position for the variant.
`clickCount`	type: int Distinct click count for the variant.
`clickThroughRate`	type: float Click through rate for the variant.
`conversionCount`	type: int Distinct conversion count for the variant.
`conversionRate`	type: float Conversion rate for the variant.
`addToCartCount`	type: int Distinct number of converted clicks with the `eventSubtype` set to `addToCart` and `clickAnalytics` set to `true`.
`addToCartRate`	type: float Add-to-cart rate is based on conversion events that have `eventSubType` set to `addToCart`, and the total number of tracked searches within the specified timeframe. The rates are computed as the number of add-to-cart conversion events divided by the number of tracked searches.
`purchaseCount`	type: int Distinct number of converted clicks with the `eventSubtype` set to `purchase` and `clickAnalytics` set to `true`.
`purchaseRate`	type: float Purchase rate is based on conversion events that have `eventSubType` set to `purchase`, and the total number of tracked searches within the specified timeframe. The rates are computed as the number of purchase conversion events divided by the number of tracked searches.
`description`	type: string Description of this variant.
`estimatedSampleSize`	type: int Estimated number of searches needed to achieve confidence for this variant. This number is pre-computed and is based on the `minimumDetectableEffect` configuration parameter.
`index`	type: string Index name of of this variant.
`noResultCount`	type: int Number of tracked searches without any results.
`searchCount`	type: int Total number of searches for this variant, including untracked searches.
`trackedSearchCount`	type: int Number of tracked searches for this variant. Tracked searches are searches with `clickAnalytics` enabled.
`userCount`	type: int Total number of users who searched on this variant, including untracked users.
`trackedUserCount`	type: int Number of users who searched this variants with at least one search that had `clickAnalytics` enabled.
`currencies`	type: object Revenue metrics for each currency. { "USD": currency object, "EUR": currency object }
`filterEffects`	type: filterEffects object Effects of the configuration filters applied to the variant. { "outliers": filterEffect, "emptySearch": filterEffect }

➔ currency object

Parameter	Description
`currency`	type: string Currency code.
`revenue`	type: float Revenue made in this currency.
`mean`	type: float Mean revenue for this currency.
`standardDeviation`	type: float Standard deviation of revenue for this currency.

➔ configuration object

Parameter	Description
`outliers`	type: outliers object Configuration for outlier exclusion from metrics calculations.
`emptySearch`	type: emptySearch object Configuration for empty search exclusion from metrics calculations.
`minimumDetectableEffect`	type: minimumDetectableEffect object Configuration for minimum detectable effect.

configuration ➔ outliers object

Parameter	Description
`exclude`	type: boolean default: true Whether outliers are excluded when calculating A/B test results.

configuration ➔ emptySearch object

Parameter	Description
`exclude`	type: boolean default: false Whether empty searches are excluded when calculating A/B test results.

configuration ➔ minimumDetectableEffect object

Parameter Description


            size

type: float

Required

The size of the minimum detectable effect. This is the smallest relative difference between the variants that you want to be able to detect. For example, if you want to be able to detect a 10% difference between the variants, you should set this to 0.1. The size must be between 0 and 1.


            effect

type: string

Required

The effect that you want to detect. This can be one of the following:

conversionRate
clickThroughRate
purchaseRate
addToCartRate

➔ filterEffects object

Parameter	Description
`outliers`	type: filterEffect Configuration effect of outlier exclusion from metrics calculations.
`emptySearch`	type: filterEffect Configuration effect of empty search exclusion from metrics calculations.

➔ filterEffect

Parameter	Description
`usersCount`	type: int Number of users excluded by this filter.
`trackedSearchesCount`	type: int Number of tracked searches excluded by this filter.

Errors:

401: AppID or API Key missing
403: Forbidden in case the API Key has no access to the main index for this A/B test.

Example:

Copy
curl -X GET \
     -H "X-Algolia-API-Key: ${API_KEY}" \
     -H "X-Algolia-Application-Id: ${APPLICATION_ID}" \
    "https://analytics.algolia.com/2/abtests/${AB_TEST_ID}"

Upon success, the response is 200 OK.

Copy
{
  "abTestID": 78,
  "clickSignificance": 1,
  "conversionSignificance": 0.9937,
  "createdAt": "2018-05-15T17:52:15.644906Z",
  "endAt": "2018-05-17T23:59:59Z",
  "name": "Custom Ranking salesRank Test",
  "status": "stopped",
  "configuration": {
    "outliers": {
      "exclude": true
    },
    "minimumDetectableEffect": {
      "size": 0.06,
      "metric": "clickThroughRate"
    }
  },
  "variants": [
    {
      "averageClickPosition": 0,
      "clickCount": 17115,
      "clickThroughRate": 0.19839107906664039,
      "conversionCount": 11508,
      "conversionRate": 0.13339670101658765,
      "description": "",
      "index": "atis-abtest-default",
      "noResultCount": 0,
      "searchCount": 86269,
      "trackedSearchCount": 70365,
      "trafficPercentage": 70,
      "userCount": 55501,
      "trackedUserCount": 44197,
      "estimatedSampleSize": 9500,
      "filterEffects": {
        "outliers": {
          "usersCount": 1,
          "trackedSearchesCount": 237
        }
      }
    },
    {
      "averageClickPosition": 0,
      "clickCount": 7716,
      "clickThroughRate": 0.20869847452125934,
      "conversionCount": 5129,
      "conversionRate": 0.13872660391647734,
      "description": "",
      "index": "atis-abtest-salesRank",
      "noResultCount": 0,
      "searchCount": 36972,
      "trackedSearchCount": 30459,
      "trafficPercentage": 30,
      "userCount": 22694,
      "trackedUserCount": 21356,
      "estimatedSampleSize": 9500,
      "filterEffects": {
        "outliers": {
          "usersCount": 0,
          "trackedSearchesCount": 0
        }
      }
    }
  ]
}

List A/B tests

Path: /2/abtests
HTTP Verb: GET
Required API Key: any key with the analytics ACL

Description:

Fetch all existing A/B tests for App that are available for the current API Key. Returns an array of metadata and metrics.

Notes:

This endpoint is paginated in the same way as other Analytics endpoints (i.e. using ?limit=10&offset=0).
As this endpoint fetches the list of all ABTests and then filters out the one that are not authorized for the given key, we do not return an error in case no ABTest is authorized for the key. We would just return an empty list of ABTests.
When no data has been processed, the metrics will be returned as null

Parameters:

Parameter	Description
`offset`	type: int default: 0 Position of the starting record. Used for paging. 0 is the first record.
`limit`	type: float default: 10 Number of records to return. +used for paging. Limit is the size of the page.
`indexPrefix`	type: string Optional Filter for A/B tests. Return A/B test variants for which the index name starts with `indexPrefix`.
`indexSuffix`	type: string Optional Filter for A/B tests. Return A/B test variants for which the index name ends with `indexSuffix`.

Response

Parameter	Description
`count`	type: integer Number of A/B tests returned.
`total`	type: integer Total number of A/B tests that can be fetched.
`abtests`	type: List of abTest

abTest

Parameter	Description
`abTestID`	type: integer ID of the A/B test.
`clickSignificance`	type: float A/B test significance based on click data. Should be > 0.95 to be considered significant (no matter which variant is winning).
`conversionSignificance`	type: float A/B test significance based on conversion data. Should be > 0.95 to be considered significant (no matter which variant is winning)
`purchaseSignificance`	type: float A/B test significance based on `purchase` conversions. Should be greater than 0.95 to be considered significant, no matter which variant is winning.
`addToCartSignificance`	type: float A/B test significance based on `addToCart` conversions. Should be greater than 0.95 to be considered significant, no matter which variant is winning.
`revenueSignificance`	type: object A/B test revenue significance for each currency. This object has currency codes as keys and significance scores as values. This object has currency codes as keys and significance scores as values.
`createdAt`	type: string Time at which the A/B test has been created. Y-m-d\TH:i:s\Z
`endAt`	type: string Time at which the A/B test will automatically stop. Y-m-d\TH:i:s\Z
`name`	type: string Name of the A/B test.
`status`	type: string Current status of the A/B test.
`variants`	type: list of variant List of 2 variants: The base index The index to test against
`configuration`	type: configuration object Configuration parameters for the A/B test. { "outliers": outliers object, "emptySearch": emptySearch object }

variants ➔ variant

Parameter	Description
`averageClickPosition`	type: int Average click position for the variant.
`clickCount`	type: int Distinct click count for the variant.
`clickThroughRate`	type: float Click through rate for the variant.
`conversionCount`	type: int Distinct conversion count for the variant.
`conversionRate`	type: float Conversion rate for this variant.
`addToCartCount`	type: int Distinct number of converted clicks with the `eventSubtype` set to `addToCart` and `clickAnalytics` set to `true`.
`addToCartRate`	type: float Add-to-cart rate is based on conversion events that have `eventSubType` set to `addToCart`, and the total number of tracked searches within the specified timeframe. The rates are computed as the number of add-to-cart conversion events divided by the number of tracked searches.
`purchaseCount`	type: int Distinct number of converted clicks with the `eventSubtype` set to `purchase` and `clickAnalytics` set to `true`.
`purchaseRate`	type: float Purchase rate is based on conversion events that have `eventSubType` set to `purchase`, and the total number of tracked searches within the specified timeframe. The rates are computed as the number of purchase conversion events divided by the number of tracked searches.
`description`	type: string Description of this variant.
`index`	type: string Index name of this variant.
`noResultCount`	type: int Number of tracked searches without any results.
`searchCount`	type: int Total number of searches made on this variant, including untracked searches.
`trackedSearchCount`	type: int Number of tracked searches. Tracked searches are searches with `clickAnalytics` enabled.
`userCount`	type: int Total number of users who searched this variant, including untracked users.
`trackedUserCount`	type: int Number of users who searched this variants with at least one search that had `clickAnalytics` enabled.
`currencies`	type: object Revenue metrics for each currency. { "USD": currency object, "EUR": currency object }
`filterEffects`	type: filterEffects object Effects of the configuration filters applied to the variant. { "outliers": filterEffect, "emptySearch": filterEffect }

➔ currency object

Parameter	Description
`currency`	type: string Currency code.
`revenue`	type: float Revenue made in this currency.
`mean`	type: float Mean revenue for this currency.
`standardDeviation`	type: float Standard deviation of revenue for this currency.

➔ configuration object

Parameter	Description
`outliers`	type: outliers object Configuration for outlier exclusion from metrics calculations.
`emptySearch`	type: emptySearch object Configuration for empty search exclusion from metrics calculations.

configuration ➔ outliers object

Parameter	Description
`exclude`	type: boolean default: true Whether outliers are excluded when calculating A/B test results.

configuration ➔ emptySearch object

Parameter	Description
`exclude`	type: boolean default: false Whether empty searches are excluded when calculating A/B test results.

➔ filterEffects object

Parameter	Description
`outliers`	type: filterEffect Configuration effect of outlier exclusion from metrics calculations.
`emptySearch`	type: filterEffect Configuration effect of empty search exclusion from metrics calculations.

➔ filterEffect

Parameter	Description
`usersCount`	type: int Number of users excluded by this filter.
`trackedSearchesCount`	type: int Number of tracked searches excluded by this filter.

Errors:

401: AppID or API Key missing

Example:

Copy
curl -X GET \
     -H "X-Algolia-API-Key: ${API_KEY}" \
     -H "X-Algolia-Application-Id: ${APPLICATION_ID}" \
    "https://analytics.algolia.com/2/abtests?offset=${OFFSET}&limit=${LIMIT}"

Upon success, the response is 200 OK.

Copy
{
  "abtests": [
    {
      "abTestID": 78,
      "clickSignificance": 1,
      "conversionSignificance": 0.9937,
      "createdAt": "2018-05-15T17:52:15.644906Z",
      "endAt": "2018-05-17T23:59:59Z",
      "name": "Custom Ranking salesRank Test",
      "status": "stopped",
      "configuration": {
        "outliers": {
          "exclude": true
        },
        "emptySearch": {
          "exclude": true
        },
        "minimumDetectableEffect": {
          "size": 0.06,
          "metric": "clickThroughRate"
        }
      },
      "variants": [
        {
          "averageClickPosition": 0,
          "clickCount": 17115,
          "clickThroughRate": 0.19839107906664039,
          "conversionCount": 11508,
          "conversionRate": 0.13339670101658765,
          "description": "",
          "index": "atis-abtest-default",
          "noResultCount": 0,
          "searchCount": 86269,
          "trackedSearchCount": 86269,
          "trafficPercentage": 70,
          "userCount": 55501,
          "trackedUserCount": 55501,
          "estimatedSampleSize": 9500,
          "filterEffects": {
              "outliers": {
                "usersCount": 1,
                "trackedSearchesCount": 300
              },
              "emptySearch": {
                "usersCount": 5,
                "trackedSearchesCount": 500
              }
           }
        },
        {
          "averageClickPosition": 0,
          "clickCount": 7716,
          "clickThroughRate": 0.20869847452125934,
          "conversionCount": 5129,
          "conversionRate": 0.13872660391647734,
          "description": "",
          "index": "atis-abtest-salesRank",
          "noResultCount": 0,
          "searchCount": 36972,
          "trackedSearchCount": 36972,
          "trafficPercentage": 30,
          "userCount": 22694,
          "trackedUserCount": 22694,
          "estimatedSampleSize": 9500,
          "filterEffects": {
              "outliers": {
                "usersCount": 0,
                "trackedSearchesCount": 0
              },
              "emptySearch": {
                "usersCount": 0,
                "trackedSearchesCount": 0
              }
           }
        }
      ]
    },
    {
      "abTestID": 80,
      "clickSignificance": 0.8187000000000001,
      "conversionSignificance": 0.6645,
      "createdAt": "2018-05-15T17:55:36.358352Z",
      "endAt": "2018-05-17T23:59:59Z",
      "name": "Searchable Attribute Manufacturer test",
      "status": "stopped",
      "variants": [
        {
          "averageClickPosition": 0,
          "clickCount": 65131,
          "clickThroughRate": 0.22,
          "conversionCount": 42808,
          "conversionRate": 0.14459719641952373,
          "description": "",
          "index": "atis-abtest-default",
          "noResultCount": 0,
          "searchCount": 296050,
          "trackedSearchCount": 294049,
          "trafficPercentage": 90,
          "userCount": 185719,
          "trackedUserCount": 184718,
        },
        {
          "averageClickPosition": 0,
          "clickCount": 7309,
          "clickThroughRate": 0.22219857724813036,
          "conversionCount": 4785,
          "conversionRate": 0.14546725846658964,
          "description": "",
          "index": "atis-abtest-manufacturer",
          "noResultCount": 0,
          "searchCount": 32894,
          "trackedSearchCount": 31245,
          "trafficPercentage": 10,
          "userCount": 20019,
          "trackedUserCount": 19018,
        }
      ]
    }
  ],
  "count": 2,
  "total": 2
}

Stop A/B test

Path: /2/abtests/{id}/stop
HTTP Verb: POST
Required API Key: any key with the setSettings ACL

Description:

Marks the A/B test as stopped. At this point, the test is over and cannot be restarted. As a result, your application is back to normal: index A will perform as usual, receiving 100% of all search requests. Associated metadata and metrics are still stored.

Parameters:

Parameter	Description
`id`	type: int Required A/B test ID.

Errors:

400: Invalid signature or invalid request
401: AppID or API Key missing
402: Feature not enabled
403: Invalid credentials
404: A/B test ID not found
409: A/B test was correctly deleted from the engine but the Analytics request failed (inconsistent state)
422: Invalid ID format
500: Internal error (possible e.g. if insertion query fails or engine query fails)

Example:

Copy
curl -X POST \
     -H "X-Algolia-API-Key: ${API_KEY}" \
     -H "X-Algolia-Application-Id: ${APPLICATION_ID}" \
    "https://analytics.algolia.com/2/abtests/#{AB_TEST_ID}/stop"

Upon success, the response is 200 OK.

Delete A/B test

Path: /2/abtests/{id}
HTTP Verb: DELETE
Required API Key: any key with the setSettings ACL

Description:

Deletes the A/B Test and removes all associated metadata & metrics.

Parameters:

Parameter	Description
`id`	type: int Required A/B test ID.

Errors:

400: Invalid signature or invalid request
401: AppID or API Key missing
402: Feature not enabled
403: Invalid credentials
404: A/B test ID not found
409: A/B test was correctly deleted from the engine but the Analytics request failed (inconsistent state)
422: Invalid payload, A/B test already running on one of the indices
500: Internal error (possible e.g. if insertion query fails or engine query fails)

Example:

Copy
curl -X DELETE \
     -H "X-Algolia-API-Key: ${API_KEY}" \
     -H "X-Algolia-Application-Id: ${APPLICATION_ID}" \
    "https://analytics.algolia.com/2/abtests/${AB_TEST_ID}"

Upon success, the response is 200 OK.

Did you find this page helpful?