feat(specs): add abtests v3 #4394

Niamsy · 2025-01-24T11:14:01Z

🧭 What and Why

🎟 JIRA Ticket: OPTIM-1443
OPTIM-1456
OPTIM-1535

Changes included:

ADD abtests-v3 specs openapi
POST /3/abtests allows more than 2 variants, a configuration, and requires a set of metrics to be retrieved during the A/B Test
GET /3/abtests returns all the metrics requested during A/B test creation
GET /3/abtests/{id} idem

🧪 Test

algolia-bot · 2025-01-24T11:14:14Z

✔️ Code generated!

Name	Link
🪓 Triggered by	`f91137027a2fc1173f38204b8d779a768c5d2163`
🍃 Generated commit	`554adae9a563cb5bce2bf774e9cc4e76d533f425`
🌲 Generated branch	`generated/docs/abtests/api-v3`

📊 Benchmark results

Benchmarks performed on the method using a mock server, the results might not reflect the real-world performance.

Language	Req/s
go	1731
javascript	1661
php	1495
csharp	1269
python	1090
java	1081
ruby	765
swift	306
scala	23

cdhawke

I did a first pass - we can make adjustments after we get a base established too!

specs/abtesting-v3/common/schemas/ABTest.yml

specs/abtesting-v3/common/schemas/Variant.yml

specs/abtesting-v3/paths/abtests.yml

specs/abtesting-v3/paths/scheduleABTest.yml

specs/abtesting-v3/paths/estimate.yml

millotp · 2025-01-28T15:30:55Z

Hello @Niamsy, this is the first time that we have a new api version in the repo, how would like it to distribute it ?
Are you planning on supporting v2 as well ?
Do you want v3 to be in a different package on npm or just change the version of the existing one ?

Niamsy · 2025-01-28T16:39:00Z

Hello @millotp, we're planning to support both v2 and v3 in parallel. As for the distribution, I'm not sure, maybe @cdhawke has a better idea.
We're still in the development phase for the v3 api but we wanted to have an openApi spec somewhere so that's why there is this PR, but we're not planning on merging it right away

cdhawke · 2025-01-28T20:09:49Z

@millotp - you might know better how to separate these. We will support both versions, and the only thing changing outside of the request signatures/objects will be the /2 -> /3 in the url used to differentiate the version. Would you recommend keeping it separate? We might in the future try to transition off of v2, but there's no official timeline for that, as it will entail an EOL process. In this future scenario would it be better to have it as a separate package?

Niamsy · 2025-04-14T13:13:06Z

Hi @millotp, we'll soon want to release abtests v3 and merge this PR. As mentioned previously, we will support both v2 and v3 in parallel. Can we just merge this PR when we're ready or do we need to do specific things before ?

specs/abtesting-v3/common/parameters.yml

specs/abtesting-v3/common/schemas/ABTest.yml

shortcuts

wonderful

specs/abtesting-v3/common/schemas/ABTest.yml

Co-authored-by: Christopher Hawke <[email protected]>

… values in variant schema

… get/list responses

…4881)

Co-authored-by: Clément Vannicatte <[email protected]>

…iers and empty search

stale review

…stimate endpoint

shortcuts

thanks!

Co-authored-by: My Lan Aragon <[email protected]> Co-authored-by: Christopher Hawke <[email protected]> Co-authored-by: Steven Evans <[email protected]> Co-authored-by: Leonardo Gavaudan <[email protected]> Co-authored-by: Clément Vannicatte <[email protected]>

cdhawke requested changes Jan 27, 2025

View reviewed changes

stevenMevans previously approved these changes Feb 4, 2025

View reviewed changes

cdhawke previously approved these changes Feb 4, 2025

View reviewed changes

Niamsy dismissed stale reviews from cdhawke and stevenMevans via c5d5d83 April 10, 2025 12:57

Niamsy requested a review from stevenMevans April 14, 2025 09:06

stevenMevans previously requested changes Apr 15, 2025

View reviewed changes

specs/abtesting-v3/common/parameters.yml Outdated Show resolved Hide resolved

specs/abtesting-v3/common/schemas/ABTest.yml Show resolved Hide resolved

shortcuts previously approved these changes Apr 24, 2025

View reviewed changes

specs/abtesting-v3/common/schemas/ABTest.yml Outdated Show resolved Hide resolved

shortcuts mentioned this pull request Apr 24, 2025

feat(javascript): generate abtesting v3 alpha client #4783

Merged

Niamsy dismissed shortcuts’s stale review via 261190f April 25, 2025 08:39

leonardogavaudan force-pushed the docs/abtests/api-v3 branch from f7b6b99 to a7654a2 Compare June 18, 2025 07:12

Niamsy and others added 14 commits July 4, 2025 11:53

docs(abtests) v3 api

90054e0

Update specs/abtesting-v3/common/schemas/Variant.yml

f09e83e

Co-authored-by: Christopher Hawke <[email protected]>

Update specs/abtesting-v3/common/schemas/Variant.yml

af3d013

Co-authored-by: Christopher Hawke <[email protected]>

Update specs/abtesting-v3/common/schemas/Variant.yml

9f41334

Co-authored-by: Christopher Hawke <[email protected]>

Update specs/abtesting-v3/paths/abtests.yml

ebb0a58

Co-authored-by: Christopher Hawke <[email protected]>

Update specs/abtesting-v3/paths/scheduleABTest.yml

0a7faeb

Co-authored-by: Christopher Hawke <[email protected]>

Update specs/abtesting-v3/paths/estimate.yml

b75ccfe

Co-authored-by: Christopher Hawke <[email protected]>

fix(abtests) remove significance from abtest

99a31a7

feat(abtests) move filterEffects in metadata + add description to pValue

92286d8

feat(abtests) add metadata into metric

a04ae8c

doc(abtests) add timeseries endpoint for v3 + add confidence interval…

9ca8fad

… values in variant schema

add confidence interval field descriptions

f2b99f0

fix(abtests) traffic percentage min and max + add migratedAbTestID to…

6c2f5e2

… get/list responses

feat(abtests) add error correction to get/list endpoint for v3

b83e6de

leonardogavaudan and others added 2 commits July 4, 2025 11:53

feat(abtests): Add query params for GET /3/abtests/{id}/timeseries (#…

0bf629d

…4881)

Update specs/abtesting-v3/common/schemas/ABTest.yml

4cebf3f

Co-authored-by: Clément Vannicatte <[email protected]>

leonardogavaudan force-pushed the docs/abtests/api-v3 branch from 5eea288 to 4cebf3f Compare July 4, 2025 09:53

leonardogavaudan added 2 commits July 4, 2025 12:16

Add filters and errorCorrection to AB test config, remove legacy outl…

5beb7b1

…iers and empty search

Add noResults to list of metrics supported

77b77ef

leonardogavaudan marked this pull request as ready for review July 4, 2025 10:18

leonardogavaudan requested a review from a team as a code owner July 4, 2025 10:18

leonardogavaudan requested review from millotp and shortcuts July 4, 2025 10:18

shortcuts previously approved these changes Jul 4, 2025

View reviewed changes

shortcuts changed the title ~~docs(abtests) v3 api~~ feat(specs): add abtests v3 Jul 4, 2025

Merge branch 'main' into docs/abtests/api-v3

c016091

shortcuts enabled auto-merge (squash) July 4, 2025 12:13

shortcuts disabled auto-merge July 4, 2025 12:25

chore: validate specs on ci/cli

2dd5504

shortcuts dismissed their stale review via 2dd5504 July 4, 2025 12:27

leonardogavaudan and others added 3 commits July 4, 2025 15:00

add metrics and configuration to the schedule endpoint

c20a329

Fix tests: replace outliers and empty search with metric filters in e…

0ce4b0c

…stimate endpoint

Merge branch 'main' into docs/abtests/api-v3

b335034

shortcuts enabled auto-merge (squash) July 4, 2025 13:31

shortcuts disabled auto-merge July 4, 2025 13:31

shortcuts enabled auto-merge (squash) July 4, 2025 13:31

shortcuts approved these changes Jul 4, 2025

View reviewed changes

shortcuts merged commit ff2f96a into main Jul 4, 2025
29 checks passed

shortcuts deleted the docs/abtests/api-v3 branch July 4, 2025 13:47

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

feat(specs): add abtests v3 #4394

feat(specs): add abtests v3 #4394

Uh oh!

Niamsy commented Jan 24, 2025 •

edited by jira bot

Loading

Uh oh!

algolia-bot commented Jan 24, 2025 •

edited

Loading

Uh oh!

cdhawke left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

millotp commented Jan 28, 2025

Uh oh!

Niamsy commented Jan 28, 2025

Uh oh!

cdhawke commented Jan 28, 2025

Uh oh!

Niamsy commented Apr 14, 2025

Uh oh!

Uh oh!

Uh oh!

shortcuts left a comment

Uh oh!

Uh oh!

shortcuts left a comment

Uh oh!

Uh oh!

Uh oh!

feat(specs): add abtests v3 #4394

feat(specs): add abtests v3 #4394

Uh oh!

Conversation

Niamsy commented Jan 24, 2025 • edited by jira bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🧭 What and Why

Changes included:

🧪 Test

Uh oh!

algolia-bot commented Jan 24, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

✔️ Code generated!

Uh oh!

cdhawke left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

millotp commented Jan 28, 2025

Uh oh!

Niamsy commented Jan 28, 2025

Uh oh!

cdhawke commented Jan 28, 2025

Uh oh!

Niamsy commented Apr 14, 2025

Uh oh!

Uh oh!

Uh oh!

shortcuts left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

shortcuts left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Niamsy commented Jan 24, 2025 •

edited by jira bot

Loading

algolia-bot commented Jan 24, 2025 •

edited

Loading