Investigate A/B test plugin infrastructure

[vianasw]

No description provided.

[mwatson]

Field Guide: PCYsg-FpD-p2

[mwatson]

The code to get this working is pretty straightforward. A very simple example (make the dropdown say "Translate With Chrome" instead of just "Translate"):

const DefaultLabel = label;
const TreatmentLabel = "Translate With Chrome";

return (
	<DropdownMenu
		className="ai-assistant__i18n-dropdown"
		icon={ globe }
		label={ label }
		toggleProps={ {
			children: (
				<>
					<div className="ai-assistant__i18n-dropdown__toggle-label">{ 
						<Experiment
							name="testing_this_experiment_does_not_exist"
							defaultExperience={ DefaultLabel }
							treatmentExperience={ TreatmentLabel }
							loadingExperience={ "Loading..." }
						/> 
					}</div>
					<Icon icon={ chevronRight } />
				</>
			),
			disabled,
		} }
	>
)

(this is projects/plugins/jetpack/extensions/blocks/ai-assistant/components/i18n-dropdown-control/index.tsx)

I need to read more docs on how to actually work with ExPlat so I can validate this with local testing. Once that part is done, this should be a matter of splitting the backend call to the API into users that continue to do that and users who have access to the supported versions of Chrome and instead making calls to the translator (seen in the comments here).

There are a lot of field guides on working with ExPlat so I need to dig in a little more and figure out which are relevant. I'm also looking through the base ExPlatClient to learn more about what it actually does, since the jetpack-explat repo is mostly just a wrapper for that.

[mwatson]

The docs I found were in the Calypso repo, but I think they're consistent with usage I'm seeing elsewhere: https://github.com/Automattic/wp-calypso/blob/trunk/packages/explat-client-react-helpers/README.md

I think for the purposes of this project and the example code above, the useExperiment function has more functionality and will be a lot easier to work with since we're not going to be messing with any frontend components, rather we'll just be deciding if we should call the wpcom API or using built in functionality of Chrome.

[mwatson]

Slack thread about the primary metric: p1737640709675809-slack-C054LN8RNVA

Also using useExperiment is a lot cleaner as expected:

const [ isLoadingExperimentAssignment, experimentAssignment ] = useExperiment( 'testing_this_experiment_does_not_exist' );

if ( experimentAssignment?.variationName === 'treatment' ) {
	label = 'Translate With Chrome';
}

return (
	<DropdownMenu
		className="ai-assistant__i18n-dropdown"
		icon={ globe }
		label={ label }
		toggleProps={ {
			children: (
				<>
					<div className="ai-assistant__i18n-dropdown__toggle-label">{ label }</div>
					<Icon icon={ chevronRight } />
				</>
			),
			disabled,
		} }
	>
)

I was also able to test it by editing the data in localstorage and on reload it successfully showed me the treatment variation:

[mwatson]

Marking this as done for now as I think we have enough info on the testing platform. We are also going to focus on implementing Chrome AI and enabling it for a12s and then shift to A/B testing and tracking stats such as acceptance rate.