13 Running Live Experiments With A/B Testing & Feature Flags

In the previous chapter, you learned about two Firebase services — Firebase Analytics and Firebase Crashlytics. While browsing through your Firebase console, you might’ve noticed that it offers quite a few other interesting services. So, in this chapter, you’ll look at two of those additional services — Firebase A/B Testing and Firebase Remote Config.

After thousands of hours spent learning good UX and UI practices, studying marketing and working on product development, there will always be situations when you won’t be sure about a specific design decision, call-to-action strategy or advertising text. Even when you think you made the right choice based on your experience, there might be cases when taking a different approach would work better. You might think these differences are irrelevant to your app, but the truth is that an increase of only a few percentage points in user engagement might reflect enormous gains in your sales. The math behind that is quite simple: more sales = more money.

But now, you might be wondering how to determine which solution to a specific problem will work best for your users. That’s where Firebase A/B Testing comes in handy. Publishing a new release of the app for every change required for your experiments would be very inconvenient. Firebase Remote Config has you covered there. With the help of a so-called feature tag, you can make changes to your app on the fly.

With Firebase A/B Testing, you can run experiments in marketing, advertising, politics, product development, design, etc. As mentioned already, you’ll have to use Firebase Remote Config to implement those.

In this chapter, you’ll learn how to:

• Add a feature flag to a specific feature.
• Perform changes in the app without rolling out a new version.
• Set up an experiment in Firebase.
• Perform and track experiments with Firebase.

While going through this chapter, you’ll work on the starter project from this chapter’s assets folder.

A Deep Dive Into A/B Testing

The introduction already mentioned a few use cases of A/B testing. But before performing your first A/B test, you have to dive a little deeper into the theory behind it.

A/B testing is a simple experiment in which two different versions of a variable — A and B — are compared. The testing group is randomized and usually equally distributed between both samples. More complex cases might have more than just two versions of a specific variable. The use of A/B testing started far before the era of modern computers. Originally, A/B tests were used to figure out which advertising pamphlets worked best. Nowadays, A/B tests are standard procedure in product development as well as preparation of marketing materials — from small businesses to tech giants.

Here’s how A/B testing works in the example of a simple call to action for a shoe store app. You start by preparing two versions of the call-to-action button text. The first one says: “Only five pairs left”, and the second one says “Limited number of pairs left”. After that, you have to clearly define the metrics you want to track for this experiment. You’ll figure out why this is very important in that stage. For the sake of this example, say that you want to find out which text attracts more users to buy the product — this becomes the metric you want to track. Finally, you randomly distribute the two versions of the app amongst your users — 50% will use the first version of the app with the first text, and the other 50% will see the second one with the second text.

After performing the test for two weeks, you’ve noticed that 7% of the users with the first version of the app ended up buying the shoes. On the other hand, only 5% of users with the second version of the app made a purchase. The result makes quite a bit of sense. Users of the first version stress more about shoes going out of stock than the users of the second version; therefore, they’re more motivated to buy. Based on that information, you clearly see that the first call-to-action text works better, and you can distribute it to all the users.

On the other hand, you might want to perform the same experiment and focus on slightly different metrics. If you want to measure the use time of your app, the result might be very different. Because the users of the second app version don’t feel such an urgency to buy the product, they might spend a little bit more time scrolling through other products.

The example above was a very simple one, with only a slight difference between the two samples. But don’t let this example fool you. In slightly more complex A/B tests, you could test the performance of two or even more completely different UIs and features, and measure multiple different metrics.

The last thing you need to understand before continuing with this chapter is the aspect of segmentation of the responses in a specific test. Even though, in the previous example, the first version of the variable — the call-to-action text — worked the best, this might not be the case for some subgroups of your testing group. For example, for a specific age group or gender, the second version of the variable might work significantly better. You must take this into account when you target your audience — you should distribute the second version of the call-to-action text to this specific subgroup once the test is finished.

Logic Behind Remote Config and Feature Flags

As already mentioned, A/B testing and Firebase Remote Config go hand in hand. The second is normally used only as a tool to correctly distribute the different variables among the users when performing A/B tests. This will also be a case in the example for this chapter, although it’s only fair to mention other use cases of Remote Config.

A feature flag defines a variable on which a variant of the specific feature is based. This can be a very simple variable, such as a string to define a button’s text, as shown in the previous section, a Boolean to show or hide a specific feature, an integer to define the perfect size of items per pagination page, etc. On the other hand, it can be a very complex model to define, for example, the theme of the app.

Firebase Remote Config is a cloud service that allows you to perform remote changes to a feature flag in your app. With its help, you can change your app’s appearance and behavior without rolling out the new version of the app. This can come in handy when you want to modify your app for a specific audience or when you want to roll out a new feature gradually. Pushing out a new feature can be a very stressful thing to do. You can’t ever know how the users will react to a new feature; plus, there might still be a few bugs that you missed. Therefore, releasing a new feature only for a certain percentage of your users might be very useful.

Installing Firebase Remote Config

Time to get your hands dirty. To add Firebase Remote Config, you’ll use the firebase_remote_config package. Locate pubspec.yaml in the packages/monitoringpackage:

Replace # TODO: add firebase_remote_config package with the following line of code:

firebase_remote_config: ^2.0.11

Note

firebase_remote_config also requires firebase_core to work, which you learned how to do in the previous chapter.

After you’ve added the required packages, run the make get command in the terminal to fetch the new packages.

In the packages/monitoring/lib/src/ folder, locate // TODO: add an implementation of RemoteValueService class in remote_value_service.dart, and replace it with the following code:

static const _gridQuotesViewEnabledKey = 'grid_quotes_view_enabled';

// 1
RemoteValueService({
  @visibleForTesting FirebaseRemoteConfig? remoteConfig,
}) : _remoteConfig = remoteConfig ?? FirebaseRemoteConfig.instance;

final FirebaseRemoteConfig _remoteConfig;

// 2
Future<void> load() async {
  // TODO: add default values for your parameters
  await _remoteConfig.fetchAndActivate();
}

// 3
bool get isGridQuotesViewEnabled => _remoteConfig.getBool(
      _gridQuotesViewEnabledKey,
    );

This is what the code above does:

1. Initializes the instance of FirebaseRemoteConfig.
2. Fetches and activates the configuration that you’ll set up on the Firebase Remote Config console.
3. Returns the value of the feature flag you defined.

Next, import the missing package by replacing // TODO: import firebase_remote_config package with the following line of code:

import 'package:firebase_remote_config/firebase_remote_config.dart';

With that, you’ve successfully added the Firebase Remote Config package and created a wrapper around it.

Creating New Parameters in Firebase Remote Config Console

Now that you’ve successfully added Firebase Remote Config into your app, you can start adding the parameters to change the app’s appearance.

Go to your Firebase console and navigate to the Remote Config tab under the Engagesection, as shown in the image below:

You may notice the Create configuration button on the screen:

Clicking it will make the action window pop up, where you can configure your first parameter.

For this example, you’ll manipulate the appearance of the quote list screen. You can see that the current layout of quotes is a staggered list. But you may want to change it to a normal list someday. To achieve this, create a new parameter called grid_quotes_view_enabled in the Parameter name (key) field, select Boolean as the Data type and set the Default value to false. For a better understanding, look at the picture above.

When you’re done filling in all the required fields, click Save at the bottom of the window. With that, you’ll be taken to the Remote Config console.

At the top of the screen, you’ll have to click Publish changes to push the configuration that your apps can later use:

After you’ve clicked the button, scroll to the end of the screen and look at the config panel:

The config panel has a list of all the parameters that you’ve configured so far. For now, you’ve only configured one. Focus on the number in the Fetch % column. This shows the percentage of apps that have fetched this configuration and, consequently, updated the UI.

Implementing UI Changes Based on Your Remote Config

First, navigate to the app’s main.dart file located in the app-level lib folder. Find // TODO: add loading feature flags from remote config, and replace it with:

await remoteValueService.load();

With that, you always call the method responsible for fetching and activating the feature flags based on the remote configuration when launching the app. You may notice that the previous line of code comes just after the following line:

final remoteValueService = RemoteValueService();

The code above initializes the RemoteValueService object. Next, open remote_value_service.dart in the monitoring package, and replace // TODO: add default values for your parameters with the following code:

await _remoteConfig.setDefaults(<String, dynamic>{
  'grid_quotes_view_enabled': true,
});

This sets the default values of the parameters defined on your Firebase Remote Config console. In your case, this is the bool parameter grid_quotes_view_enabled. By default, you want to keep your quote’s list screen layout as the grid. Therefore, you set the grid_quotes_view_enabled parameter to true. Remember, you’ve set the value of this same parameter to false in the Remote Config console, which means this will be overridden when you publish the changes.

In quote_list_screen.dart located in packages/features/quote_list/lib/src, find // TODO: display different UI based on the value of grid_quotes_view_enabled parameter. Replace the QuoteSliverGrid widget with the code below:

child: widget.remoteValueService.isGridQuotesViewEnabled
    ? QuotePagedGridView(
        pagingController: _pagingController,
        onQuoteSelected: widget.onQuoteSelected,
      )
    : QuotePagedListView(
        pagingController: _pagingController,
        onQuoteSelected: widget.onQuoteSelected,
      ),

With the code above, you’ll display the quotes in the layout of either the grid or list. Build and run the app, and you’ll see the following changes in the UI:

Note

If you’re having trouble running the app, you might have forgotten to propagate the configurations you did in the first chapter’s starter project to the following chapters’ materials. If that’s the case, please revisit Chapter 1, “Setting up Your Environment”.

Go back to your Remote Config Console and look at the value in the Fetch % column — you’ll see that it changed to 100%:

This means that the change has been applied to all of your app’s users. If your UI didn’t successfully update, delete the app and restart it.

Notice the edit button on the image above. By clicking it, you can change the value of a specific parameter. Try changing grid_quotes_view_enabled to true and refresh the app. You’ll see that the appearance of the quote list screen changes back to the grid.

Performing A/B Tests

By implementing Firebase Remote Config, you have control over the appearance and behavior of your app in the sense of predefined parameters. But now, your app has multiple different appearances, and you may be wondering which is the best for the end user.

In the case of the previous example, you have two variations of the quote list screen. You can find out which is better for your app’s users with the help of A/B testing. Go back to your Firebase Remote Config console and click the Create your first A/B Testbutton on the A/B Testing panel:

It will take you to the following screen:

By going through the following steps, you’ll create a new A/B test. Set the Experiment name to Quote List Retention and click Next to continue. Then, select the app — iOS or Android — on which you want to perform tests. By clicking the and button, you may specify even more details on targeting.

Change the Exposure to 100%. In real-life situations, you wouldn’t want to perform the tests on every user, but for demonstrational purposes, this will work better for you:

In the third step, Goals, set Primary metric to track (determines leader) to Retention (1 day):

Finally, continue to the fourth step, Variants. Set the Baseline Parameter to grid_quotes_view_enabled with the value of false. Then, set the Variant A Parameterto grid_quotes_view_enabled with the Value of true:

When you’ve defined your first experiment, continue to the next screen by clicking Review. Then, start your first experiment by clicking Start experiment:

After starting the experiment, the parameters will be distributed to the users with the help of Firebase Remote Config, and the data on retention will be collected. Based on that data, deciding which appearance of the quote list screen works the best for the end user won’t be such a difficult job anymore. After collecting some data, this is how your console will look:

Key Points

• A/B tests are crucial for making your app more suitable for its end users.
• Firebase Remote Config allows you to perform changes in the app without needing to release a new version.
• By performing different A/B tests on your app with the help of Remote Config, you may be able to make better decisions about your app’s appearance and behavior.