Appearance
This wiki is for Framework CI, and is not applicable to other repositories like Engine, Packages. The integration test is referred to an end-to-end target/test presented in Flutter build dashboard, which is an one-on-one mapping to the entries listed in the .ci.yaml file.
Overview
Types of integration tests (based on how they are being executed):
- DeviceLab
- Uses test harness:
test_runner.dart
- Relies on recipe:
devicelab_drone.py
- This consists of two types further
- One needs a physical phone (a valid value for either
device_type
ordevice_os
in .ci.yaml) - The other runs on a host only testbed (either
none
or not defined for bothdevice_type
ordevice_os
in .ci.yaml)DeviceLab
here for host only testbed is a legacy name which refers to using thedevicelab_drone.py
recipes and relying on atask.dart
file defined underdev/devicelab/bin/tasks
. But this does NOT need a physical device. In the long term, we may want to rename to avoid confusion.
- One needs a physical phone (a valid value for either
- Uses test harness:
- Shard
- Uses test harness:
test.dart
- Relies on recipe:
flutter_drone.py
- A
shard
property is defined for these targets
- Uses test harness:
- Others
- most likely are specific targets used to test specific functionality
- examples:
The word DeviceLab
initially was used to represent targets running in Flutter's self-maintained hardware lab where bots are connected with a physical device. Later it has been extended to represent targets that use the test harness test_runner.dart
which is located under dev/devicelab/bin
. All these targets need an entry defined under dev/devicelab/bin/tasks
, and they include ones that do not need a physical device (known as host only tests).
Shard
tests are using the test harness test.dart
, which supports targets that are shardable to run in parallel. Additionally it supports tests with a single shard, which means these tests are not feasible to run in parallel. These tests have only a single shard running a block of scripts.
There is an overlap happens between DeviceLab
and Shard
: a single shard test can also run under the DeviceLab
test harness.
Where to add an integration test
Most likely, we can fit a new integration test to existing types, like DeviceLab
, Shard
or other case-by-case tests that use their own recipes
in addition to DeviceLab
and Shard
, e.g. firebaselab, packaging, docs, etc. If your new test doesn't fit in any of these (very rarely), it may need a new recipe.
NOTE
Recipes
are just python scripts detailing steps to setup env. and execute corresponding test harness. Different recipes basically mean different test harness with different environment setup.
For the two main types (DeviceLab
/Shard
):
- if a new integration test needs a physical device, it should be under
DeviceLab
- if a new integration test doesn't need a physical device but needs to collect benchmarks, it should be under host only
Devicelab
- if a new integration test need to run in parallel with sharding, it should be under
Shard
- others should be good with either host only
DeviceLab
orShard
with a single shard.
How to add an integration test as a DeviceLab
target
Please refer to how to write a DeviceLab
test and how to add it to continuous integration.
Quick steps:
- creates a test file under
dev/devicelab/bin/tasks/<test>.dart
- adds a new .ci.yaml entry by mirroring an existing target with
recipe: devicelab_drone
(see .ci.yaml readme)- begins with
bringup: true
- specifies
device_type
ordevice_os
if needed - removes
bringup: true
after validated in post-submit CI (in staging pool).
- begins with
- adds an ownership entry to TESTOWNERS
- adds entries for other platforms if needed
How to add an integration test as a Shard
target
Please refer to steps-to-add-a-new-framework-test-shard.
How to add an integration tests with Android emulator support
In this section we will build a new target for the Linux
platform that will run in the DeviceLab
with an Android emulator. Note: it is also supported in Shard
.
Name and platform
To add a test in the Framework Repository with Android Emulators via the DeviceLab recipe, you will not have to do anything on the recipe side of the code as simply specifying the configuration will allow you to create an Android Emulator on demand. Using other custom tests will possibly require changes in the recipes repository.
When adding a new target make sure that the target platform is Linux_android_emu
. This is done through the name of the target. This means that you can define your new target as something like:
yaml
- name: Linux_android_emu new_test_to_add
This tells the CI that you want to use the Linux
platform and your test is named new_test_to_add
. See [.ci.yaml] (https://github.com/flutter/cocoon/blob/main/CI_YAML.md) for more details. The platform-level config already defines all necessary dimensions/properties that an emulator test needs.
The dimensions
are a way to use the correct machine type with the supported virtualization, the dependency
on the android_virtual_device tells the recipes framework that an emulator was requested and which api level to use and finally the device_type
tells it to use a machine without a connected device. This will avoid issues with multiple devices found during testing.
Target configs
Add any additional properties/dependencies your test may need.
yaml
name: Linux_android_emu new_test_to_add
recipe: devicelab/devicelab_drone
bringup: true
properties:
tags: >
["framework","hostonly","linux"]
task_name: android_views
timeout: 60
You will notice that task_name
is new and the tags
are new. The task_name
is the name of your test script (minus the .dart suffix) and the tags allow infra to perform statistical analysis based on these in order to monitor SLO for task times, execution time as well as many other metrics.
The above target can be added and run assuming there exists a ·new_test_to_add.dart· file in the Flutter repo.