Mn Python tests

Marionette is the codename of a remote protocol built in to Firefox as well as the name of a functional test framework for automating user interface tests.

The in-tree test framework supports tests written in Python, using Python’s unittest library. Test cases are written as a subclass of MarionetteTestCase, with child tests belonging to instance methods that have a name starting with test_.

You can additionally define setUp and tearDown instance methods to execute code before and after child tests, and setUpClass/tearDownClass for the parent test. When you use these, it is important to remember calling the MarionetteTestCase superclass’ own setUp/tearDown methods since they handle setup/cleanup of the session.

The test structure is illustrated here:

from marionette_harness import MarionetteTestCase

class TestSomething(MarionetteTestCase):
    def setUp(self):
        # code to execute before any tests are run
        MarionetteTestCase.setUp(self)

    def test_foo(self):
        # run test for 'foo'

    def test_bar(self):
        # run test for 'bar'

    def tearDown(self):
        # code to execute after all tests are run
        MarionetteTestCase.tearDown(self)

Test assertions

Assertions are provided courtesy of unittest. For example:

from marionette_harness import MarionetteTestCase

class TestSomething(MarionetteTestCase):
    def test_foo(self):
        self.assertEqual(9, 3 * 3, '3 x 3 should be 9')
        self.assertTrue(type(2) == int, '2 should be an integer')

The API

The full API documentation is found here, but the key objects are:

MarionetteTestCase: a subclass for unittest.TestCase used as a base class for all tests to run.
Marionette: client that speaks to Firefox

Registering Test Manifests

To run Marionette Python tests locally via mach or as part of the Mn tests jobs in CI they need to be registered. This happens by adding a manifest file to the tree, which includes a reference to the test files and expectations for results.

Such a manifest file can look like the following and is stored with the extension .toml:

[DEFAULT]

["test_expected_fail.py"]
expected = "fail"

["test_not_on_windows.py"]
skip-if = ["os == 'win'"]

The registration of such a manifest file is done in two different ways:

To run the tests locally via ./mach test or ./mach marionette-test the created Marionette manifest file needs to be referenced in the folder’s related moz.build file by adding it to the MARIONETTE_MANIFESTS variable like:

MARIONETTE_MANIFESTS += [“test/marionette/manifest.toml”]
To run the tests in CI the manifest file also needs to be included in the Marionette’s own master manifest file. This ensures that the test packaging step will find the tests and include them as well in the test package.