Writing new DAMP performance tests

See DAMP Performance tests for an overall description of our performance tests. Here, we will describe how to write a new test and register it to run in DAMP.

Note

Reuse existing tests if possible! If a custom page already exists for the tool you are testing, try to modify the existing custom test rather than adding a new individual test. New individual tests run separately, in new tabs, and make DAMP slower than just modifying existing tests. Complexifying custom test pages should also help cover more scenarios and catch more regressions. For those reasons, modifying existing tests should be the preferred way of extending DAMP coverage. custom tests are using complex documents that should stress a particular tool in various ways. They are all named custom.${tool} (for instance custom.inspector). The test pages for those tests can be found in pages/custom. If your test case requires a dedicated document or can’t run next to the other tests in the current custom test, follow the instructions below to add a new individual test.

This page contains the general documentation for writing DAMP tests. See also:

Test location

Tests are located in testing/talos/talos/tests/devtools/addon/content/tests. You will find subfolders for panels already tested in DAMP (debugger, inspector, …) as well as other subfolders for tests not specific to a given panel (server, toolbox).

Tests are isolated in dedicated files. Some examples of tests:

Basic test

The basic skeleton of a test is:

const {
  testSetup,
  testTeardown,
  SIMPLE_URL,
} = require("devtools/docs/head");

module.exports = async function() {
  await testSetup(SIMPLE_URL);

  // Run some measures here

  await testTeardown();
};

  • always start the test by calling testSetup(url), with the url of the document to use

  • always end the test with testTeardown()

Test documents

DevTools performance heavily depends on the document against which DevTools are opened. There are two “historical” documents you can use for tests for any panel:

  • “Simple”, an empty webpage. This one helps highlighting the load time of panels,

  • “Complicated”, a copy of bild.be, a German newspaper website. This allows us to examine the performance of the tools when inspecting complicated, big websites.

The URL of those documents are exposed by tests/head.js. The Simple page can be found at testing/talos/talos/tests/devtools/addon/content/pages/simple.html. The Complicated page is downloaded via tooltool automatically the first time you run the DAMP tests.

You can create also new test documents under testing/talos/talos/tests/devtools/addon/content/pages. See the pages in the custom subfolder for instance. If you create a document in pages/custom/mypanel/index.html, the URL of the document in your tests should be PAGES_BASE_URL + "custom/mypanel/index.html". The constant PAGES_BASE_URL is exposed by head.js.

Note that modifying any existing test document will most likely impact the baseline for existing tests.

Finally you can also create very simple test documents using data urls. Test documents don’t have to contain any specific markup or script to be valid DAMP test documents, so something as simple as testSetup("data:text/html,my test document"); is valid.

Test helpers

Helper methods have been extracted in shared modules:

  • tests/head.js for the most common ones

  • tests/{subfolder}/{subfolder}-helpers.js for folder-specific helpers (example)

To measure something which is not covered by an existing helper, you should use runTest, exposed by head.js.

module.exports = async function() {
  await testSetup(SIMPLE_URL);

  // Calling `runTest` will immediately start recording your action duration.
  // You can execute any necessary setup action you don't want to record before calling it.
  const test = runTest(`mypanel.mytest.mymeasure`);

  await doSomeThings(); // <== Do an action you want to record here

  // Once your action is completed, call `runTest` returned object's `done` method.
  // It will automatically record the action duration and appear in PerfHerder as a new subtest.
  // It also creates markers in the profiler so that you can better inspect this action in
  // profiler.firefox.com.
  test.done();

  await testTeardown();
};

If your measure is not simply the time spent by an asynchronous call (for instance computing an average, counting things…) there is a lower level helper called logTestResult which will directly log a value. See this example.

Test runner

If you need to dive into the internals of the DAMP runner, most of the logic is in testing/talos/talos/tests/devtools/addon/content/damp.js.

How to name your test and register it?

If a new test file was created, it needs to be registered in the test suite. To register the new test, add it in damp-tests.js. This file acts as the manifest for the DAMP test suite.

If your are writing a test executing against Simple and Complicated documents, your test name will look like: (simple|complicated).${tool-name}.${test-name}. So for our example, it would be simple.inspector.click and complicated.inspector.click. For independent tests that don’t use the Simple or Complicated documents, the test name only needs to start with the tool name, if the test is specific to that tool For the example, it would be inspector.click.

In general, the test name should try to match the path of the test file. As you can see in damp-tests.js this naming convention is not consistently followed. We have discrepancies for simple/complicated/custom tests, as well as for webconsole tests. This is largely for historical reasons.

You will see that tests are split across different subsuites: damp-inspector, damp-other and damp-webconsole. The goal of this split is to run DAMP tests in parallel in CI, so we aim to keep them balanced in terms of number of tests, and mostly running time. Add your test in the suite which makes the most sense. We can add more suites and rearrange tests in the future if needed.

How to run your new test?

You can run any performance test with this command:

./mach talos-test --suite damp --subtest ${your-test-name}

By default, it will run the test 25 times. In order to run it just once, do:

./mach talos-test --suite damp --subtest ${your-test-name} --cycles 1 --tppagecycles 1

--cycles controls the number of times Firefox is restarted --tppagecycles defines the number of times we repeat the test after each Firefox start

Also, you can record a profile while running the test. To do that, execute:

./mach talos-test --suite damp --subtest ${your-test-name} --cycles 1 --tppagecycles 1 --gecko-profile --gecko-profile-entries 100000000

--gecko-profile enables the profiler --gecko-profile-entries defines the profiler buffer size, which needs to be large while recording performance tests

Once it is done executing, the profile lives in a zip file you have to uncompress like this:

unzip testing/mozharness/build/blobber_upload_dir/profile_damp.zip

Then you have to open https://profiler.firefox.com/ and manually load the profile file that lives here: profile_damp/page_0_pagecycle_1/cycle_0.profile