Hunting the Shmoo

Screencasts and blog posts on workflow, productivity, tools, Mozilla and whatever else tickles my fancy.

How to Push a Custom Test Command Line to Try


The steps in this article are no longer accurate. Pushing a custom command line is easier now. See here for more details.

Have you ever wanted to see the test results of a custom command line in try? Things like –test-manifest, –shuffle or –run-slower? Now you can! The process isn’t exactly optimized for the developer use case, but neither is it really difficult to do once you know how.

Basically, you just edit the ‘testing/config/’ file. Add a new key called ‘<test_suite>_options’ and set its value to a list containing the full command line you want to pass into the test suite. Substitute ‘<test_suite>’ with the name of the test suite (e.g mochitest, reftest, xpcshell etc). It’s important to note that there is no way to simply extend the official list, doing this will replace the original list wholesale.

For example, lets say I wanted to add the –shuffle option to all mochitests, my diff would be:

diff --git a/testing/config/ b/testing/config/
--- a/testing/config/
+++ b/testing/config/
@@ -20,9 +20,15 @@ mochitests.
 You must also provide the complete command line to avoid errors. The official
 configuration files containing the default values live in:
 config = {
     # Add custom mozharness config options here
+    "mochitest_options": [
+        "--appname=%(binary_path)s", "--utility-path=tests/bin",
+        "--extra-profile-file=tests/bin/plugins", "--symbols-path=%(symbols_path)s",
+        "--certificate-path=tests/certs", "--autorun", "--close-when-done",
+        "--console-level=INFO", "--setpref=webgl.force-enabled=true", "--shuffle",
+    ],

There are a few limitatons to this system I want to call out.

  1. You need to know the official options to use as a baseline for your changes. You can inspect official command lines by looking at the mozharness repo: Note: Android doesn't currently use mozharness so this trick is not yet possible on Fennec.
  2. There is no way to specify separate command lines for different platforms. For example the mochitest command lines for B2G and desktop are very different. There is no way to specify a custom command line for one without breaking the other. To get around this you'll just need to do two separate try pushes with two separate command lines.
  3. Certain options (like --this-chunk and --total-chunks) might confuse the automation and have unintended consequences. For example specifying '--this-chunk 1' will tell every job (even chunks 2+) to run the first chunk.

In conclusion, there are many things we can do to make this easier for developers. The current solution was really implemented to allow tree specific mozharness config values, the fact that you can push custom command lines to try was just a useful side effect. I hope in the future someone will make the developer story around this less confusing, but I’m not sure how high of a priority it is at the moment.


Leave a comment

Your comment has been submitted and will be published once it has been approved.