Hunting the Shmoo

OUTDATED

The steps in this article are no longer accurate. Pushing a custom command line is easier now. See here for more details.

Have you ever wanted to see the test results of a custom command line in try? Things like –test-manifest, –shuffle or –run-slower? Now you can! The process isn’t exactly optimized for the developer use case, but neither is it really difficult to do once you know how.

Basically, you just edit the ’testing/config/mozharness_config.py’ file. Add a new key called ‘<test_suite>_options’ and set its value to a list containing the full command line you want to pass into the test suite. Substitute ‘<test_suite>’ with the name of the test suite (e.g mochitest, reftest, xpcshell etc). It’s important to note that there is no way to simply extend the official list, doing this will replace the original list wholesale.

For example, lets say I wanted to add the –shuffle option to all mochitests, my diff would be:

diff --git a/testing/config/mozharness_config.py b/testing/config/mozharness_config.py
--- a/testing/config/mozharness_config.py
+++ b/testing/config/mozharness_config.py
@@ -20,9 +20,15 @@ mochitests.
 
 You must also provide the complete command line to avoid errors. The official
 configuration files containing the default values live in:
     https://hg.mozilla.org/build/mozharness/configs
 """
 
 config = {
     # Add custom mozharness config options here
+    "mochitest_options": [
+        "--appname=%(binary_path)s", "--utility-path=tests/bin",
+        "--extra-profile-file=tests/bin/plugins", "--symbols-path=%(symbols_path)s",
+        "--certificate-path=tests/certs", "--autorun", "--close-when-done",
+        "--console-level=INFO", "--setpref=webgl.force-enabled=true", "--shuffle",
+    ],
 }

There are a few limitatons to this system I want to call out.

In conclusion, there are many things we can do to make this easier for developers. The current solution was really implemented to allow tree specific mozharness config values, the fact that you can push custom command lines to try was just a useful side effect. I hope in the future someone will make the developer story around this less confusing, but I’m not sure how high of a priority it is at the moment.