Peptest: A new harness for testing responsiveness
While responsiveness is one of the main goals for Firefox this quarter, we still don't quite have the means to measure and test our progress towards this goal. The good news is that there are, and have been for some time, several efforts to fix this problem. Back in June, Ted wrote some event tracing instrumentation that gives us a reasonable idea of when the browser becomes unresponsive. This event tracer is already being used by some Talos tests which gives us a good general idea of whether or not Firefox is more or less responsive than it was previously. What it doesn't give us is a method for developers to write their own tests and determine whether a specific action or feature they are working on is causing unresponsivness.
Peptest is designed for the missing use case. Namely, it can be used to automate user interactions in the browser and determine whether those actions are causing unresponsiveness. This may be useful for creating a suite of responsiveness regression tests, or for developers working on a responsiveness related feature or fix. The Peptest harness is designed to be lightweight (so as not to interfere with results), simple to run and easy to write tests for.
Currently, it is possible to run tests locally on your machine, though I could potentially add features or change any aspect of the harness. I've also been working on a Mozharness script in bug 692091 so we can run tests automatically for tinderbox builds.
Finally, I'd like to say: I need feedback! The requirements of this harness have been very vague from the outset. I've been doing my best to interpret the requirements in a way that makes sense, but I'm still kind of flying blind so to speak. What I mean is, I'm not sure what developers want and/or need. I'm also not sure how useful what I've thrown together so far will be. So if you have any ideas or general comments, please ping ahal on irc, or email email@example.com and I'd be very grateful.