TestOps: catching a new trend in performance management

Earlier this week, I wrote a breezy introduction to “DevOps” for those in need of a few executive-level references. “Break through the silos” is the emphasis of this attempt to reunite Development and Operations after decades of having them specialize and grow apart. DevOps also has the “automation attitude” which regards software as a product improved through such industrial engineering techniques as integration of design and manufacturing. DevOps is often advertised as a natural way to extend “agility” to a wider scope within the organization.

I mentioned that, in my own career, I’ve battled against silos that also isolate such other domains as Training and Support. Sometimes there are good reasons to keep distance between or even subordinate these departments to Operations; certainly there are few organizations which put them on the same level during planning.

A tiny but growing number, though, are exploring “TestOps”, which, at its most aggressive, claims that “[t]esters are uniquely qualified to lead the way to a more nimble enterprise.” TestOps remains so small that there is no Wikipedia entry for it as I write this, although I expect that status to change any day; it’s a common term of art within Microsoft and a few other large industry players.

Part of the impetus behind TestOps is the perception that it applies particularly well to provision of software services, as opposed to products. This naturally ties in to the growing importance of cloud computing. Most dramatically, as one commentator expresses the contrast, “On premise you can always scale up, improve the storage i/o, decrease latency by changing the network, or any other infrastructure tweaks available. On the public cloud the only tool that you have is the ability to test continuously in a production environment.” In this view, organizations that choose the advantages of the cloud must also build in performance management practices from the first day.

Success with TestOps initiatives depends on a couple of elements that other commentators occasionally leave implicit: first, the information flow needs to be two-way. Not only do testers in TestOps teams raise alarms when tests fail, but they also benefit from intimate daily involvement with normal operations. “Perpetual testing” gives them insight into how services should work which makes it possible to improve the tests themselves.

Also, the main reason that TestOps exists in a way that DocumentationOps or TrainingOps doesn’t yet is that testers have deep experience with automation. Any attempt to improve process through a large span of an organization relies on a mentality which knows how to make that process explicit and rework it. Testers are good at that.

All these characteristics imply there are circumstances where TestOps is not likely to thrive. Quality-of-service (QoS) measurements, for instance, are more immediately meaningful to financial traders than to advertising “creatives”. If you’re in any field with objective success criteria, though, and especially if the cloud is the basis for at least part of your infrastructure, it’s time for you to explore how to make the most of TestOps principles.

Special thanks to my colleague Matt Heusser who has helped clue me in to TestOps perspectives.