Hexawise reduces regression suite by 95% at Fidelity Investments

Kathleen Poulsen
Lead Software Engineer in Test
Hexawise contact person

Fidelity Investments is one of the largest asset management companies in the world, with nearly $2.7 trillion in assets under management.

Problem

A large set of regression tests for a “Know Your Customer Rules” application had continued to grow without providing adequate coverage for the system under test - leading to unknown, and inadequate, test coverage and defects found in production. Production defects in this type of application exposed Fidelity to signifiant monetary and legal risk.

Despite a regression suite of more than 12,000 tests, the project still suffered from under-testing due to widespread repetition of tests in the suite, along with scattered, outdated documentation. These factors made it difficult to understand the coverage achieved and led to poor test coverage.

Challenges

Defects in production

Repetitive test cases

Scattered testing documentation

"I can now have a discussion with other teams about my test coverage and give them real numbers... I love that."

Solution

The team used Hexawise to generate a drastically-smaller set of more powerful, more thorough tests to replace the repetitive manually-selected sets of scenarios previously being executed.

To ensure the scenarios indicated by business analysts as “high value” were always included, these scenarios were entered into Hexawise as requirements.

Outcomes

95% reduction in tests

Business risks accounted for

100% pairwise test coverage

Simplified documentation management

This project is an extremely successful example of Hexawise’s ability to pack more coverage into fewer, algorithmically-generated test cases. Not only did Hexawise eliminate the need for 95% of the original (largely redundant) tests, but Hexawise also enhanced the remaining 5% of the test cases to provide the same coverage as the removed 95%.

All high-priority scenarios were included in the set of 582 Hexawise tests - ensuring business risk was accounted for and 100% 2-way interaction coverage was achieved. On top of that, Hexawise import and export capabilities allowed a single spreadsheet to serve as the ‘Source of Truth’ document for the entire project.

In addition to this project, Fidelity Investments has used Hexawise to optimize many existing test suites: 40%-75% reductions have been common.