I read a great post by Aleksis Tulonen Brainstorming Test Ideas With Developers and wanted to share it with you.

The ideas in the post provide a very efficient way to:

  1. increase the robustness and thoroughness of testing,
  2. prioritize among different testing ideas (both from a relative importance standpoint and from a relative timing perspective)
  3. surface new testing ideas that would not otherwise be considered
  4. provide some useful risk mitigation protection against the reality that "all models are wrong; some models are useful" problem

I'd suggest also scheduling a debrief with the same testers and developers a few weeks or months after the code was deployed.

At that time take a look at photos of the pre-testing Post It notes and a list of defects that slipped past testing and ask

  1. "what extra tests would we need to have added to have found these defects?"
  2. "what specific inputs did we forget to include?",
  3. would techniques such as pairwise testing have been beneficial?
  4. what areas did we spent a lot of effort on that did not result in learning much?
  5. what lessons learned should we incorporate for next time?

The idea of delibrately examining your software development and testing practices will be familar to those using agile retrospectives. The power of continually improving the development practices used withing the organization is hard to appreciate but it is immense. The gains compound over time so the initial benefits are only a glimpse of what can be achieve by continuing to iterate and improve.

Related: Pairwise and Combinatorial Software Testing in Agile Projects - Applying Toyota Kata to Agile Retrospectives - Create a Risk-based Testing Plan With Extra Coverage on Higher Priority Areas

By: John Hunter and Justin Hunter on Dec 15, 2016

Categories: Agile, Software Testing, Testing Strategies

At Hexawise we aim to improve the way software is tested. Achieving that aim requires not only providing our clients with a wonderful software tool (which our customers say we’re succeeding at) but also a commitment from the users of our tool to adopt new ways of thinking about software testing.

We have written previously about our focus on the importance of the values Bill Hunter (our founder's father) to Hexawise. That has led us to constantly focus on how maximize the benefits our customers gain using Hexawise. This focus has led us to realize that our customers that take advantage of the high-touch training services and ongoing expert test design support on demand that we offer often realize unusually large benefits and roll out usage of Hexawise more quickly and broadly than our customers who acquire licenses to Hexawise and try to “get the tool and make it available to the team.”

We are now looking for someone to take on the challenge of helping our clients succeed. The principles behind our decision to put so much focus on helping our customers succeed are obvious to those that understand the thinking of Bill Hunter, W. Edwards Deming, Russel Ackoff etc. but they may seem a bit odd to others. The focus of this senior-level position really is to help our customers improve their software testing results. It isn't just a happy sounding title that has no bearing on what the job actually entails.

The person holding this position will report to the CEO and work with other executives at Hexawise who all share a commitment to delighting our customers and improving the practice of software testing.

Hexawise is an innovative SaaS firm focused on helping large companies use smarter approaches to test their enterprise software systems. Teams using Hexawise get to market faster with higher quality products. We are the world’s leading firm in our niche market and have a growing client base of highly satisfied customers. Since we launched in 2009, we have grown both revenues and profits every year. Hexawise is changing the way that large companies test software. More than 100 Fortune 500 companies and hundreds of other smaller firms use our industry leading software.

Join our journey to transform how companies test their software systems.

Hexawise office

Description: VP of Customer Success

In the Weeks Prior to a Sale Closing

  • Partner with sales representatives to conduct virtual technical presentations and demonstrations of our Hexawise test design solution.

  • Clearly explain the benefits and limitations of combinatorial test design to potential customers using language and concepts relevant to their context by drawing upon your own “been there, done that” experiences of having successfully introduced combinatorial test design methods in multiple similar situations.

  • Identify and assess business and technical requirements, and position Hexawise solutions accordingly.

Immediately Upon a New Sale Closing

  • Assess a new client’s existing testing-related processes, tools, and methods (as well as their organizational structure) in order to provide the client with customized, actionable recommendations about how they can best incorporate Hexawise.

  • Collaborate with client stakeholders to proactively identify potential barriers to successful adoption and put plans in place to mitigate / overcome such barriers.

  • Provide remote, instructor-led training sessions via webinars.

  • Provide multi-day onsite instructor-led training sessions that: cover basic software test design concepts (such as Equivalence Class Partitioning, the definition of Pairwise-Testing coverage, etc.) as well as how to use the specific features of Hexawise.

  • Include industry-specific and customer-specific customized training modules and hands-on test design exercises to help make the sessions relevant to the testers and BA’s who attend the training sessions.

  • Collaborate with new users and help them iterate, improve, and finalize their first few sets of Hexawise-generated software tests.

  • Set rollout and adoption success criteria with clients and put plans in place to help them achieve their goals.

Months After a New Sale Closing

  • Continue to engage with customers onsite and virtually to understand their needs, answer their test design questions, and help them achieve large benefits from test optimization.

  • Monitor usage statistics of Hexawise clients and proactively reach out to clients, as appropriate, to provide proactive assistance at the first sign that they might be facing any potential adoption/rollout challenges.

  • Collaborate with stakeholders and end users at our clients to identify opportunities to improve the features and capabilities of Hexawise and then collaborate with our development team to share that feedback and implement improvements.

Required Skills and Experience

We are looking for a highly-experienced combinatorial test design expert with outstanding analytical and communication skills to provide these high touch on-boarding services and partner with our sales team with prospective clients.

Education and Experience

  • Bachelor’s or technical university degree.

  • Deep experience successfully introducing combinatorial test design methods on multiple different kinds of projects to several different groups of testers.

  • Set rollout and adoption success criteria with multiple teams and put plans in place to achieve them.

  • Minimum 5 years in software testing, preferably at a IT consulting firm or large financial services firm.

Knowledge and Skills

  • Ability to present and demonstrate capabilities of the Hexawise tool, and the additional services we provides to our clients beyond our tool.
  • Exhibit excellent communication and presentation skills, including questioning techniques.
  • Demonstrate passion regarding consulting with customers.
  • Understand how IT and enterprise software is used to address the business and technical needs of customers.
  • Demonstrate hands-on level skills with relevant and/or related software technology domains.
  • Communicate the value of products and solutions in terms of financial return and impact on customer business goals.
  • Possess a solid level of industry acumen; keeping current with software testing trends and able to converse with customers at a detailed level on pertinent issues and challenges.
  • Represents Hexawise knowledgeably, based on a solid understanding of Hexawise’s business direction, portfolio and capabilities
  • Understand the competitive landscape for Hexawise and position Hexawise effectively.
  • A cover letter that describes who you are, what you've done, and why you want to join Hexawise.
  • Ability to work and learn independently and as part of a team
  • Desire to work in a fast-paced, challenging start-up environment

Why join Hexawise?

salary + bonus; medical and dental, 401(k) plans; free parking and very slick Chapel Hill office! Opportunity to experience work with a fast-growing, innovative technology company that is changing the way software is tested.

Key Benefits:

Salary: Negotiable, but minimum of $100,000 + Commissions based upon client license renewals Benefits: Health, dental included, 401k plan Travel: Average of no more than 2-3 days onsite per week Location: Chapel Hill, NC*

*Working from our offices would be highly preferable. We might consider remote working arrangements for an exceptional candidate based in the US.

Apply for the VP of Customer Success position at Hexawise.

By: John Hunter on May 12, 2016

Categories: Hexawise, Career, Software Testing, Lean, Customer Success, Agile

We have thousands of users and we're growing by hundreds of new users a month. We don't have a way to accurately breakdown our users into Agile vs Waterfall users but I know from conversations with Hexawise users that many users are using Hexawise with great results on Agile projects.

Agile projects involve short sprints in which new features are added to existing functionality. These sprints are often a couple weeks long. In such situations, it is important to be able to create tests quickly. Hexawise is extremely good at accomplishing exactly that.

Let me explain by way of an example. A Hexawise user named Sandeep asked how Hexawise could be used to create tests for an insurance ratings engine. I was able to create a sample set of 37 2-way tests within 20 minutes.

Now lets imagine that a sprint creates 2 new features that we'd like to test in combination with the existing test inputs in the existing plan. Let's assume "Feature 1" is a new ability of the application to handle input coming in from three sources: (1) web, (2) agent, and (3) call center. Let's assume "Feature 2" is a new ability of the application to handle Group Discounts. The Group Discount parameter will have 3 categories of values as well: (N/A) for people who aren't able to claim them, "Private company," and "Government / Public."

Added the new features, and created another, completely different set of tests that achieve 100% 2-way coverage. Each value in the new feature is tested in at least one test with every other value in the plan. It took less than 4 minutes to create this new set of tests incorporating the new features. It is an extremely attractive benefit of using Hexawise that is excellent for both Waterfall projects (which inevitably have late-breaking requirements changes) as well as Agile projects (where late-breaking feature additions are expected to happen as part of the process).

You want to keep testing documentation light, you want to minimize the amount of time you spend in selecting and documenting tests, and it is helpful to achieve as much coverage in as little time as possible and minimize the risk that you'll accidentally forget to test things that you meant to test (which can be easier to do if you have light test cases with enough detail in them to maximize your odds of testing the important stuff without accidentally omitting tests). Hexawise helps you achieve all of thise objectives. The example shows, as clearly and concisely as I'm able to explain it, how you can use Hexawise to create a small set of powerful tests within four minutes (from 1.7 trillion possible tests) that are well suited to test new features developed in an Agile project iteration.


Related: What is Agile? What is not Agile? - Why isn't Software Testing Performed as Efficiently and Effecively as it could be? - Cem Kaner: Testing Checklists = Good / Testing Scripts = Bad?

By: Justin Hunter on Oct 19, 2012

Categories: Agile, Hexawise tips, Pairwise Software Testing, Software Testing

An unusually hectic work-schedule has been keeping me hopping lately. I returned this weekend from a great two-week trip to the UK in which I visited with 5 testing teams using our Hexawise tool to design test cases for applications being used in two banks, a consulting and systems integration firm, a grocery store chain, and a telecoms company.

Every product manager worth his or her salt will tell you it is a good idea to go meet with customers, listen to them, and watch them as they use your application. Even though everyone I know agrees with this, I find it difficult to make happen as regularly as I would like to. This trip provided me with a reminder of how valuable in-depth customer interactions can be. The two weeks of on-site visits with testing teams proved to be great way to: (a) reconnect with customers, (b) get actionable input about what users like / don't like about our tool, (c) identify new ways we can continue to refine our tool, and even (d) understand a couple unexpected ways teams are using it.

Bret Petticord's tweets on "What is Agile?" / "not Agile?" prompted me to write this quick post. I like them a lot.




When we first created our Hexawise tool, we followed the 4 steps Bret lays out in his description of "What is Agile?" My experience in the UK over the last two weeks was the start of one of many "Repeat" cycles.

I admire people who can succinctly summarize wisdom into bite-sized quips like Bret achieved with his two tweets. Another guy who excels at creating sound-bites is James Carville. Love him or hate him, he has that skill in spades. When I watched the movie "War Room," I felt like I was watching the "master of the sound-bite" in his element. Me? I'm more of a rambling, meandering, verbose communicator. I've just taken 332 words and a screen shot with Bret's tweets when all I set out to do in starting to write this post was to share Bret's 32 words with you.

By: Justin Hunter on Feb 25, 2010

Categories: Agile, Combinatorial Testing, Hexawise test case generating tool

lessons from car manufacturing-20090826-1718522


Tony Baer from Ovum recently wrote a blog post titled: Software Development is like Manufacturing which included the following quotes.

"More recently, debate has emerged over yet another refinement of agile – Lean Development, which borrows many of the total quality improvement and continuous waste reduction principles of [lean manufacturing](http://www.lean.org/WhatsLean/. Lean is dedicated to elimination of waste, but not at all costs (like Six Sigma. Instead, it is about continuous improvement in quality, which will lead to waste reduction....

In essence, developing software is like making a durable good like a car, appliance, military transport, machine tool, or consumer electronics product.... you are building complex products that are expected to have a long service life, and which may require updates or repairs."

Here are my views: I see valid points on both sides of the debate. Rather than weigh general high-level pro's and cons, though, I would like to zero in on what I see as an important topic that is all-too-often missing from the debate. Specifically, Design of Experiments has been central to Six Sigma, Lean Manufacturing, the Toyota Production System, and Deming's quality improvement approaches, and is equally applicable to software development and testing, yet adoption of Design of Experiments methods in software design and testing remains low. This is unfortunate because significant benefits consistently result in both software development and software testing when Design of Experiments methods are properly implemented.

What are Design of Experiments Methods and Why are they Relevant?

In short, Design of Experiments methods are a proven approach to creating and managing experiments that alter variables intelligently between each test run in a structured way that allows the experimenter to learn as much as possible in as few experiments as possible. From wikipedia: “Design of experiments, or experimental design, (DoE) is the design of all information-gathering exercises where variation is present, whether under the full control of the experimenter or not. Often the experimenter is interested in the effect of some process or intervention (the “treatment”) on some objects (the “experimental units”).”

Design of Experiments methods are an important aspect of Lean Manufacturing, Six Sigma, the Toyota Production System, and other manufacturing-related quality improvement approaches/philosophies. Not only have Design of Experiments methods been very important to all of the above in manufacturing settings, they are also directly relevant to software development. By way of example, W. Edwards Deming, who was extremely influential in quality initiatives in manufacturing in Japan and the U.S. was an applied statistician. He and thousands of other highly respected quality executives in manufacturing, including Box, Juran and Taguchi (and even my dad), have regularly used Design of Experiments methods as a fundamental anchor of quality improvement and QA initiatives and yet relatively few people who write about software development seem to be aware of the existence of Design of Experiments methods.

What Benefits are Delivered in Software Development by Design of Experiments-based Tools?

Application Optimization applications, like Google’s Website Optimizer are a good example of Design of Experiments methods can deliver powerful benefits in the software development process. It allows users to easily vary multiple aspects of web pages (images, descriptions, colors, fonts, colors, logos, etc.) and capture the results of user actions to identify which combinations work the best. A recent YouTube multi-variate experiment (e.g., and experiment created using Design of Experiment methods) shows how they used the simple tool and increased sign-up rates by 15.7%. The experiment involved 1,024 variations.

What Benefits are Delivered in Software Testing by Design of Experiments-based Tools

In addition, software test design tools, like the Hexawise test design tool my company created, enable dramatically more efficient software testing by automatically varying different elements of use cases that are tested in order to achieve an optimal coverage. Users input the things in the application they want to test, push a button and, as in the Google Web Optimizer example, the tool uses DoE algorithms to identify how the tests should be run to maximize efficiency and thoroughness. A recent IEEE Computer article I contributed to, titled "Combinatorial Testing" shows, on average, over the course of 10 separate real-world projects, tester productivity (measured in defects found per tester hour) more than doubled, as compared to the control groups which continued to use their standard manual methods of test case selection: http://tinyurl.com/nhzgaf

Unfortunately, Design of Experiments methods – one of the most powerful methods in Lean Manufacturing, Six Sigma, and the Toyota Production System – are not yet widely adopted in the software development industry. This is unfortunate for two reasons, namely:

  1. Design of Experiments methods will consistently deliver measurable benefits when implemented correctly, and

  2. Sophisticated new tools designed with very straightforward user interfaces make it easier than ever for software developers and testers to begin using these helpful methods.

By: Justin Hunter on Aug 25, 2009

Categories: Agile, Design of Experiments, Efficiency, Lean, Multi-variate Testing, Software Testing, Software Testing Efficiency