# Whaddya Mean "No Possible Value"?

### What Does it Mean?

One of the more common support inquiries we receive is when a Hexawise generated test case includes "No possible value" for a parameter. The first time you see this, it can be a bit unclear what it means and what you can do to address it.

A "no possible value" in a test case is telling you the test case is providing coverage for a needed pair in some other parameters, and in light of that needed pair your invalid and married pairs are then leaving then no value allowed for the parameter with "no possible value". That sounds confusing, but an example is much easier to understand.

### An Example

Let's say we have a test plan with just 3 parameters, each with 2 values:

Fruit: Apple, Pear Car: Toyota, Dodge Dog: Collie, Mutt

And let's further suppose we have 2 invalid pairs:

if Fruit = Apple then Car cannot = Toyota if Car = Dodge then Dog cannot = Mutt

This all seems simple, but a hidden problem lurks in this simple setup. To create 2-way coverage, Hexawise will ensure you've paired every parameter value with every other parameter value (unless a constraint says it shouldn't be paired), which in this case means that Hexawise will necessarily pair Fruit as Apple with Dog as Mutt in at least one test case, since that pairing could be the source of a bug. You probably already see the problem!

In the test case that has Fruit as Apple and Dog as Mutt we need to have a value for the Car parameter. You can't have Car as Toyota, because Apple can't be paired with Toyota, and you can't have Car as Dodge, because Mutt can't be paired with Dodge. So what value can Hexawise provide for Car in this test case? It has no value to provide, so it provides "no possible value".

That's why you can get test cases with "no possible value". Sometimes you can leave them be, sometimes you might want to introduce a "N/A" value for a parameter, and sometimes your invalid and married pairs may need a bit of adjusting. Generally, given the real context of your actual test plan, it is clear which path to take to resolve them.

### Dealing With "No Possible Value"

Let's use a more realistic flight reservation example, to show how the context of your real tests indicates what you should do to resolve instances of "no possible value".

Our flight reservation example test plan has these 5 parameters:

Destination: USA, China, Australia Class: First, Business, Economy Reserve a Car: Yes, No Type of Car: Luxury, Economy Payment Method: Credit Card, Frequent Flier Miles, Upgrade Coupons

Without any constraints, we're going to get test cases that pair Reserve a Car as No with Type of Car as Luxury. This makes no sense, and even if a tester were just to ignore it, it hides the fact that the nonsensical test case may be the only test case where we attempt to pay for a Luxury car with Frequent Flier Miles. If this pairing led to a bug, we'd miss the bug if we just ignored the car portion of the generated test case, so it's much better to use constraints to eliminate having Type of Car as Luxury when Reserve a Car is No.

We have options for how to constrain this, but let's suppose we create the following 2 uni-directional married pairs:

When Type of Car = Luxury then Reserve a Car = Yes When Type of Car = Economy then Reserve a Car = Yes

We've taken both our values for Type of Car and married them to Reserve a Car as Yes. Hexawise is still duty-bound to pair Reserve a Car as No with things like Destination as USA and Payment Method as Credit Card. For these test cases there won't be a valid value for Type of Car, so it will get "no possible value". This leads us to our first option for dealing with no possible values.

Option 1: Do nothing! There really is no possible value in that case, and you're OK with it.

If we use option 1, the "no possible value" is intentional. Option 1 is the simplest, but not always our best option. Not every "no possible value" is intentional, many in fact are unintentional, and are a result of inconsistent or missing constraint logic. A "no possible value" therefore tends to be a "quality smell" in a test plan. Meaning... you're not sure something is rotten when you see it, but it sure smells like something might be rotten. So that brings us to another option for handling intentional no possible values.

Option 2: Introduce a "Not Applicable" value.

We can add a third value to Type of Car.

Type of Car: Luxury, Economy, N/A

Now, whenever Reserve a Car is No there will be just 1 value left for Hexawise for Type of Car which is N/A so all those cases of "no possible value" will become "N/A". Problem fixed? Almost, but not quite! Hexawise is now duty-bound to pair Type of Car as N/A with Reserve a Car as Yes. To avoid this new nonsense, we add an invalid pair:

When Reserve a Car = Yes then Type of Car cannot = N/A

Option 1 and option 2 work when you discover the source of the "no possible value" is intentional. But what about when you discover it's not intentional. In that case, the "quality smell" has led us to something that truly is rotten!

Let's suppose we have Class as Business only on international, not domestic flights. Easy enough. Let's add an invalid pair to reflect this business rule.

When Destination = USA then Class cannot = Business

Let's also suppose that an Upgrade Coupon can only be used for Business class, not Economy or First Class. Also easy. Let's add a uni-directional married pair for this:

Have you spotted the trouble yet? Hexawise is duty-bound to pair Destination as USA with Payment Method as Upgrade Coupon, since their could be a bug caused by that pairing, and if there is, you want to be sure to find it. In the test case that has this pairing, Hexawise is going to have "no possible value" for Class!

Option 3: Correct your constraint logic.

In this case, our constraint logic is incomplete. We need an invalid pair:

When Destination = USA then Payment Method cannot = Upgrade Coupon

This relieves Hexawise of its duty to pair Destination as USA with Payment Method as Upgrade Coupon and eliminates the "no possible value".

### Mysterious "No Possible Value"

For cases of an unitentional "no possible value" where you need to use option 3 to correct an inconsistency in your constraint logic, it's not always clear where the inconsistency is. In fact, it can be downright mysterious. The more constrained your plan is, the harder it can be to find the root cause.

My best advice for tracking down the cause of a "no possible value" in complicated cases is to stay patient and stick with it. Practice makes you better at it. Here are some additional tips:

• Look at each test case with "no possible values" in isolation, don't try to analyze more than 1 at once
• Start with the test case with the fewest number of "no possible values" and then fix the issues your analysis turns up, then regenerate your tests and repeat, until you have none left
• You need pencil and paper! Write out the test case that you're analyzing on paper, then do the analysis from the "Define Inputs" page where it's easiest to see all the possible values for each parameter
• Take advantage of the hover highlighting of applicable value pairs in the Define Inputs screen. Collapse every section in the left panel except the Paired Values (constraints) so you can see as many as value pairings at one time as possible
• If you get completely stuck, reach out to a more experienced colleague or ask Hexawise for help

Doing this analysis can be tricky, so having Hexawise do the analysis for you instead is a great idea. We'd like to have a tooltip with an explanation for the no possible value. Since it can be tricky for us humans to determine why a particular no possible value case exists in complicated cases, the algorithm to do the same is also tricky to get right, but we're working toward having a tooltip to explain the cause of the "no possible value", and are going to be very excited when this feature is ready. We might accidentally create a self-aware AI that enslaves the humans in the process though. If we do, don't say we didn't warn you.

In addition to an automated explanation of where the "no possible value" came from, we also know that prevention is better than a cure, so we're working to identify these cases as soon as the constraints are entered so we can prompt you in the moment to correct the issue. We already do this for some of the simpler causes of "no possible value", you may have been prompted by Hexawise already. We're working on Hexawise being able to detect more complicated causes so we can prevent more of these cases early.

In the end, always keep in mind that every "no possible value" is simply a conflict between a parameter value pairing Hexawise is duty-bound to provide to ensure you get 100% coverage, and the constraints you've applied to the plan in the form of married pairs and invalid pairs. With that in mind, have fun "no possible value" hunting.

By: Sean Johnson on May 1, 2014

# Exploring a Cave or Software

Justin posted this to the Hexawise Twitter feed

The parallels to software testing are easy to see. In order to learn useful information we need to understand what the purpose of the visit to the cave is (or what we are seeking to learn in software testing). Normally the most insight can be gained by adjusting what you seek to learn based on what you find. As George Box said

Always remember the process of discovery is iterative. The results of each stage of investigation generating new questions to answered during the next.

Some "software testing" amounts to software checking or confirming that a specific known result is as we expect. This is important for confirming that the software continues to work as expected as we make changes to the code (due to the complexity of software unintended consequences of changes could lead to problems if we didn't check). This can, as described in the tweet, take a form such as 0:00 Step 1- Turn on flashlight with pre-determined steps all the way to 4:59 Step N- Exit But checking is only a portion of what needs to be done. Exploratory testing relies on the brain, experience and insight of a software tester to learn about the state of the software; exploratory testing seeks to go beyond what pre-defined checking can illuminate. In order to explore you need to understand the aim of the mission (what is important to learn) and you need to be flexible as you learn to adjust based on your understanding of the mission and the details you learn as you take your journey through the cave or through the software. Exploratory software testing will begin with ideas of what areas you wish to gain an understanding of, but it will provide quite a bit of flexibility for the path that learning takes. The explorer will adjust based on what they learn and may well find themselves following paths they had not thought of prior to starting their exploration.

By: John Hunter on Apr 8, 2014

# Using Hexawise to Export Values for Gherkin (Given, When --> Then)

Some of those using Hexawise use Gherkin as their testing framework. Gherkin is based on using a given [a], when [b] --> then [c] format. The idea is this helps make communication clear and make sure business rules are understood properly. Portions of this post may be a bit confusing for new Hexawise users, links are provided for more details on various topics. But, if you don't need to create output for Gherkin and you are confused you can just skip this post.

A simple Gherkin scenario: Making an ATM withdrawal

``````Given a regular account
And the account was originally opened at Goliath National
And the account has a balance of \$500
When using a Goliath National Bank
And making a withdrawal of \$200
Then the withdrawal should be handled appropriately
``````

Hexawise users want to be able to specify the parameters (used in given and when statements) and then import the set of Hexawise generated test cases into a Gherkin style output.

In this example we will use Hexawise sample test plan (Gherkin example), which you can access in your Hexawise account.

I'll get into how to export the Hexawise created test plans so they can be used to create Gherkin data tables below (we do this ourselves at Hexawise).

In the then field we default to an expected value of "the withdrawal should be handled appropriately." This is something that may benefit from some explanation.

If we want to provide exact details on exactly what happens on every variation of parameter values for each test script those have to be manually created. That creates a great deal of work that has very little value. And it is an expensive way to manage for the long term as each of those has to be updated every time. So in general using a "behaves as expected" default value is best and then providing extra details when worthwhile.

For some people, this way of thinking can be a bit difficult to take in at first and they have to keep reminding themselves how to best use Hexawise to improve efficiency and effectiveness.

To enter the default expected value mouse-over the final step in the auto scripts screen. When you mouse over that step you will see the "Add Expected Results" link. Click that and add your expected result text.

The expect value entered on the last step with no conditions (the when drop down box is blank) will be the default value used for the export (and therefor the one imported into Gherkin).

In those cases when providing special notes to tester are deemed worth the extra effort, Hexawise has 2 ways of doing this. In the event a special expected value exists for the particular conditions in the individual test case then the special expected value content will be exported (and therefore used for Gherkin).

Conditional expected results can be entered using the auto scripts feature.

Or we can use the requirements feature when we want to require a specific set of parameter values to be tested. If we chose 2 way coverage (the default, pairwise coverage) every pair of parameter values will be tested at least once.

But if we wanted a specific set of say 3 exact parameter values ([account type] = VIP, [withdrawal ATM] = bank-owned ATM, [withdrawal amount] = \$600 then we need to include that as a requirement. Each required test script added also includes the option to include an expected result. The sample plan includes a required test case with those parameters and an expected result of "The normal limit of \$400 is raised to \$600 in the special case of a VIP account using a Goliath National Bank owned ATM."

So, the most effective way to use Hexawise to create a pairwise (or higher) test plan to then use to create Gherkin data tables will be to have the then case be similar to "behaves as expected." And when there is a need for special expected results details to use the auto script or requirements features to include those details. Doing so will result the expected result entered for that special case being the value used in the Gherkin table for then.

When you click auto script button the test are then generated, you can download them using the export icon.

You will download a zip file that you can then unzip to get 2 folders with various files. The file you want to use for this is the combinations.txt file in the csv directory.

The Ruby code we use to convert the commas to pipes | used for Gherkin is

`!/usr/bin/env ruby`
`require 'csv'`
`tests = CSV.read("combinations.csv")`
`table = []`
`tests.each do |test|`
`table << "| " + test[1..-1].join(" | ") + " |\n"`
`end`
`IO.write("gherkin.txt", table.join())`

Of course, you can use whatever method to convert the format you wish, this is just what we use. See this explanation for a few more details on the process.

Now you have your Gherkin file to use however you wish. And as the code is changed over time (perhaps adding parameter value options, new parameters, etc.) you can just regenerate the test plan and export it. Then convert it and the updated Gherkin test plan is available.

By: John Hunter on Mar 27, 2014

# "Grapefruit Juice Bugs" - A New Term for a Surprisingly Common Type of Surprising Bugs

Coining a New Term

I'm coining a new term today, "grapefruit juice bugs."

My inspiration for this term is a blog post in the New York Times that David Pogue wrote. I was fascinated by the post and it got me to thinking about a particular kind of bugs in software that are more common than most people may realize. You could say that these bugs are surprisingly common. In fact, if you wanted to be more precise, you could even say that this term applies to a specific type of "surprisingly common type of surprising bugs." Let me explain.

There's something about the chemical makeup of grapefruit juice that makes it interact with our biology and a large number of different drugs in ways which result in dangerous conditions. For example, certain drugs lose their effectiveness dramatically when interacting with grapefruit juice which can have life-threatening consequences. Other times, the interactions with grapefruit juice can dramatically increase a drug's potency. This can result in "safe doses" becoming very unsafe.

Grapefruit Is a Culprit in More Drug Reactions

The 42-year-old was barely responding when her husband brought her to the emergency room. Her heart rate was slowing, and her blood pressure was falling. Doctors had to insert a breathing tube, and then a pacemaker, to revive her.

They were mystified: The patient’s husband said she suffered from migraines and was taking a blood pressure drug called verapamil to help prevent the headaches. But blood tests showed she had an alarming amount of the drug in her system, five times the safe level.

Did she overdose? Was she trying to commit suicide? It was only after she recovered that doctors were able to piece the story together.

“The culprit was grapefruit juice,” said Dr. Unni Pillai, a nephrologist in St. Louis, Mo. ...

The previous week, she had been subsisting mainly on grapefruit juice. Then she took verapamil, one of dozens of drugs whose potency is dramatically increased if taken with grapefruit. In her case, the interaction was life-threatening.

Last month, Dr. David Bailey, a Canadian researcher who first described this interaction more than two decades ago, released an updated list of medications affected by grapefruit. There are now 85 such drugs on the market, he noted, including common cholesterol-lowering drugs, new anticancer agents, and some synthetic opiates and psychiatric drugs, as well as certain immunosuppressant medications taken by organ transplant patients, some AIDS medications, and some birth control pills and estrogen treatments. ... Under normal circumstances, the drugs are metabolized in the gastrointestinal tract, and relatively little is absorbed, because an enzyme in the gut called CYP3A4 deactivates them. But grapefruit contains natural chemicals called furanocoumarins, that inhibit the enzyme, and without it the gut absorbs much more of a drug and blood levels rise dramatically.

For example, someone taking simvastatin (brand name Zocor) who also drinks a small 200-milliliter, or 6.7 ounces, glass of grapefruit juice once a day for three days could see blood levels of the drug triple, increasing the risk for rhabdomyolysis, a breakdown of muscle that can cause kidney damage.

So what do interactions between grapefruit juice and drugs have to do with software testing?

Like grapefruit juice's impact on prescription drugs, software testing involves critical interactions between different parts of the system. And risks exist when these different parts interact with one another. This is true whether you're talking about "large parts" interact in System Testing or "small parts" interact in Unit Testing.

Interactions between things are a very rich source of bugs in software. As anyone who has heard the infernal phrase "works on my machine" can tell you, software features and functions often work perfectly fine in many usage scenarios, hardware and software configurations , etc. - only to fail to work in ever-so-slightly different situations.

The difference between plain old every-day "Dual-Mode Faults" and "Grapefruit Juice Bugs"

A dual-mode fault occurs whenever two test inputs must both be present to trigger a defect. Most software testers start encountering them quite frequently within days of starting their jobs. Some examples:

• This "buy" button works fine. Except when the customer is a "new user." (First, action = "click on the buy button" and Second, customer = "new user")

• Transaction prices for share purchases are calculated correctly. Except when denominated in Japanese Yen. (First, Action = "sell shares" and Second, Currency = "Japanese Yen")

Like grapefruit juice's impact on prescription drugs, software testing involves critical interactions between different parts of the system. And risks exist when these different parts interact with one another. This is true whether you're talking about "large parts" interact in System Testing or "small parts" interact in Unit Testing.

While all grapefruit juice bugs are dual-mode faults, not all dual-mode faults are Grapefruit Juice Bugs:

• Grapefruit juice bugs have got to have a little of the element of surprise in them. When you explain them to a developer, their first reaction should be "Huh? How is that even possible?" or at least "Hmmm... That's odd. Let me investigate."

• Anything along the lines of "This feature usually works, except in IE6, when..." is almost definitely not a grapefruit juice bug. Problematic interactions with IE6 are an incredibly common type of dual-mode fault, not a surprising one.

Whenever you hear "works on my machine" replies to your bug reports, and it takes a while for the issue to be replicated, odds are pretty good that a grapefruit juice bug might be involved.

Here's an example of an especially surprising grapefruit juice bug. This excerpt from Apple's online help files that the company posted after users of the original iPad complained about problems with Wi-Fi connectivity. Certain screen brightness settings were causing problems with the Wi-Fi signals. I'm not even to begin to guess how one would have anything to do with the other.

How to identify grapefruit juice bugs during your testing?

What is a tester to do when faced with more possible potential grapefruit juice bugs than he can handle using traditional methods?

If you're a software tester trying to do your best to determine whether a feature or function in your System Under Test will work "on everyone's machine," you've got a nightmare on your hands . Really nasty combinatorial explosions arise when you consider all of the possible combinations that would be required to test multiple hardware options, multiple software options, multiple usage scenarios, multiple test data inputs (and multiple combinations of the test data itself), multiple ways in which users enter data, and all of the rest of the "stuff that could vary" when people use your application. If you take the time to think expansively about the possible variations in a medium-sized applications, Quadrillions of possible tests often result.

While not eating grapefruit and not drinking grapefruit juice might be wise if you are taking drugs, there is rarely, if ever, such an easy method for eliminating the possibility of negative results due to software interactions. Refusing to support IE 6 in order to avoid the disproportionate number of grapefruit juice-like problematic interactions associated with IE6 would be as close as you could come in the world of software.

Design of Experiments-based test design methods can help testers come to grips with this challenge. Orthogonal array software testing (often referred to as OATS or simply OA testing) is a test design strategy that allows us to efficiently detect bugs created by interactions within the system. Orthogonal array software testing is based on the principles of multifactor designed experiments as first explored by Sir RA Fisher.

Design of Experiments-based test design methods are very-closely related to pairwise testing (AKA allpairs testing, all pairs testing, and pairwise-testing). Any of these test design strategies will allow a software tester to quickly generate a set of tests that includes tests for every single pair of test inputs.

This approach to test design often has multiple advantages, including faster test creation, more varied test scenarios, 100% coverage of all potential dual-mode faults (including hard-to-predict grape-fruit juice bugs), and often a smaller resulting set of tests that will be quicker to execute. Having said that, it is by no means a magical silver bullet. This approach to test design requires test designers with above average analytical abilities to identify the appropriate Parameters and Values for their system under test; this is sometimes easier said than done because it requires a new mindset from test designers.

Software testers can take solace that the challenges of software testing, while significant, are simple when compared to trying to understand the effects of drug interactions in people.

Combinatorial testing can look at bugs created by the interaction between multiple (3, 4, 5, 6...) variables. So if there was a bug that didn't get triggered just by using Chrome on Windows but it would get triggered if you also tried to replace an existing photo in your profile with a new profile photo into your profile (test idea number 3), then pairwise testing might not catch it. Pairwise test design would create a set of tests that would include at least one test for each of these pairs:

• Chrome & Windows and

• Chrome & replace photo and

• Windows & replace photo, but...

A set of pairwise might not fail to test for the specific combination of all three of those test inputs in the same test. With the use of combinatorial test design approaches, you could create test plans with 100% coverage for 3 way interactions and be sure that all 3-way interactions or 4-way interactions are covered. When you create sets of 3-way tests, 4-way tests, 5-way tests, and 6-way tests though, you'll quickly discover that the number of tests required starts to balloon.

Hexawise allows you to create test plans with the coverage interactions you desire. This allows you to create sets of tests from 2-way up all the way up to phenomenally-thorough 6-way sets of tests. In fact, it even lets you generate clever sets of risk-based tests that will, say, prioritize comprehensive 4-way coverage on 4 sets of Parameter Values while ensuring only pairwise coverage of the other, lower-priority, interactions in your system under tests. Hexawise also lets you create mixed strength test plans so if you have certain factors that you are very concerned about and want to provide coverage for more possible interactions you can set the interaction levels for those at a higher level.

By: Justin Hunter on Feb 11, 2014

# What Hexawise Delivered in 2013

Hexawise has had another great year and we owe it to you, our users. Thank you! As a result of your input and word-of-mouth recommendations, in 2013, the Hexawise test design tool:

• Became even easier to use,

• Introduced lots of new practical instructional content, and

• Doubled usage again.

### Ease-of-Use Enhancements

Instructional Guides for Hexawise Features
We’ve added illustrated step-by-step instructions describing how to use Hexawise features.

Find them at help.hexawise.com. For our advanced features, like creating risk-based test plans, auto-generating complete scripts, and using Value Expansions, we’ve gone beyond “how?” to explain “why?” you would want to use these features.

Practical Test Design Tips
Want to see tips and tricks for creating unusually powerful tests? Want to learn about common mistakes (and how to avoid them)? Want to understand whether pairwise test design really works in practice? These topics and more can now be found at training.hexawise.com.

Frog-Powered Self-Paced Learning

Want to become a Hexawise guru? Listen to the frogs. If you complete the achievements recommended by your friendly amphibian guides, you will level up from a Novice to Practitioner to Expert to Guru.

You’ll complete two kinds of achievements on your way to guru-ness. To complete some achievements you only need to use certain Hexawise features. Completing the other achievements requires learning important test design concepts and demonstrating that you understand them. The frogs, ever-ready to guide you towards test design mastery, will greet you immediately upon logging into your account.

### Powerful New Features

Recently-added features that will make you a more powerful and speedy test designer, include:

Coverage of Specific High-Priority Scenarios
You can now force specific scenarios to appear in the tests you generate using the Requirements feature.

Requirements Traceability
Requirements traceability is easier to manage than ever with the Requirements feature.

Generation of Detailed Test Scripts
The Auto-Scripting feature allows you to automatically transform sets of optimized test conditions into test scripts that contain detailed instructions in complete sentences.

Auto-Population of Expected Results in Test Scripts
If you want to, you can even automatically generate rules-based Expected Results to appear as part of your test steps by using the Expected Results feature.

To find out more about these features Hexawise added in 2013, please check out these cool slides: "Powerful New Hexawise Features".

### Public Recognition and Rapid Growth

Kind Words
As a five-year old company working as hard as we can to make Hexawise the best damn tool it can be, hearing input from you keeps us motivated and headed in the right direction. Once a week or so, we hear users say nice things about our tool. Here are some of the nice things you guys said about Hexawise this past year:

“Working coaching session with customer today. Huge data/config matrix was making them weep. Stunned silence when I showed them @Hexawise :)”

-Jim Holmes (@aJimHolmes)

“That would be @Hexawise & combinatorial testing to the rescue once again. #Thanks”

-Vernon Richards (@TesterFromLeic)

Freaking awesome visualisation of test data coverage. Kind courtesy of @Hexawise at Moolya!”

-Moolya Testing (@moolyatesting)

“Using @Hexawise combinatorial scenarios for e-commerce basket conditions. Team suitably impressed by speed and breadth of analysis. #Win”

-Simon Knight (@sjpknight)

“Just discovered Hexawise today, brilliant tool for creating test cases based on coverage of many variables.”

“This changes everything.”

-Dan Caseley (@Fishbowler)

Using Hexawise is one of the highest ROI software development practices in the world.

-Results, paraphrased, of independent study by industry expert Capers Jones and colleagues.

Rapid Growth
Throughout 2013, Hexawise continued to be piloted and adopted at new companies week after week. Hexawise is currently being used to generate tests at:

More than 100 Fortune 500 firms More than 2,000 smaller firms

New offices
Having moved into our new offices in October, the Hexawise team now gets together to do all our best stuff at this swanky new location in Chapel Hill, North Carolina.

### What's Next?

Constant Improvements
We keep a public running tally of all of our enhancements. As you can see, we’re making improvements at a rate of more than once a week.

Want us to Add a New Feature?
If you have an additional feature, please let us know. We listen. There’s not much that makes us happier than finding ways to improve Hexawise.

Our growth has been almost purely due to word-of-mouth recommendations from users like you. So if you find Hexawise to be helpful, please help spread the word. We appreciate it more than you know.

You can even let your friends in on a bit of a testing industry secret: while company-wide Hexawise licenses start at \$50,000 per year, we allow the first five users to use fully-featured Hexawise accounts at no cost!

Thank you for all of your help and support this past year. And have a great 2014!

By: Justin Hunter on Jan 28, 2014

# Retaining Simplicity While Providing Power Users Advanced Features

A common mistake software companies make is creating products where features built for advanced users overwhelm and confuse average users. At Hexawise we have focused on creating a great experience for average and new users while providing advanced users powerful options.

How to Avoid a Common Product Mistake Many Teams Make by Mark Suster

The single biggest mistake most product teams make is building technology for what they believe the user would want rather than what the actual end-user needs. … My philosophy is simply. You design your product for the non-technologist. The “Normal.”

Give people fewer options, fewer choices to make. Give them the bare essentials. Design products for your mom. … power users will always find the features they need in your product even if they’re hidden away. Give them a configure button somewhere. From there give them options to soup up the product and add the complexity that goes with newfound power.

Make sure you read his full post and subscribe to his blog, the posts are clearly-written, pragmatic, and insightful.

Our experiences at Hexawise match the points the post makes. We've designed our web-based tool with Jason Fried/37 Signals-inspired "KISS" (Keep It Simple Stupid) design principles in mind. Our interesting debates about how to add (or whether to add) new features have often been based on the exact tensions you mention here. "Power users want new features" vs. "... but users love our tool precisely because it's easy-to-use and it doesn't have 'feature bloat'."

We've experimented with the suggestion you raise here (e.g., rather than say "no" to every advanced user request, we build them in hidden places for advanced users without distracting our regular users). Results have been good so far.

The Bulk add/bulk edit feature in Hexawise is an example of a powerful feature that is implemented in a way that doesn't interfere with the ease of use for those that don't take advantage of this feature.

For us, there are few things more important to our tool's success in the marketplace than our ability to get the balance right between "uncluttered simplicity that everyone wants" vs. "with the powerful capabilities that advanced users want."

There are natural tensions between the two opposing design goals. Sean Johnson (Hexawise CTO) is the strongest advocate for simplicity. John and Justin, Hexawise's CEO, love simplicity, and understand the important of simplicity for usability, but also find ourselves pushing for advanced features.

I am a strong believer in the simplicity with the option for power users to access advanced features. Turning this concept into practice isn't easy. Thankfully Sean has the mindset and skill to make sure we don't sacrifice simplicity and usability while providing power features to power users at Hexawise. I was also lucky enough to have another amazing co-worker, Elliot Laster, at a previous employer that provided this valuable role. One of the tricks to making this work is hiring a great person with the ability to make it a reality (it requires deep technical understanding, an understanding of usability principles and a deep connection to real users, all of which Sean and Elliot have to an amazing degree).

Getting the balance right is one of the most important things a software company can do. We've tried really hard to get it right. We're probably overdue for formal usability testing of basic functionality. Reading blogs like Mark's is useful for new ideas and also for the push to do what you know you should do but seem to keep putting off.

By: John Hunter and Justin Hunter on Jan 21, 2014

# Software Tester Value - Inefficient Labor Market

The process used to hire employees is inefficient in general and even more inefficient for knowledge work. Justin Hunter, Hexawise CEO, posted the following tweet:

The labor market is highly inefficient for software testers. Many excellent testers are undervalued relative to average testers. Agree?

The tweet sparked quite a few responses:

I think there are several reasons for why the job market is inefficient in general, and for why it is even more inefficient for software testing than for most jobs.

• Often, how companies go about hiring people is less about finding the best people for the organization and more about following a process that the organization has created. Without intending to, people can become more more concerned about following procedural rules than in finding the best people.

• The hiring process is often created much like software checking, a bunch of simple things to check - not because doing so is actually useful but because simple procedural checks are easy to verify. So organizations require a college degree (and maybe even require a specific major). And they will use keywords to select or reject applicants. Or require certification or experience with a specific tool. Often the checklist used to disqualify people contains items that might be useful but shouldn't be used as barriers but it is really easy for people that don't understand the work to apply the rules in the checklist to filter the list of applicants.

• It is very hard to hire software testers well when those doing the hiring don't understand the role software testing should play. Most organizations don't understand, so they hire for software checkers. They, of course, don't value people that could provide much more value (software testers that go far beyond checking). The weakness of hiring without understanding the work is common for knowledge work positions and likely even more problematic for software testing due to the even worse understanding of what they should be doing compared to most knowledge workers.

And there are plenty more reasons for the inefficient market.

Here are few ideas that can help improve the process:

• Spend time to understand and document what your organization seeks to gain from new hires.

• Deemphasize HR's role in the talent evaluation process and eliminate dysfunctional processes that HR may have instituted. Talent evaluation should be done by people that understand the work that needs to get done. HR can be useful in providing guidance on legal and company-decided policies for hiring. Don't have people that can't evaluate the difference between great testers and good testers decide who should be hired or what salary is acceptable. Incidentally, years of experience, certifications, degrees, past salary and most anything else HR departments routinely use are often not correlated to the value a potential employee brings.

• A wonderful idea, though a bit of a challenge in most organizations, is to use job auditions. Have the people actually do the job to figure out if they can do what you need or not (work on a project for a couple weeks, for example). This has become more common in the last 10 years but is still rare.

• I also believe you are better off hiring for somewhat loose job descriptions, if possible, and then adjusting the job to who you hire. That way you can maximize the benefit to the organization based on the people you have. At Hexawise, for example, most of the people we hire have strengths in more than one "job description" area. Developers with strong UI skills, for instance, are encouraged to make regular contributions in both areas.

• Creating a rewarding work environment helps (this is a long term process). One of the challenges in getting great people is they are often not interested in working for dysfunctional organizations. If you build up a strong testing reputation great testers will seek out opportunities to work for you and when you approach great testers they will be more likely to listen. This also reduces turnover and while that may not seem to relate to the hiring process is does (one reason we hire so poorly is we don't have time to do it right, which is partly because we have to do so much of it).

• Having employees participate in user groups and attending conferences can help your organization network in the testing community. And this can help when you need to hire. But if your organization isn't a great one for testers to work in, they may well leave for more attractive organizations. The "solution" to this risk is not to stunt the development of your staff, but to improve the work environment so testers want to work for your organization.

Great quote from Dee Hock, founder of Visa:

Hire and promote first on the basis of integrity; second, motivation; third, capacity; fourth, understanding; fifth, knowledge; and last and least, experience. Without integrity, motivation is dangerous; without motivation, capacity is impotent; without capacity, understanding is limited; without understanding, knowledge is meaningless; without knowledge, experience is blind. Experience is easy to provide and quickly put to good use by people with all the other qualities.

Please share your thoughts and suggestions on how to improve the hiring process.

By: John Hunter on Jan 14, 2014

Categories: Checklists, Software Testing, Career

# Software Testing Carnival #5

The Hexawise Software Testing blog carnival focuses on sharing interesting and useful blog posts related to software testing.

• Using mind-mapping software as a visual test management tool by Aaron Hodder - "I want to be able to give and receive as much information as I can in the limited amount of time I have and communicate it in a way that is respectful of others' time and resources. These are my values and what I think constitutes responsible testing."

• Healthcare.gov and the Tyranny of the Innocents by James Bach - "Management created the conditions whereby this project was 'delivered' in a non-working state. Not like the Dreamliner. The 787 had some serious glitches, and Boeing needs to shape that up. What I’m talking about is boarding an aircraft for a long trip only to be told by the captain 'Well, folks it looks like we will be stuck here at the gate for a little while. Maintenance needs to install our wings and engines. I don’t know much about aircraft building, but I promise we will be flying by November 30th. Have some pretzels while you wait.'"

Rope bridge in the jungle by Justin Hunter

• Software Testers Are Test Pilots by John Hunter - "Software testers should be test pilots. Too many people think software testing is the pre-flight checklist an airline pilot uses."

• Where to begin? by Katrina Clokie - "Then you need to grab your Product Owner and anyone else with an interest in testing (perhaps architect, project manager or business analyst, dependent on your team). I'm not sure what your environment is like, usually I'd book an hour meeting to do this, print out my mind map on an A3 page and take it in to a meeting room with sticky notes and pens. First tackle anything that you've left a question mark next to, so that you've fleshed out the entire model, then get them to prioritise their top 5 things that they want you to test based on everything that you could do."

• Being a Software Tester in Scrum by Dave McNulla - "Pairing on development and testing strengthens both team members. With people crossing disciplines, they improve understanding of the product, the code, and what other stakeholders find important."

• Stop Writing Code You Can’t Yet Test by Dennis Stevens - "The goal is not to write code faster. The goal is to produce valuable, working, testing, remediated code faster. The most expensive thing developers can do is write code that doesn’t produce something needed by anyone (product, learning, etc). The second most expensive thing developers can do is write code that can’t be tested right away."

• Is Healthcare.gov security now fixed? by Ben Simo - "I am very happy that the most egregious issue was immediately fixed. Others issues remain. The vulnerabilities I've listed above are defects that should not make it to production. It doesn't take a security expert or “super hacker” to exploit these vulnerabilities. This is basic web security. Most of these are the kinds of issues that competent web developers try to avoid; and in the rare case that they are created, are usually found by competent testers."

• Embracing Chaos Testing Helps Create Near-Perfect Clouds - "Chaos Monkey works on the simple premise that if we need to design for high availability, we should design for failure. To design for failure, there should be ways to simulate failures as they would happen in real-world situations. This is exactly what a Chaos Monkey helps achieve in a cloud setup.
Netflix recently made the source code of Chaos Monkey (and other Simian Army services) open source and announced that more such monkeys will be made available to the community."

• Bugs in UK Post Office System had Dire Consequences - "A vocal minority of sub-postmasters have claimed for years that they were wrongly accused of theft after their Post Office computers apparently notified them of shortages that sometimes amounted to tens of thousands of pounds. They were forced to pay in the missing amounts themselves, lost their contracts and in some cases went to jail. Second Sight said the Post Office's initial investigation failed at first to identify the root cause of the problems. The report says more help should have been given to sub-postmasters, who had no way of defending themselves."

• Traceability Matrix: Myth and Tricks by Adam Howard - "And this is where we get to the crux of the problem with traceability matrices. They are too simplistic a representation of an impossibly complex thing. They reduce testing to a series of one to one relationships between intangible ideas. They allow you to place a number against testing. A percentage complete figure. What they do not do is convey the story of the testing."

• Six Tips for Your Software Testing Career by John Hunter - "Read what software testing experts have written. It’s surprising how few software testers have read books and articles about software testing.Here are some authors (of books, articles and blogs) that I've found particularly useful..."

By: John Hunter on Dec 16, 2013

Categories: Software Testing

# Best Customer Interaction in the History of Hexawise?

So, we've been experimenting with a live customer support feature in our tool lately. We're rolling out the live chat support on a beta basis to see how useful our users find it.

The motivations for creating it were two-fold. Our first motivation was the Mayday button that Amazon Kindle announced recently. How cool is that, right? Live support available on demand any time you want it! Ingenius.

Our second motivation for building a live chat support feature into our tool is that - while software test designers consistently tell us that they find Hexawise's features to be really easy to use - new users will often encounter test design questions as they start using the tool. We wanted to be available instantly to collaborate with them - and help them address questions in real time, like: "How should I be thinking about different ways of defining equivalence classes?" "Given what I'm trying to test in my system, how much detail is too much in this context?" etc. We wanted to be there to help users answer them. We're obsessive about customer service. Having the opportunity to have our expert test designers be a click away from every user of our tool every time they encounter a question. That's just too good an opportunity to pass up.

Early indications of how useful this service is to users are extremely promising. Users are telling us it is an amazingly helpful service. And, while we were worried that we might start to feel too stretched with tons of user questions to answer, we haven't felt that way at all. Interactions have been at manageable volumes. We've found them to be really positive. Many of the interactions have helped us learn about ways to improve our tool and/or how we can make certain test design concepts easier for Hexawise users to understand. Often, customers will click on a "call me" button to talk through questions live by phone rather than by typing back and forth. I'm glad this next conversation was done with keyboards.

This conversation happened about 45 minutes ago. Everyone at Hexawise headquarters is still smiling broadly. It made all of our days and stands apart from the rest. Enjoy!

(17:15:25) Visitor Hi

(17:15:40) Sean Johnson Hey

(17:15:44) Sean Johnson What's up?

(17:15:51) Visitor Hi Sean!

(17:15:57) Visitor Hey, I created 308 test cases

(17:16:02) Sean Johnson k

(17:16:02) Visitor out of a possible 18 trillion

(17:16:07) Sean Johnson nice!

(17:16:11) Visitor I consolidated all my user stories in 6 sprints total

(17:16:13) Visitor to 1 test set

(17:16:14) Visitor which is fine

(17:16:28) Visitor but I noticed from test case # 100 plus to 308

(17:16:37) Visitor most of my test cases are now 'any value'

(17:16:48) Visitor I was wondering if there's an option to force hexawise to pick a value for me

(17:16:52) Visitor but I don't there is

(17:16:56) Visitor but that would be a good enhancement

(17:17:05) Sean Johnson ha! are you spying on me?

(17:17:09) Visitor for 'any value' you can have the app just pick a random one

(17:17:10) Visitor LOL

(17:17:11) Visitor Nope

(17:17:14) Sean Johnson seriously… that's what I'm working on right now

(17:17:19) Visitor NO WAY!

(17:17:20) Sean Johnson what are the odds?

(17:17:24) Visitor O M G

(17:17:26) Sean Johnson yes way

(17:17:33) Visitor I've been meaning to provide that feedback 2 weeks ago

(17:17:39) Visitor but never took the time!

(17:17:46) Visitor I WOULD LOVE TO HAVE THAT FEATURE!

(17:17:49) Visitor Oh

(17:17:51) Visitor in the mean time

(17:17:56) Visitor my testers workaround

(17:18:00) Visitor is to print the test plan

(17:18:17) Visitor and just pick the values randomly from the value expansion list and input parameter list

(17:18:26) Visitor AWESOME Sean!

(17:18:31) Visitor Well let me know when it's available

(17:18:41) Sean Johnson that's really crazy

(17:18:41) Sean Johnson well… I guess I'm working on the right thing!

(17:18:42) Sean Johnson Will tomorrow be soon enough for you?

(17:18:42) Sean Johnson :-)

(17:18:43) Sean Johnson for now… it's going to be hidden

(17:18:43) Sean Johnson you'll add ?full=true

(17:18:43) Sean Johnson to the end of the URL

(17:18:57) Sean Johnson and that'll force Hexawise to fill in the any_values

(17:19:03) Visitor that's AWESOME!

(17:19:06) Visitor I will do the workaround

(17:19:09) Visitor OMG, you made my day!

(17:19:13) Visitor THANKS A TON!

(17:19:15) Visitor :-)

(17:19:18) Sean Johnson I'll send you note this evening or in the morning when it's live on [your company's Hexawise instance]

(17:19:23) Visitor I am so happy

(17:19:24) Visitor LOL

(17:19:28) Visitor Thanks so much

(17:19:32) Sean Johnson thanks for chatting! made my day to know I picked the right next priority.

(17:19:32) Visitor this will make my life easier

(17:19:38) Visitor Oh yeah, totally

(17:19:38) Sean Johnson excellent.

(17:19:58) Visitor I honestly think I'm not the only one who will appreciate this enhancement

(17:20:01) Visitor you guys are the best!

(17:20:03) Visitor Thanks so much

(17:20:05) Sean Johnson :-) we try

(17:20:15) Visitor You all do an amazing job

(17:20:21) Visitor this tool is the best

(17:20:21) Sean Johnson look for an email from me shortly

(17:20:26) Visitor will do

(17:20:27) Visitor thanks!

(17:20:33) Sean Johnson thanks!

We've been working hard for the past ~5 years building and continuously improving Hexawise so that it will be a tool that software test designers find to be extremely useful and - equally importantly - a tool that software test designers will find enjoyable to use. It is hard to put into words how satisfying it is to see an interaction like this one.

By: Justin Hunter on Dec 4, 2013

Categories: Uncategorized

Since creating Hexawise, I've worked with executives at companies around the world who have found themselves convinced in the value of pairwise testing. And then they need to convince their organization of the value.

They often follow the following path: first thinking "pairwise testing is a nice method in theory, but not applicable in our case" then "pairwise is nice in theory and might be applicable in our case" to "pairwise is applicable in our case" and finally "how do I convince my organization."

In this post I review my history helping convince organizations to try and then adopt pairwise, and combinatorial, software testing methods.

About 8 years ago, I was working at a large systems integration firm and was asked to help the firm differentiate its testing offerings from the testing services provided by other firms.

While I admittedly did not know much about software testing then but by happy coincidence, my father was a leading expert in the field of Design of Experiments. Design of Experiments is a field that has applicability in many areas (including agriculture, advertising, manufacturing, etc.) The purpose of Design of Experiments is to provide people with tools and approaches to help people learn as much actionable information as possible in as few tests as possible.

I Googled "Design of Experiments Software Testing." That search led me to Dr. Madhav Phadke (who, by coincidence, had been a former student of my father). More than 20 years ago now, Dr. Phadke and his colleagues at ATT Bell Labs had asked the question you're asking now. They did an experiment using pairwise test design / orthogonal array test design to identify a subset of tests for ATT's StarMail system. The results of that testing effort were extraordinarily successful and well-documented.

Shortly after doing that, while working at that systems integration firm, I began to advocate to anyone and everyone who would listen that designing approach to designing tests promised to be both (a) more thorough and (b) require (in most but not all cases) significantly fewer tests. Results from 17 straight projects confirmed that both of these statements were true. Consistently.

Repeatable Steps to Confirm Whether This Approach Delivers Efficiency and Thoroughness Improvement (and/or document a business case/ROI calculation)

How did we demonstrate that this test design approach led to both more thorough testing and much more efficient testing? We followed these steps:

1. Take an existing set of 30 - 100 existing tests that had already been created, reviewed, and approved for testing (but which had not yet been executed).

2. Using the test ideas included in those existing tests, design a set of pairwise tests (often approximately half as many tests as were in the original set). When putting your tests together, if there are particular, known, high-priority scenarios that stakeholders believe are important to test, it is important to make sure that that you "force" your pairwise test generator to include such high-priority scenarios.

3. Have two different testers execute both sets of tests at the same time (e.g., before developers start fixing any defects that are uncovered by testers executing either set of tests)

Document the following:

• How long did it take to execute each set of tests?

• How many unique defects were identified by each set of tests?

• How long did it take to create and document each set of tests?*

*This third measurement was usually an estimate because a significant number of teams had not tracked the amount of time it took to create the original set of tests.

The results in 17 different pairwise testing "bake-off" projects conducted at my old firm included:

• Defects found per tester hour during test execution: when pairwise tests were used, more than twice as many defects were found per tester hour

• Total defects found: pairwise tests as many or more defects in every single project (despite the fact that in almost every case there were significantly more tests in the each original set of tests)

• Defects found by pairwise tests but missed by traditional tests: a lot (I forget the exact number)

• Defects found by traditional tests but missed by pairwise tests: zero

• Amount of time to select and document tests: much less time required when a pairwise test generator was used (As mentioned above, precise measurements were difficult to gather here)

More recent project benefits have included these:

Those experiences - combined with the realization that many Fortune 500 firms were starting to try to implement smarter test design methods to achieve these kinds of benefits but were struggling to find a test design tool that was user-friendly and would integrate into their other tools - led me to the decision to create Hexawise.

Once the testing the value of pairwise software testing at a specific organization it is very common to find the proponent of taking advantage of pairwise testing advantages to find themselves saying:

I have already elaborated some test plans that would save us up to 50% effort with that method. But now my boss and other colleagues are asking me for a proof that these pairwise test cases suffice to make sure our software is running well.

In that case, my advice is three-fold:

First, appreciate how your own thinking has evolved and understand that other people will need to follow a similar journey (and that others won't have as much time to devote as you have had to experience learnings first-hand).

When I was creating Hexawise, George Box, a Design of Experiments expert with decades of experience explaining to skeptical executives how Design of Experiments could deliver transformational improvements to their organizations' efficiency and effectiveness, told me "Justin, you'll experience three phases of acceptance and acceptance will happen more gradually than you would expect it to happen. First, people will tell you 'It won't work.' Next, they'll say "It won't work here." Eventually, he said with a smile, they'll tell you 'Of course this works. I thought of it first!'

When people hear that you can double their test execution productivity with a new approach, they won't initially believe you. They'll be skeptical. Most people you're explaining this approach to will start with the thought that "it is nice in theory but not applicable to what I'm doing." It will take some time and experience for people to understand and appreciate how dramatic the benefits are.

Second, people will instinctively be dismissive of pairwise testing case study after case study after case study that show how effective this approach has been for testers in virtually all types of industries and all types and phases of testing. George Box was right when he predicted that people will often respond with 'It won't work here.' Sometimes it is hard not to smile when people take that position.

Case in point: I will be talking to a senior executive at a large capital markets firm soon about how our tool can help them transform the efficiency and effectiveness of their testing group. And I can introduce them to a client of ours that is using our test design tool extensively in every single one of their most important IT projects. Will that executive take me up on my offer? I hope so, but based on past experience, I suspect odds are good that he'll instead react with 'Yes, yes, sure, if companies were people, that company would be our company's identical twin, but still... It won't work here.'

Third, at the end of the day, the most effective approach I have found to address that understandable skepticism and to secure organizational-level buy-in and commitment is through gathering hard, indisputable evidence on multiple projects that the approach works at the company itself through a bake-off approach (e.g., following those four steps outlined above. A few words of advice though.

My proposed approach isn't for the faint of heart. If you're working at a large company with established approaches, you'll need patience and persistence.

Even after you gather evidence that this approach works in Business Unit A, and B and C, someone from Business Unit D will be unconvinced with the compelling and irrefutable evidence you have gathered and tell you 'It won't work here. Business Unit D is unique.' The same objections may likely arise with results from "Type of Testing" A, B, and C. As powerful and widely-applicable as this test design approach is, always remember (and be clear with stakeholders) that it is not a magical silver bullet.

James Bach raises several valid limitations with using this approach. In particular, this approach won't work unless you have testers who have relatively strong analytical skills driving the test design process. Since pairwise test case generating tools are dependent upon thoughtful test designers to identify appropriate test inputs to vary, this approach (like all test design approaches) is subject to a "garbage in / garbage out" risk.

Project leads will resist "duplicating effort." But unless you do an actual bake-off stakeholders won't appreciate how broken their existing process is. There's inevitably far more wasteful repetition hidden away in standard tests than people realize. When you start reporting a doubling of tester productivity on several projects, smart managers will take notice and want to get involved. At that point - hopefully - your perseverance should be rewarded.

Some benefits data and case studies that you might find useful:

If you can't change your company, consider changing companies

Lastly, remember that your new-found skills are in high demand whether or not they're valued at your current company. And know that, despite your best efforts and intentions, your efforts might not convince skeptics. Some people inevitably won't be willing to take the time to understand. If you find yourself in a situation where you want to use this test design approach (because you know that these approaches are powerful, practical, and widely-applicable) but that you don't have management buy-in, then consider whether or not it would be worth leaving your current employer to join a company that will let you use your new-found skills.

Most of our clients, for example, are actively looking for software test designers with well developed pairwise and combinatorial test design skills. And they're even willing to pay a salary premium for highly analytical test designers who are able to design sets of powerful tests. (We publicize such job openings in the LinkedIn Hexawise Guru group for testers who have achieved "Guru" level status in the self-paced computer-based-training modules in the tool).

By: Justin Hunter on Nov 21, 2013