When we built our test case design tool, we thought:

  1. When used thoughtfully, this test design approach is powerful.

  2. The kinds of things going on "underneath the covers," like the optimization algorithms and combinatorial coverage risk confusing and alienating potential users before they get started.

  3. Designing an easy-to-use interface is absolutely critical. (We kept in the "delighter" in our MVP release.)

  4. As we have continued to evolve, we've kept (a) focusing on design, and (b) having learned that customers don't find the tool itself hard to use but they do sometimes find the test design principles confusing, we've focused on adding self-paced training modules into the tool (in addition to training and help web resources and Hexawise TV tutorials.

 

hexawise-levels

View of the screen showing a user's progress through the integrated learning modules in Hexawise

 

Our original hope was that if we focused on design (and iterated it based on user feedback), that Hexawise would "sell itself." And that users would tell their friends and drive adoption through word of mouth. That's exact what we've seen.

We did learn that we needed to include more help getting started with the challenges of learning the test design principles needed to successful create tests. We knew that was a bit of a hurdle but we have seen it was more of a hurdle than we anticipated so we have put more resources into helping get over than initial resistance. And we have increased the assistance to clients on how to think differently about creating test plans.

A recent interview with Andy Budd takes an interesting look at the role of design in startups.

Andy Budd: Design is often one of the few competitive advantages that people have. So I think it’s important to have design at the heart of the process if you want to cross that chasm and have a hugely successful product. ... Des Traynor: If a product is bought, it means that users are so attracted to it, that they’ll literally chase it down and queue up for it. If it’s sold, it means that there are people on commission trying to force it into customer’s hands. And I find design can often be the key difference between those two types of models for business.

Andy: There’s a lot of logic in spending less money on marketing and sales, and more money on creating a brilliantly delightful product. If you do something that people really, really want, they will tell their friends.

 

As W. Edward Deming said in Out of the Crisis

Profit in business comes from repeat customers, customers that boast about your product and service, and that bring friends with them

 

The benefits of delighting customers so much that they help promote your product are enormous. This result is about the best one any business can hope for. And great design is critical in creating a user experience that delights users so much they want to share it with their friends and colleagues.

The article also discusses one of the difficult decision points for a software startup. The minimal viable product (MVP) is a great idea to help test out what customers actually want (not just what they say they want). This MVP is a very useful concept (pushed into many people's consciousness by the popularity of lean startup and agile software development).

MVP is really about testing the marketplace. The aim is to get a working product in people's hands and to learn from what happens. If the user experience is a big part of what you are offering (and normally it should be) a poor user experience (Ux) on a MVP is not a great way to learn about the market for what you are thinking of offering.

In my opinion, adding features to existing software can be tested in a MVP way with less concern for Ux than completely new software, but I imagine some would want the great Ux for this case too. My experience is that users that appreciate your product can understand the rough Ux in a mock up and separate the usefulness from the somewhat awkward Ux. This is especially true for example with internal software applications where the developers can directly interact with the users (and you can select people you know who can separate the awkward temporary Ux from the usefulness of a new feature).

 

Related: The two weeks of on-site visits with testing teams proved to be great way to: (a) reconnect with customers, (b) get actionable input about what users like / don’t like about our tool, (c) identify new ways we can continue to refine our tool, and even (d) understand a couple unexpected ways teams are using it. - Automatically Generating Expected Results for Tests in Hexawise - Pairwise and Combinatorial Software Testing in Agile Projects

By: John Hunter on Sep 17, 2013

Categories: Experimenting

tl;dr: When you have parameters that only have sensible values depending on certain conditions you should include a value like "N/A" or "Does not appear" for those parameters.

 

You can try this example out yourself using your Hexawise account. If you do not have an account yet you can create a demo account for free that lets you create effective test plans.

Let's take a simple, made up example from version 1 of a restaurant ordering system that has 3 parameters:

Entree: Steak, Chicken, Salmon
Salad: Caesar, House
Side: Fries, Green Beans, Carrots, Broccoli

Everything is just fine with our test plan for version 1, but then let's suppose the business decides that in version 2, people that order "Chicken" don't get a "Salad". Easy enough, we just make an invalid pair between "Chicken" and "Caesar" and "Chicken" and "House", correct? No, Hexawise won't let us. Why? Because then it has no value available for "Salad" to pair with "Chicken" as the "Entree".

But that's what we want! "Salad" will disappear from the order screen as soon as we select "Chicken". So there is no value. That's OK. We just need to add that as the value:

Entree: Steak, Chicken, Salmon
Salad: Caesar, House, Not Available
Side: Fries, Green Beans, Carrots, Broccoli

At this point we could create the invalid pairs between "Chicken" and "Caesar" and "Chicken" and "House", and Hexawise will allow it because there is still a parameter value, "Not Available", left to pair with "Chicken" in the "Salad" parameter.

 

If we do this though, we'll find that Hexawise will force a pairing between "Steak" and "Not Available" and "Salmon" and "Not Available". Not exactly what we wanted! So we can also add an invalid pair between "Steak" and "Not Available" and "Salmon" and "Not Available".

With these four invalid pairs, we have a working test plan for version 2, but rather than the four invalid pairs, this scenario is exactly why Hexawise has bi-directional married pairs. A bi-directional married pair between "Chicken" and "Not Available" tells Hexawise that every time "Entree" is "Chicken", "Salad" must be "Not Available" and every time "Salad" is "Not Available", "Entree" must be "Chicken". So it gives us precisely what we want for this scenario by creating just one bi-directional married pair rather than four invalid pairs.

Now let's suppose version 3 of the menu system comes out, and now there is a fourth Entree, "Pork". And "Pork", being the other white meat, also does not have a salad option:

Entree: Steak, Chicken, Salmon, Pork
Salad: Caesar, House, Not Available
Side: Fries, Green Beans, Carrots, Broccoli

When we go to connect "Entree" as "Pork" and "Salad" as "Not Available" with a bi-directional married pair, Hexawise will rightly stop us. While we can logically say that every time "Entree" is "Chicken", "Salad" is "Not Available" and every time "Entree is Pork", "Salad" is "Not Available", we can't say the reverse. It's nonsensical to say that every time "Salad" is "Not Available", "Entree" is "Chicken" and every time "Salad" is "Not Available", "Entree" is "Pork".

This is precisely why Hexawise has uni-directional married pairs. What we do in this case is create an uni-directional married pair between "Chicken" and "Not Available" which says that every time "Entree" is "Chicken", "Salad" is "Not Available", but it's not the case that every time "Salad" is "Not Available", "Entree" is "Chicken". This of course leaves us free to create a uni-directional married pair between "Pork" and "Not Available". With this design, we're back to Hexawise wanting to pair "Steak" and "Not Available" and "Salmon" and "Not Available" since our uni-directional married pairs don't prohibit that, so we need to add our invalid pairs for those two pairings.

So our final solution for version 3 looks like:

Entree: Steak, Chicken, Salmon, Pork
Salad: Caesar, House, Not Available
Side: Fries, Green Beans, Carrots, Broccoli

Uni-directional Married Pair - Entree:Chicken → Salad:Not Available Uni-directional Married Pair - Entree:Pork → Salad:Not Available Invalid Pair - Entree:Steak ↔ Salad:Not Available Invalid Pair - Entree:Salmon ↔ Salad:Not Available

Let's suppose the specifications for version 4 now hit our desks, and they specify that those that chose the "House" "Salad" get a choice of two dressings, "Ranch" or "Italian". We can then end up with a dependent value that's dependent on another dependent value. That's ok. We've got this!

Entree: Steak, Chicken, Salmon, Pork
Salad: Caesar, House, Not Available
Dressing: Ceasar, Ranch, Italian, Not Available
Side: Fries, Green Beans, Carrots, Broccoli

Uni-directional Married Pair - Entree:Chicken → Salad:Not Available
Uni-directional Married Pair - Entree:Pork → Salad:Not Available
Uni-directional Married Pair - Entree:Chicken → Dressing:Not Available
Uni-directional Married Pair - Entree:Pork → Dressing:Not Available
Bi-directional Married Pair - Salad:Caesar ↔ Dressing:Caesar
Bi-directional Married Pair - Salad:Not Available ↔ Dressing:Not Available
Invalid Pair - Entree:Steak ↔ Salad:Not Available
Invalid Pair - Entree:Salmon ↔ Salad:Not Available
Invalid Pair - Entree:Steak ↔ Dressing:Not Available
Invalid Pair - Entree:Salmon ↔ Dressing:Not Available

Hexawise tests can uncover any pair-wise defects in the identified parameters for version 4 of our hypothetical menu ordering system in just 20 tests out of a possible 192. We just saved ourselves from executing 172 extra tests or missing some defects!

 

Related: How do I create an "Invalid Pair" to prevent impossible to test for Values from appearing together? - How do I prevent certain combinations from appearing using the "Married Pair" feature? - Hexawise Tip: Using Value Expansions and Value Pairs to Handle Dependent Values

By: Sean Johnson on Sep 9, 2013

Categories: Hexawise tips, Testing Strategies

Software testing is becoming more critical as the proportion of our economic activity that is dependent on software (and often very complex software) continues to grow. There seems to be a slowly growing understanding of the importance of software testing (though it still remains an under-appreciated area). My belief is that skilled software testers will be appreciated more with time and that the opportunities for expert software testers will expand.

Here are a few career tips for software testers.

 

  • Read what software testing experts have written. It's surprising how few software testers have read books and articles about software testing.Here are some authors (of books, articles and blogs) that I've found particularly useful. Feel free to suggest other software testing authors in the comments.

James Bach

Cem Kaner

Lisa Crispin

Michael Bolton

Lanette Creamer

Jerry Weinberg

Keith Klain

James Whittaker - posts while at Google and his posts with his new employer, Microsoft

Pradeep Soundararajan

Elisabeth Hendrickson

Ajay Balamurugadas

Matt Heusser

our own, Justin Hunter

 

  • Join the community You'll learn a lot as a lurker, and even more if you interact with others. Software Testing Club is one good option. Again, following those experts (listed above) is useful; engaging with them on their blogs and on Twitter is better. The software testing community is open and welcoming. In addition, see: 29 Testers to Follow on Twitter and Top 20 Software Testing Tweeps. Interacting with other testers who truly care about experimenting, learning, and sharing lessons learned is energizing.

 

  • Develop your communication skills Communication is critical to a career in software testing as it is to most professional careers. Your success will be determined by how well you communicate with others. Four critical relationship are with: software developers, your supervisors, users of the software and other software testers.Unfortunately in many organizations, managers and corporate structures restrict your communication with some of these groups. You may have to work with what you are allowed, but if you don't have frequent, direct communication with those four groups, you won't be able to be as effective.
    Work on developing your communication skills. Given the nature of software testing two particular types of communication - describing problems ("what is it?") and explaining how significant problem is ("why should we fix it?") - are much more common than in many other fields. Learning how to communicate these things clearly and without making people defensive is of extra importance for software testers.
    A great deal of your communication will be in writing and developing your ability to communicate clearly will prove valuable to your continued growth in your career.
    Writing your own blog is a great way to further you career. You will learn a great deal by writing about software testing. You will also develop your writing ability merely by writing more. And you will create a personal brand that will grown your network of contacts. It also provides a way for those possibly interested in hiring you to learn more. We all know hiring is often done outside the job announcement - job application process. By making a name for yourself in the software testing community you will increase the chances of being recruited for jobs.

 

  • Do what you love You career will be much more rewarding if you find something you love to do. You will most likely have parts of your job you could do without but finding something you have a passion for makes all the difference in the world. If you don't have a passion for software testing, you are likely better off finding something you are passionate about and pursuing a career in that field.

Your work is going to fill a large part of your life, and the only way to be truly satisfied is to do what you believe is great work. And the only way to do great work is to love what you do. If you haven’t found it yet, keep looking. Don’t settle.

Steve Jobs

 

  • Practice and learn new techniques and ideas. James Bach provides such opportunities. Also the Hexawise tool, lets you easily and quickly try out different scenarios and learn by doing. We also provide guided training, help and webcasts explaining software testing concepts and how to apply them using Hexawise.We include a pathway that guides you through the process of learning about software testing, with learning modules, practical experience and tests along the way to make sure you learn what is important. And once you reach the highest level and become a Hexawise guru we have offer a community experience to allow all those reaching that level to share their experience and learn from each other.

 

  • Go to good software testing conferences.I've heard great things about CAST (by the Association for Software Testing), Test Bash, and Let's Test in particular. Going to conferences is energizing because while you're there, you're surrounded by the minority of people in this industry who are passionate about software testing. Justin, Hexawise founder and CEO, will be presenting at CAST next week, August 26-28 in Madison, Wisconsin.

 

Related: Software Testing and Career Advice by Keith Klain - Looking at the Empirical Evidence for Using Pairwise and Combinatorial Software Testing - Maximizing Software Tester Value by Letting Them Spend More Time Thinking

By: John Hunter on Aug 22, 2013

Categories: Software Testing

As I looked at our administrative dashboard for Hexawise.com today I was struck by how diverse our users are. I was looking at the most active users in the last week and the top 11 users were from 8 different countries: USA, France, Norway, India, Israel, Spain, Thailand and Canada. The only country with more than 1 was the USA with 2 users from Florida and 1 from Wisconsin. Brazil, Belgium and the Russian Federation were also represented in the top 25.

If you look at the top 25 users in the last month, in addition to the countries above (except Belgium) 3 more countries are represented: China, Netherlands and Malaysia.

hexawise-visitors

Visitors to the Hexawise web site in the last month by country.

 

Looking at our web site demographics the top countries of our visitors were: United States, India, Philippines, Australia, Brazil, United Kingdom, Israel, Malaysia, Italy and Netherlands. In the last month we have had visitors from 84 countries.

It is exciting to see the widespread use of Hexawise across the globe. The feedback on our upgrades included in Hexawise 2.0 have been very positive. We continue to get more and more users which makes us happy: we believe we have created a valuable tool for software testers and it is exciting to get confirmation from users. Please share your experiences with us; knowing what you like is helpful and we have made numerous enhancements based on user feedback.

 

Related: Empirical Evidence for Using Pairwise and Combinatorial Software Testing - Hexawise TV (webcasts on Hexawise and wise software testing practices) - Training material on Hexawise and software testing principles

By: John Hunter on Jul 17, 2013

Categories: Hexawise test case generating tool

The Hexawise Software Testing blog carnival focuses on sharing interesting and useful blog posts related to software testing.

 

  • T-Shaped Testers and their role in a team by Rob Lambert - "I believe that testers, actually – anyone, can contribute a lot more to the business than their standard role traditionally dictates. The tester’s critical and skeptical thinking can be used earlier in the process. Their other skills can be used to solve other problems within the business. Their role can stretch to include other aspects that intrigue them and keep them interested."

  • Testing triangles, pyramids and circles, and UAT by Allan Kelly - "Thus: UAT and Beta testing can only truly be performed by USERS (yes I am shouting). If you have professional testers performing it then it is in effect a form of System Testing.
    This also means UAT/Beta cannot be automated because it is about getting real life users to user the software and get their feedback. If users delegate the task to a machine then it is some other form of testing."

 

flower-justin

Photo by Justin Hunter, taken in South Africa.

 

  • Software Testing in Distributed Teams by Lisa Crispin - "Remote pairing is a great way to promote constant communication among multiple offices and to keep people working from home 'in the loop'.
    I often remote pair with a developer to specify test scenarios, do exploratory testing, or write automated test scripts. We use screen-sharing software that allows either person to control the mouse and keyboard. Video chat is best, but if bandwidth is a problem, audio will do. Make sure everyone has good quality headphones and microphone, and camera if possible."

  • Seven Kinds of Testers by James Bach - "I propose that there are at least seven different types of testers: administrative tester, technical tester, analytical tester, social tester, empathic tester, user, and developer. As I explain each type, I want you to understand this: These types are patterns, not prisons. They are clusters of heuristics; or in some cases, roles. Your style or situation may fit more than one of these patterns."

  • Which is Better, Orthogonal Array or Pairwise Software Testing? by John Hunter and Justin Hunter - "After more study I have concluded that: Pairwise is more efficient and effective than orthogonal arrays for software testing. Orthogonal Arrays are more efficient and effective for manufacturing, and agriculture, and advertising, and many other settings."

  • Experience Report: Pairing with a Programmer by Erik Brickarp - "We have different investigation methods. The programmer did low level investigations really well adding debug printouts, investigating code etc. while I did high level investigations really well checking general patterns, running additional scenarios etc. Not only did this make us avoid getting stuck by changing 'method' but also, my high level investigations benefited from his low level additions and vice versa."

  • I decided to evolve a faster test case by Ben Tilly - "I first wrote a script to run the program twice, and report how many things it found wrong on the second run. I wrote a second script that would take a record set, sample half of it randomly, and then run the first script.
    I wrote a third script that would take all records, run 4 copies of my test program, and then save the smaller record set that best demonstrated the bug. Wash, rinse, and repeat..."

  • Tacit and Explicit Knowledge and Exploratory Testing by John Stevenson - "It is time we started to recognise that testing is a tacit activity and requires testers to think both creativity and critically."

  • Tear Down the Wall by Alan Page - "There will always be a place for people who know testing to be valuable contributors to software development – but perhaps it’s time for all testing titles to go away?"

By: John Hunter on Jul 10, 2013

Categories: Software Testing

Here is a wonderful webcast that provides a very quick, and informative, overview of rapid software testing.

Software testing is when a person is winding around a space searching that space for important information.

 

James Bach starts by providing a definition of software testing to set the proper thinking for the overview.

Rapid software testing is a set of heuristics [and a set of skills]. Heuristics live at the border of explicit and tacit knowledge... Heuristics solve problems when they are under the control of a skilled human... It takes skill to use the heuristics effectively - to solve the problems of testing. Rapid software testing focuses on the tester... Tacit skills are developed through practice.

 

Automated software tests are useful but limited. In the context of rapid software testing only a human tester can do software testing (automated checks are defined as "software checking"). See his blog post: Testing and Checking Refined.

 

Related: People are better at figuring out interesting ideas to test. Generating a highly efficient, maximally varied, minimally repetitive set of tests based on a given set of test inputs is something computer algorithms are more effective at than a person. - Hexawise Lets Software Testers Spend More Time Focused on Important Testing Issues - 3 Strategies to Maximize Effectiveness of Your Tests

By: John Hunter on Jul 2, 2013

Categories: Software Testing, Testing Strategies

We have created the Hexawise Software Testing Glossary. The purpose is to provide some background for both general software testing tool and some Hexawise specific details.

While we continue to develop, expand and improve the Hexawise software itself we also realize that users success with using Hexawise rests not only on the tool itself but on the knowledge of those using the tool. The software testing glossary adds to the list of offerings we provide to help users. Other sources of help include sample test plans, training material (on software testing in general and Hexawise in particular), detailed help on specific topics, webcast tutorials on using Hexawise and this blog.

In Hexawise 2.0 we integrated the idea of building your expertise directly into the Hexawise application with specific actions, reading and practicums that guide users from novice to practitioner to expert to guru.

achievements

View for a Hexawise user that is currently a practitioner and is on their way to becoming an expert.

 

We aim to provide software services, and educational material, that help software testers focus their energy, insite, knowledge and time where it is most valuable while Hexawise software automates some tasks and does other tasks - that are essentially impossible for people to do manually (such as creating the most effective test plan, with pairwise, or better, coverage, given the parameters and values determined by the tester).

Our help, training and webcast resources have been found to be very useful by Hexawise users. We hope the software testing glossary will also prove to be of value to Hexawise users.

 

Related: Maximizing Software Tester Value by Letting Them Spend More Time Thinking - Hexawise Tip: Using Value Expansions and Value Pairs to Handle Dependent Values - Maximize Test Coverage Efficiency And Minimize the Number of Tests Needed

By: John Hunter on Jun 24, 2013

Categories: Hexawise test case generating tool, Hexawise tips, Software Testing

Many teams are trying to generate unusually powerful and varied sets of software tests by using Design of Experiments-based methods to generate many or most of their tests. The two most popular software test design methods are orthogonal array testing and pairwise testing. This article describes how these two approaches are similar but different and suggests that in most cases, pairwise testing is preferable.

Before advancing, it may be worth pointing out that Orthogonal Array Testing is also known as OA or OATS. Similarly, pairwise testing is sometimes referred to as all pairs testing, allpairs testing, pair testing, pair-wise testing, or simply 2-way testing. The difference between these two very similar approaches of pairwise vs. orthogonal array is that orthogonal array-based solutions require the same coverage goal that pairwise solutions do (e.g., that every pair of inputs is tested at least once) plus an additional hurdle/characteristic, that there be a uniform distribution throughout the domain.

I have studied the question of how can software testing inputs be combined most efficiently and effectively pretty steadily for the last 7 years. I started by searching the web for "Design of Experiments" and "software testing" and found references to Dr. Madhav Phadke (who, by coincidence, turns out was a former student of my father).

  • I discovered that Dr. Phadke had designed RDExpert which, although it had been primarily created to help with Research & Design projects in manufacturing settings, could also be used to select small sets of powerful test sets in software testing projects, using the Orthogonal Array-based test selection criteria.

  • I used RDExpert to create test sets (and compared those test sets against sets of tests that had been selected manually by software testers)

  • I gathered results by asking one tester to execute the manually selected tests and another tester to execute the the Orthogonal Array-based tests; the OA-based tests dramatically outperformed the manually-selected ones in terms of defects found per tester hour and defexts found overall.

So, in short, I had confirmed to my satisfaction that an OA-based test data combination strategy was far more effective than manually selecting combinations for the kinds of projects I was working on, but I was curious if other techniques worked better.

 

After more study I have concluded that:

  • Pairwise is more efficient and effective than orthogonal arrays for software testing.

  • Orthogonal Arrays are more efficient and effective for manufacturing, and agriculture, and advertising, and many other settings.

 

And we have built Hexawise as a software tool to help software producers test their software, based on what I have learned from my experience. We take full advantage of the greatly increased efficiency and effectiveness of letting testers to determine what needs to be tested and software algorithms to quickly create comprehensive test plans that provide more coverage with dramatically fewer tests.

But we also go well beyond this to create a software as a service solution that aids the software testing team with many huge advantages such as: automatically generating Expected Results in test scripts, automated importing of data from Excel or mind maps, exporting tests into other tools, preventing impossible to test for values from appearing together, and much more.

 

Why is a pairwise testing strategy better than an orthogonal array strategy?

  • Pairwise testing almost always requires fewer tests than orthogonal array-based solutions (it is possible, in some situations, for them to have an equal number of tests).

  • Remember, the reason that orthogonal array-based solutions require more tests than a pairwise solution to reach the coverage goal of testing all pairs of test conditions together in at least one test is the additional hurdle/characteristic that orthogonal array testing has, e.g., that there be a uniform distribution throughout the domain.

  • The "cost" of the extra tests (AKA experiments) is worth paying in many settings outside of the software testing industry because the results are non-binary in those tests. Someone seeking a desired darkness and gloss and luminosity and luster for a particular shade of green in the processing of film, for example, would benefit from with the information obtained from the added information gathered from orthogonal arrays.

  • In software testing, however, the added costs imposed by the the extra tests are not worth it. You're generally not seeking some ideal point in a continuum; you're looking to see what two specific pieces of data will trigger a defect when they appear in the same transaction. To identify that binary approach most efficiently and effectively, what you want is a pairwise solution (with fewer tests), not a longer list of orthogonal array-based tests.

 

Let me also add these points.

  • First, unlike some of my other views on combinatorial test design, my opinion on this narrow subject is not based on multiple empirical studies; it is based on (a) the reasoning I laid out above, and (b) a dozen or so conversations I've had with PhD's who specialize in the intersection of "Design of Experiments" and software test design, and (c) anecdotal evidence from using both methods.

  • Secondly, to my knowledge,very few, if any, studies have gathered empirical data showing benefits of pairwise solutions vs. orthogonal array-based solutions in software testing scenarios.

  • Thirdly, I strongly suspect that if you asked Dr. Phadke, he would give you his reasons for why orthogonal array-based solutions are appropriate (and even preferable) to pairwise test case selection methods for certain kinds of software projects. I have a huge amount of respect for both him and his son.

 

Time doesn't allow me to get into this last point much now, but "mixed strength" tests are another even more powerful test design approach for you to be aware of as well. With mixed strength testing solutions, the test designer is able to select a default coverage strength for the entire plan (e.g., pairwise / AKA 2-way coverage) and, in the same set of tests, select certain high priority values to receive higher coverage strength (e.g., 4-way coverage strength selected for each "Credit Rating" and "Income" and "Loan Amount" and "Loan to Value Ratio" would give you a palm that achieved pairwise coverage for everything in the plan plus comprehensive coverage for every imaginable combination of values from those four high priority parameters. This approach allows you to focus on risk-based testing considerations.

 

Sorry if I got a bit long-winded. It's a topic I'm passionate about.

Originally posted on Stack Exchange, Additional note added after the first 3 comments were submitted:

@Hannibal, @Peter K., and @MichaelF, Thanks for your comments! If you'd like to read more about this stuff, I recommend the multiple links available through this "bundle of links" about pairwise testing and combinatorial testing. In particular, Michael Bolton's article on pairwise testing is directly relevant and very clearly written. It is one of the few introductory articles around that accurately describes the difference between orthogonal array-based solutions and pairwise solutions. If I remember correctly though, the example Michael uses is a rare exception to the rule; the OA solution has the same number of tests as an optimal pairwise solution does.

Related: The Empirical Evidence for Using Pairwise and Combinatorial Software Testing - 3 Strategies to Maximize Effectiveness of Your Tests - Hexawise TV

More than 100 Fortune 500 firms use Hexawise to design their software tests. While large companies pay six figures per year for enterprise licenses, Hexawise is available for free to schools, open source projects, other non-profits, and teams of up to 5 users from any kind of company. Sign up for your Hexawise account.

By: John Hunter and Justin Hunter on Jun 11, 2013

Categories: Combinatorial Testing, Design of Experiments, Efficiency, Multi-variate Testing, Pairwise Software Testing, Software Testing, Testing Strategies, Experimenting

The Expected Result feature in Hexawise provides 3 benefits in the software testing process. Specifically, it...

  • Saves you time documenting test scripts.

  • Makes it less likely for you to make a mistake when documenting Expected Results for your tests.

  • Makes maintaining sets of tests easy from release to release and/or in the face of requirements changes.

There are two different places you can insert auto-generated Expected Results into your tests: the "Auto-script" screen, and the "Requirements" screen. So which place should you use?

It may depend upon how many values are required to trigger the Expected Result you're documenting.

If you want to create an Expected Result that can be triggered by 0, 1, or 2 values, you can safely add your Expected Result using the Auto-script screen. If you're creating an Expected Result that can only be triggered when 3 or more specific values appear in the same test case, then - at least if you're creating a set of 2-way tests - you will probably want to add that particular Expected Result to the Requirements screen, not on the Auto-script screen. Why is that?

It's because if you use the Expected Result feature on the Auto-script screen, it is like telling the test generation engine "if you happen to see a test that matches the setting I provided then show this Expected Result text to the tester." This automates the process so the tester will be provided clear instructions on what to look for in cases that might otherwise be less clear to the tester. If the only way

Using the requirements feature is similar but a bit different. Using the requirements feature, in contrast, is like telling the test generation engine "make sure that the specific combination of values I'm specifying here definitely appear together in the set of tests at least one time (and, if I tell you that there's an expected result associated with that scenario, please be sure to include it)." The requirements feature is used when you not only want to provide an expected value for a specific test case situation but to require that the specific test case scenario be included in the generated test plan.

See our help page for more details: How to save test documentation time by automatically generating Expected Results in test scripts.

 

Related: How to Model and Test CRUD Functionality - 3 Strategies to Maximize Effectiveness of Your Tests

By: John Hunter on Jun 4, 2013

Categories: Hexawise test case generating tool, Hexawise tips, Software Testing

As the CEO of a small but quickly-growing SaaS (Software as a Service) firm that often doubles software tester productivity, I can attest that Fortune 500 firms I'm talking to are way less "anti-SaaS" than they were just 12 and 24 months ago. Business is booming. More than 100 Fortune 500 firms currently have testers using our tool to design their software tests.

It doesn't take Nostradamus to predict that news stories talking about how "SaaS solutions are innovative and beating out 'traditional' software" will become more and more rare. Increasingly, SaaS solutions, with data stored remotely "in the cloud" by hosting providers like Amazon Web Services, are receiving mainstream acceptance.

The situation we're in now reminds me of when I helped launch Asia's first internet-based stock brokerage firm in 1996/97. It was "big news!" that generated coverage from CNN, Time, a front page article on the South China Morning Post business section, etc. Every reporter we talked to focused a lot of their attention on the potentially grave security risks of this new way of trading stocks. Today, trillions of dollars worth of online trade executions later, a Hong Kong brokerage firm offering its customers the ability to trade stocks online wouldn't be worthy of a mention in a neighborhood newspaper. It's just accepted as the way things are done.

We're quickly heading that way with SaaS solutions too.

 

Related: Looking at the Empirical Evidence for Using Pairwise and Combinatorial Software Testing - A Fun Presentation on a Powerful Software Test Design Approach - What Software Testers Can Learn from the Game of 20 Questions

By: Justin Hunter on May 28, 2013

Categories: User Experience