Hexawise is hiring a senior consultant to help our clients improve their software testing processes and results.

Job description

Your mission will be to help Hexawise’s clients achieve dramatic improvements to their software testing efficiency and effectiveness. To do so, you will be providing consulting, training, and implementation support services to ensure that our customers are successfully achieving their business objectives using our test optimization and test automation SaaS solutions and are progressively expanding their usage of our tools.

Your testing expertise will make you uniquely qualified to share best practices and recommendations with existing and target customers. Your customer expertise will make you uniquely qualified to advocate on behalf of Hexawise customers and influence internal strategy and provide leadership to the overall activities of Hexawise’s professional services.

Your job will encompass a diverse set of responsibilities. You will be a highly valued member of the Hexawise team, reporting directly to the CEO.

Responsibilities: Existing Clients

  • Develop strong operational relationships with clients’ project teams and stakeholders to maximize customer satisfaction and seek additional service opportunities.
  • Provide training and implementation support during initial product implementation followed by project-specific consulting, and ongoing adoption support.
  • Contribute to increase revenue throughout the post-sales lifecycle: increase product utilization; identify and close new consulting business within existing accounts; and minimize churn.
  • Offer guidance to clients during launches of new products, features, and/or service offerings.
  • Lead project-specific consulting engagements, and provide test optimization and test automation guidance to Hexawise implementation initiatives.
  • Return important customer insights to the Hexawise team, with the goal of influencing internal strategy and securing the success of our customers.

Responsibilities: Target Clients

  • Clearly explain the benefits and limitations of combinatorial test design to potential customers using language and concepts relevant to their context by drawing upon your own “been there, done that” experiences of having successfully introduced combinatorial test design methods in similar situations.
  • Develop tailored rollout strategies which include integration of Hexawise Optimize and Hexawise Automate into client processes.
  • Define and present comprehensive training and consulting proposals that will enhance Hexawise adoption and keep customer churn extremely low.


Matt Dengler in Japan
Pictured: Matt Dengler, a Hexawise consultant who recently traveled to Japan to help an insurance client design more thorough sets of software tests.

Requirements:

  • 3-5 years of experience with software testing, preferably with an IT consulting firm or a large financial services organization
  • Ability to master the functional capabilities, methodology, and use cases of Hexawise solutions in order to advise customers and promote best practices
  • Excellent communication and interpersonal skills, with the ability to persuasively communicate recommendations, thoughtfully answer tough questions, effectively champion customer needs, and overcome organizational inertia
  • Industry acumen, with knowledge of current software testing trends and an ability to converse with customers at a detailed level on pertinent issues and challenges and describe to clients where Hexawise fits into the competitive landscape of software testing solutions
  • Ability to travel regularly (likely to be no more than 40%)
  • You must be eligible to legally work in the USA.
  • Working from our offices in Durham, NC would be highly preferable. We might consider remote working arrangements for an exceptional candidate based in the USA.

Learn more about the position and apply.

By: John Hunter on Aug 23, 2017

Categories: Career, Customer Success, Hexawise, Software Testing

Mind maps are an effective way to gather information quickly and organization those ideas. For software testing they provide a great tool to share test plans with product owners and testers in an easy to comprehend manner. The visual clarity of mind maps display content in a usable manner.


Hexawise allows you to import and export mind maps. So you can brainstrom ideas together (users, business analysts, product owners, testers, managers...) and agree on the imporant items to test. And then you can import the mind map into Hexawise and it will generate an optimized test plan with efficient combinatorial coverage (enhanced pairwise testing to test the performance of the software interactions between parameters and parameter values).

You can even use mind maps to edit and maintain your test plans: see Hexawise training explanation of how to use the editable mind maps to edit your plan.

image showing mind map edit screen for airplane ticket reservation in Hexawise


Image of the edit screen for the mind map for an airplane ticket reservation system (the first Hexawise sample plan - you can view the plan in your Hexawise account and experiment with the mind map feature).


Related: Mind Maps: What, Why and How - Create a Risk-based Testing Plan With Extra Coverage on Higher Priority Areas - Automatically Generating Expected Results for Tests Using Hexawise

By: John Hunter on Aug 1, 2017

Categories: Combinatorial Software Testing, Hexawise, Hexawise tips, Software Testing, Testing Strategies

This interview with Angie Jones is part of the Hexawise “Testing Smarter with…” software testing interview series. Our goal with these interviews is to highlight insights and experiences as told by many of the software testing field’s leading thinkers.

photo of Angie Jones
Angie Jones

Angie Jones is a Senior Software Engineer in Test at Twitter who has developed automation strategies and frameworks for countless software products. Angie shares her wealth of knowledge by speaking and teaching at software conferences all over the world and leading tech workshops for young girls through Black Girls Code.

Personal Background

Hexawise: Wow! You’re everywhere. Conference presentations around the globe at a break-neck pace. How’d that happen, exactly?

Angie: Ha ha. I’ve been doing test automation for quite some time. In early 2016, I attended a testing conference and didn’t learn much of anything new. It wasn’t a knock on the conference, but it was a wake up call for me.

I also interviewed people for automation roles quite often and found it was extremely hard to find people that were on the same level as my team. I realized that I have something to contribute and should be working to advance the industry in any small way that I can.

Diversity is also something that’s extremely important to me, so I thought being a black female on the stages of white male dominated conferences could also be my way of shaking up the game a little bit.

Actually, seeing a black woman announced as a speaker at a tech conference on Twitter was something that stopped me in my tracks. It wasn’t something I’d seen before. The conference was Write/Speak/Code, an event that empowers women to become thought leaders by writing blogs, speaking at conferences, and contributing to open source software! It was exactly what I needed. I attended and it absolutely changed my life. In less than 6 months after that conference, I’d become an international speaker and was being invited to keynote across the world. It all happened so fast...it was crazy!

Hexawise: You have helped Black Girls Code and TechGirlz work to provide girls the opportunity to learn to code. Do those efforts leave you with hope that the future is in good hands?

Angie: You know, I set out to inspire young girls, but every time I work with them I’m the one who leaves totally inspired and renewed. The girls are so smart and innovative! I give them a little push and they end up creating things that totally blow me away. The idea is to plant the seed. I wasn’t exposed to technology much as a young girl and had no idea that computer programming was even a career option. I almost missed my calling because of that. I don’t want our industry to miss out the next Tech Rock Star because she didn’t know about us.

The greatest businesses are ones that observe how their customers are misusing their products/services and adapt accordingly to make it easier for them to do what it is that they want to do, and still gain the benefit of the product/service.

Hexawise: Recently you moved from the Raleigh-Durham area (home to Hexawise, among other technology companies) to San Francisco and took a new role at Twitter. How did that come about? What is your new job?

Angie: Yes, I’m so excited about this new opportunity at Twitter! I told myself that my next role was going to be something fun and cutting edge. Twitter is just that! I love the platform, I love the innovative culture, and I love the possibilities for growth. I’m working on critical automation tasks and helping to drive automation efforts related to revenue...so ads and live video.

Hexawise: Have you gained a new insight into some aspect of software testing from your work at Twitter that you can share? You haven't been at Twitter long but sometimes in a new environment people are especially alert and gain insights that others may not notice.

Angie: When I have the privilege of choosing the work I'll be doing, I lean towards companies who understand the need for both a Tester and an Automation Engineer. Twitter gets that, which played a big role in my decision to accept this position. I, along with a few other industry leaders, have been preaching for a while now that we don't necessarily need all testers writing automation, but with the complexity of today's applications, testers do need to be technical in order to do a thorough job.

At Twitter, I've now seen just how true that is. I'm working with top notch testers who aren't programmers but they understand the intricate plumbing of our systems and are capable of digging into the guts of the application to ensure all pieces are working as they should. This requires quite a bit of technical acumen yet very little coding. I'll further explore this in my keynotes this Fall at Targeting Quality (Canada), and Agile Testing Days (Germany).

Views on Software Testing

Hexawise: Your article BDD Without the Three Amigos: Maybe Talking To Yourself Isn't So Bad is really thought provoking. In it, you acknowledge that Behavior-Driven Development (BDD) “done right” includes the 3 amigos, but you go on to explore what happens in situations when BA’s and Developers aren’t wiling to “play ball”. You suggest that you have seen testers gain significant value on their own by using BDD. Testers, for example, can create Gherkin feature files with clear “Given / When / Then” instructions that enable rapid creation of executable test scripts.

Angie: Yeah, we often make these hard fast rules and try to force everyone to follow them. Whenever someone “misuses” a practice, all hell breaks loose on the internet. I’ve consulted with a lot of teams and came to realize that these teams are not naive. They realize that they are using the approach differently than it was intended to be used. However, they’ve made it work for them. The greatest businesses are ones that observe how their customers are misusing their products/services and adapt accordingly to make it easier for them to do what it is that they want to do, and still gain the benefit of the product/service. To that point, Matt Wynne, co-founder of Cucumber BDD, actually read this piece and realized that there was more he could do in the industry to push for collaboration from the other amigos.

In a fast-paced software delivery model, automation is definitely needed, but many companies make the mistake of thinking it’s a replacement for testing... Automation should be used as a tool... a technique to enhance testing efforts.

Hexawise: What testing practice(s) do you most wish the software testing community would embrace?

Angie: That’s an easy one...Automation. But there’s definitely some education needed around this, which is why I write and speak on topics in this space. In a fast-paced software delivery model, automation is definitely needed, but many companies make the mistake of thinking it’s a replacement for testing. Anytime I’ve seen companies take this approach, the quality of their product has greatly suffered. Automation should be used as a tool... a technique to enhance testing efforts.

Hexawise: In your presentation at the 2016 SeleniumConf UK conference, you explore "How to Get Automation Included in Your Definition of Done." In that talk you discuss the idea that automation is useful but also that not all tests should be automated. How should a test team go about determining what software tests should be automated?

Angie: Ha ha. That’s a talk in and of itself. Actually, I’m going to give that talk at STPCon in the Fall of 2017. I get asked this question all the time, and it’s such a tricky thing to nail down. That’s because it’s highly contextual to the needs of the business. To do this topic any justice, I plan to explore several case studies in the STPCon talk and demonstrate how context plays such an essential part in answering this question correctly.

Industry Observations / Industry Trends

Hexawise: What trends do you foresee impacting the software testing community in the next 5 to 10 years?

Angie: With the explosion of the Internet of Things (IoT) and artificial intelligence, the systems that we test are going to become a lot more complex. This will require an evolution of testing. Our roles will become even more technical, and yet also require a healthy balance of humanity. The scenarios realized by these smart systems will require thorough testing and solid judgment. Software testing will look a lot different in 10 years, but will be extremely exciting!

Hexawise: Do you see organizations integrating software testers into the software development process (as compared to those instances where the first time software testers are involved is when software is delivered to be tested)? Do you believe more integration of software testers throughout the software development and maintenance process would be useful as software testing evolves?

Angie: I’ve already seen a huge improvement in integrating software testers into the development process with the embracing of agile practices. Quality is no longer solely owned by testers. Developers are testing their features before check-in, and testers are present throughout the entire process, essentially offering insight as early as the requirements phase. This has been essential as teams are adopting continuous integration and deployment processes. Time is of the essence, and the earlier we can avoid/find/correct mistakes, the better!

Hexawise: For those who are not used to involving testers early in the software development process, how would you describe the benefits to the business of involving software testers early in the process?

Angie: By involving testers early in the software development process, the team is able to identify and correct assumptions. This essentially eliminates potential bugs before production even begins. Testers bring in a breadth of knowledge about the application as a whole and how the individual components work together. Testers also serve as a customer advocate, ensuring that their goals are being considered. So, while the team is discussing a potential feature, the tester is advocating for the customer and also calling out how this can affect existing features in the system.

I sat in a design meeting recently and watched the tester poke holes in the proposed design and call out omissions that would cost us millions of dollars. The developers left that meeting with a list of requirements and considerations that they had missed. That's beyond valuable.

Staying Current / Learning

Hexawise: How do you stay current on improvements in software testing practices?

Angie: I read a ton. I’m subscribed to several blogs, and I use Twitter to find new ones all the time. People are always creating new tools or approaches to solve interesting problems and I try to absorb as much of it as I can.

I don’t limit myself to testing blogs. I also read development blogs and tech news sites to stay up to date with trends in the software industry as a whole. This helps me think past my current role and prepare myself for the future as well.

I’ve already seen a huge improvement in integrating software testers into the development process with the embracing of agile practices. Quality is no longer owned by testers. Developers are testing their features before check-in, and testers are present throughout the entire process, essentially offering insight as early as the requirements phases.

Hexawise: You have presented at and attended many technology conferences. What advice do you have for people attending software conferences so that they can get more out of the experience?

Angie: It’s so funny how people tend to flock towards topics they are familiar with. They end up sitting there nodding in agreement with everything the speaker says, sharing their own experiences during Q&A, and leaving thinking that was such a great talk. However, did they learn anything new?

I try to attend talks that will address an unsolved problem that I have, or is in an area that I know very little about. These are the talks where I gain the most insight and can bring something of value back to my job.

Also, be sure to network! You won’t remember everything that people said during their talks, but leaving with a catalog of names and contact information of not just the speakers, but attendees as well, is gold! I often hit problems and I recall meeting someone a year ago at a conference who talked about this very problem during the cocktail hour. Because I networked with them, I feel comfortable reaching out and asking for help.

Hexawise: What blogs would you recommend should be included in a software tester's RSS feed reader?

Angie: Fortunately, Ministry of Testing has an aggregated feed that I use all the time. It consists of more than 500 testing blogs! It’s my morning newspaper.

Profile

Angie Jones is a Senior Software Engineer in Test at Twitter who has developed automation strategies and frameworks for countless software products. As a Master Inventor, she is known for her innovative and out-of-the-box thinking style which has resulted in more than 20 patented inventions in the US and China.

Angie shares her wealth of knowledge by speaking and teaching at software conferences all over the world and leading tech workshops for young girls through Black Girls Code.

Links

Blog: Angie Jones

Twitter: @techgirl1908

Read previous Testing Smarter with... interviews: Testing Smarter with Alan Page - Testing Smarter with Dorothy Graham - Testing Smarter with Mike Bland

Subscribe to the RSS feed for the Hexawise software testing blog.

By: John Hunter and Justin Hunter on Jun 14, 2017

Categories: Interview, Interesting People , Software Testing, Testing Smarter with...

Kathleen Poulsen of Fidelity Investments gave a presentation at STAREAST 2017 sharing her experience using Hexawise to improve their software testing performance. Watch a 10 minute video with highlights of that talk:

We didn't really have what I'd call a scientific methodology to approaching the tests...

Our regression test suites were continuously expanding... We found there was a repition of tests.

We had 3 different projects that I will talk about that I feel like combinatorial or pairwise testing was the key to answering all of those problems.

Hexawise allows you to harness the power of combinatorial software testing with test plans designed to provide thourogh testing of interaction impacts on the software being testing. Hexawise provides more coverage with fewer tests.

All the teams that are using Hexawise can use that same file, they can talk to each other. [Another] thing I liked about Hexawise was the coverage chart... I go back to my business partners and say I am not running these tests. If they are important to you I add them back in with the click of a button. I love that... it was a game changer for me.

Using the Hexawise exporting options

the tests that we produced were converted into the given then when type scenarios automatically and when they are exported into excel you can use them to drive the Sellenium test automation framework. No additional work from us involved.

Using Hexawise's ability to create highly optimized test plans Fidelity was able to greatly reduce the number of tests while also greatly improving coverage.

We were able to reduce from 12,000 tests down to 600.

This type of result sounds amazing, and it is. But it is also what we find consisently from clients over and over. There are certain things people just cannot do well and designing test plans to cover incredible large numbers of interactions between test values and conditions is one of those things. Using highly optimized alogorithms to create test plans to cover these interactions in order to reliably create software customers will love is key. This also frees people to do what they do best.

Kathleen also discussed the significant improvement in communication within Fidelity that was brought about by using Hexawise.

The common language has become the test plan that comes out of Hexawise today.

Improving communiction is an area many organization see as important but finding concrete ways to achieve better communication is often difficult. We have designed Hexwise to aid the communication between stakeholders, including: software developers, software testers, product owners, help desk support staff and senior management.

The simplicity of this tool along with the way you can enter your parameters using the mind map tool, getting that coverage chart automatically out of it, having it export your data into a pretty commonly usable format - those are things that were teribly important to me. They gave me real value... I love that.

I can accomodate many differnt types of testing. We are testing at the class method level, at the services interface level, at the UI level...

Related: 84% of Software Defects Found in Production Could Have Been Found Using Pairwise Testing - Create a Risk-based Testing Plan With Extra Coverage on Higher Priority Areas - 2 Minute Introduction to Hexawise Software Testing Solution

By: John Hunter on Jun 9, 2017

Categories: Combinatorial Software Testing, Combinatorial Testing, Efficiency, Hexawise, Hexawise test case generating tool, Multi-variate Testing, Pairwise Software Testing, Pairwise Testing, Recommended Tool, Software Testing, Software Testing Presentations, Software Testing Efficiency, Testing Case Studies, User Experience

This interview with Matt Heusser is part of the Hexawise “Testing Smarter with…” software testing interview series. Our goal with these interviews is to highlight insights and experiences as told by many of the software testing field’s leading thinkers.

Matt Heusser
Matt Heusser

Matt Heusser is a software craftsman with a deep background in software delivery and testing.

In 2014, Matt received the Most Influential Agile-Test Professional Award at Agile Testing Days in Potsdam, Germany.

Personal Background

Hexawise: If you could write a letter and send it back in time to yourself when you were first getting into software testing, what advice would you include in it?

Matt: My advice would be to trust your instincts and experiences more than what you read in books or online. Often the advice I read in books and online seemed vapid (shallow), or simplistic, or I felt it "just wouldn't work here." Eventually I realized that a lot of it (this was the testing advice of the 1990's) wasn't working well most places.

Today we have better advice. The time from research to publish is short, and there is a lot less "loss" in the system. Still, what works for Google and Microsoft might not work for your 20 person company, and what works for that cool, 100% physically distributed, 40 person software company might not work for your 2,000 employee insurance company. Take it with a grain of salt, trust your instincts - but always keep exploring and experimenting.

Hexawise: Which person or people have had the greatest influence on your understanding and practice of software testing?

Matt: It's hard to come up with a list of influences, but Cem Kaner, James Bach, Ken Pier, Brian Marick, Johanna Rothman, Lee Copeland, Jerry Weinberg, Kent Beck, and Ron Jeffries all come to mind.

Hexawise: What one or two software testing-related experiences have you found to be most personally satisfying in your career?

Matt: I remember once we wanted to get a insurance data extract out on a friday, but it had to be right. Serious students of testing will tell me you can never know that it is right - but non-serious student customers don't know that. I had about a half a day. The change was to one field, and we had the results in a database table. The table-to-file and the file transfer we had confidence in; the change, not so much. So I wrote my own computer program to loop through every current member in the insurance plan, over four hundred thousand, calculate the expected result, and compare them.

We found a small bug in the requirements; an edge case that was ambiguous. The programmer and I had interpreted the requirements in different ways that were both arguably correct. The "differ" program I wrote found the case, the customer explained that either was acceptance - and we shipped!

Views on Software Testing

Hexawise: What do you wish more developers, business analysts, and project managers understood about software testing?

Matt: A decade ago, when I went to the Google Test Automation Conference, one of the speakers said that automation was better because it was "repeatable." I almost stood up and asked aloud "so what?"

Every build of the software different. If the software is different, if the risk picture has changed, if we have some idea of what tests we ran before and how valid they might be on this build - why would we ever test the exact same way?

Software isn't an assembly line. Every build is different. The way we test it can be different, but I certainly don't see a ton of value in spending extra money to make sure we test things the exact same every time.

Turns out you don't need to remove the friction from handoffs. Often, you're better served to make communication cheaper and getting good at collaboration.

Hexawise: Can you describe a view or opinion about software testing that you have changed your mind about in the last few years? What caused you to change your mind?

Matt: It's going back a bit, but in graduate school, I wrote a paper "On the optimization of physically distributed requirements, development, test, operations, and management development groups" The body of the paper was considerably shorter than the title - the body was "You're Screwed."

At the time, the literature suggested that communication costs were high, so we needed get everything right prior to "the handoff." We needed to get every document, every bit of code, everything complete, consistent, correct, unambiguous, before "the handoff."

I knew that trick never worked, and thought the fix was co-location.

Then I learned about agile/XP, which was still focused on co-location - still, part of XP was making communication, collaboration, and change cheaper. Then I worked at Socialtext, where my co-workers were all over the world - Ingy took a skiing vacation in France, skiing during the day and working core hours at night. And it was far more productive than any other job I had ever had.

Turns out you don't need to remove the friction from handoffs. Often, you're better served to make communication cheaper and getting good at collaboration.

Hexawise: You have written about the benefits of lean thinking in software testing. What advantages do organizations gain when they adopt a lean thinking view of software testing?

Matt: You know that thing that happens, where you can't do your job because you filed a ticket and it will take the DBA's a week to add a column to a table, so you can't do your job, for a week?

Or whatever else it is? Right now I've got a contractor billing on my team with no laptop. He'll have it nine days after he started ... if we're lucky.

Typically, when a company goes to lean thinking, that kind of stuff stops happening.

Industry Observations / Industry Trends

Hexawise: Do you have specific suggestions for testers working within an organization using agile or lean software development methods?

Matt: It's hard to come up with examples without context, but generally, I'd start by looking at the delays we have in our job and the amount of multitasking. Be sure to include failure demand - things that should be reasonably expected to work the first time, but took a round of fixes. Often you'll find what should take an hour is taking you a week.

Hexawise: What do you see as the most powerful trends in the software testing field over the last 5 to 10 years? What trends do you believe will be the most powerful over the next 5 to 10 years?

Matt: The past ten years? Test-Driven Development, Continuous Integration, REST APIs that can be tested at the integration level, Virtualization, Lightweight Virtualization.

The next five? The promise of continuous delivery might just be realized. I realize that sounds like a lot of technical stuff, and I'm a process and people guy. The challenges of the next few years will be all skill, people and process.

Hexawise: There is still a widespread belief in fairly mechanistic software testing (checking) by some of those using software testing. Lean thinking, exploratory testing, etc. encourage engaging the minds of software testers. Are you optimistic about the prospects for tapping more of the potential software testers have going forward?

Matt: Oh yes. That's what I hope the next five or ten years are all about.

You know that thing that happens, where you can't do your job because you filed a ticket and it will take the DBA's a week to add a column to a table, so you can't do your job, for a week? ... Typically, when a company goes to lean thinking, that kind of stuff stops happening.

Hexawise: Have you seen a particularly effective process where the software testing team was integrated into the feedback from a deployed software application (getting feedback from users on problems, exploring issues the software noted as possible bugs...)? What was so effective about that instance?

Matt: My preference is for cross-functional delivery teams, so I get a little down on the term "test team", but yes, I have seen delivery teams where customer feedback was part of the process. One team that Justin Rhorman and I worked with managed a fortune 500 retail website; they had a process that periodically popped-up requests for feedback. The testers worked through these in a rotation, summarizing them, reporting them to management and working with management.

I think that worked well because of the rotation - no single bottleneck. If multiple testers observed the same feedback independently week over week, it was harder to dismiss.

Staying Current / Learning

Hexawise: You have spoken at many conferences. What advice do you have for people attending software conferences so that they can get more out of the experience?

Matt: Try to come up with three things to do on Monday that justify the investment. These things should be entirely within your power to do. They should not require training, a team-level change in process, particular learning time, purchase of tools or hiring of consultants. Things you can just do - that you will not ask permission for. (Your boss probably doesn't know what you do anyway.)

Then do it and tell the team about it. If you want to write a trip report, it should be a one-pager, and describe what you are doing, not what you heard.

Hexawise: What software testing-related books would you recommend should be on a tester’s bookshelf? What blogs would you recommend should be included in a software tester's RSS feed reader?

Matt: I don't read blogs like I used to, instead I follow twitter and click on interesting links. A few weeks ago I wrote an article on 28 testers to follow on twitter that might be a good place to start.

As for books, I'd say How To Reduce the Cost of Software Testing, Lee Copeland's A Practitioner's Guide to Software Test Design, Lessons Learned in Software Testing and Jez Humble's Continuous Delivery book (the writing is a bit tough but it's worth it). I'd also suggest some non-testing books, like Out Of The Crisis by Deming, Peter Drucker on Management, and Taleb's Black Swan book.

Hexawise: We share an interest in seeing lean thinking concepts be adopted by software testers. How would you suggest someone interested in learning more about lean thinking in software testing do so?

Matt: You could try a google search for "heusser lean" and see what it returns, seriously I've written a lot. Here are two older articles that I think cover the start of it Applying lean concepts to software testing and The secrets of successful Lean software testing.

Profile

Matt Heusser is a software craftsman with a deep background in software delivery and testing. After earning his undergraduate degree in Math with a concentration in computer science in 1997, Matt began his career as a programmer, writing code in C, perl, PL/SQL, and Visual C++. Along the way, Matt was the initial organizer of Grand Rapids Perl User's Group ("Perlmongers"), earned a master's degree in CIS from Grand Valley State University, taught IS part time at night at Calvin College, and served as the initial lead organizer of the Great Lakes Software Excellence Conference.

After leaving Priority Health, a Health Insurance Company, in 2008, Matt went on to become a member of the technical staff at Socialtext, the world's first wiki company, where he worked with Audrey Tang, Ingy DotNet, and Dan Bricklin, the creator of the Spreadsheet, to help build a web-based spreadsheet/wiki that predated google docs. The test framework Matt worked on at Socialtext, WikiQTests, is documented as a case study (chapter 16) of O'Reilly's book "Beautiful Testing."

Matt left Socialtext in 2011 to become a full-time consultant. Since that time he reviewed Robert C. Martin's books "Clean Code", "The Clean Coder" (for which he wrote the preface) and "Clean Architecture." In 2014, Matt received the Most Influential Agile-Test Professional Award at Agile Testing Days in Potsdam, Germany.

Links

Blog: Creative Chaos

Twitter: @mheusser

Previous Testing Smarter with... interviews: Testing Smarter with Rikard Edgren - Testing Smarter with James Bach - Testing Smarter with Michael Bolton

Subscribe to the RSS feed for our blog.

By: John Hunter on May 23, 2017

Categories: Testing Smarter with..., Software Testing, Lean, Agile

This interview with Ajay Balamurugadas is part of the Hexawise “Testing Smarter with…” software testing interview series. Our goal with these interviews is to highlight insights and experiences as told by many of the software testing field’s leading thinkers.

Ajay considers being inducted into the 'Bach Brothers' Testing Legion of Merit, presenting his keynote at CAST 2015 and attending Problem Solving Leadership workshop by Jerry Weinberg and Esther Derby to be his biggest achievements to this date.

His contributions to the software testing community include co-founding Weekend Testing and Test Maniac. His short books are popular with many testers for the practical, ready to use tips.


Ajay Balamurugadas

Personal Background

Hexawise: I like the Weekend Testing concept that you helped bring into existence. What gave you the idea to do so? What were you trying to accomplish? How is it working?

Ajay: Thank you. I wanted to practice software testing and had the first online paired testing session with Parimala Hariprasad (@Curioustester). It was fun learning about a new tool. Next weekend, Sharath Byregowda (@Sharathb), Manoj Nair and myself tested for two hours. That night, we decided that this could be even more fun and enriching if more folks joined us. We then opened the forum to the public on August 15th (Indian Independence Day), 2009. Regarding the aspirations, I never knew that this would grow as big as it has grown today.

With more than 300 sessions spread over India, Americas, Europe, Australia/New Zealand, the forum seems to have connected many testers and helped them hone their skills across different approaches, techniques, tools and products. Thanks to so many volunteers and facilitators who have jumped in at various stages of this journey and helped Weekend Testing evolve over the years.

We are still having the sessions during the weekends, the frequency has reduced to one or two per month. To be honest, testers need not wait for a session to participate in a weekend testing session. They can pickup any session report, timebox their testing activity for an hour and then compare their report with other testers' reports.

Hexawise: If you could write a letter and send it back in time to yourself when you were first getting into software testing, what advice would you include in it?

Ajay:

  • Focus on the basics
  • Spend money on learning
  • Take care of health

Hexawise: What kinds of activities do you enjoy when you’re not at work?

Ajay: I love playing and watching cricket other than reading books. I and my wife (@AbiTheTester) enjoy going to the mall, watch a movie, eat ice cream and play a video game of bike racing.

I have come to the realization that agile teams are one of the ideal places for a skilled software tester. With so little time, emphasis on finding critical information early and in a crisp manner is necessary. Who, other than a skilled tester can switch contexts, interact with multiple stakeholders, think critically and add value to the teams?

Hexawise: What one or two software testing-related experiences have you found to be most personally satisfying in your career?

Ajay: I am thankful to many software testing mentors who have helped me grow personally and professionally. Knowing so many testers at a professional and personal level is in itself a satisfying moment for me.

Attending Problem Solving Leadership with Jerry Weinberg, Esther Derby and so many testers from different countries, presenting a keynote at CAST 2015, receiving the "Bach Brothers Testing Legion of Merit" award, helping Fiberlink (now acquired by IBM) adopt mind maps are some of the moments that make me think that I must have done something right in my testing career.

I certainly celebrate every small moment that has taught me - for example I cherish the moment when parents of a tester called me and thanked me for helping their son get a job in software testing.

Views on Software Testing

Hexawise: Can you describe a view or opinion about software testing that you have changed your mind about in the last few years? What caused you to change your mind?

Ajay: My immaturity at the start of my career made me repel automation. I always thought that it was the skill of the testers that is more important than the tools. Later, I realized that there are many activities that would be done better if they were automated. So, I started to shift my focus on learning to automate and here I am evangelising Sahi Pro - The Tester's Web Automation Tool.

Hexawise: What testing practice(s) do you most wish the software testing community would embrace?

Ajay: These are a few practices that I would like the whole software testing community to think about, not necessarily embrace:

  • Spend more time interacting with the application under test than with documents, processes that have little value for the customer.
  • Learning to test well is a long journey. Do not accept shortcuts like 2 days workshops as __the__ solution to your learning. They accelerate your learning, highlight the probable mistakes you can do and give you a learning path.
  • Trying to get everyone on the team to learn everything is a sure recipe for killing motivation. Get people with complementary skills as part of one team and see how they create magic.

To be convincing, you might have to work on your reputation first. Work on it. People most of the times say "No" until they are convinced. Once you highlight the benefits of pairing and collaboration, people will listen.

Industry Observations / Industry Trends

Hexawise: Do you have specific suggestions for testers working within an organization using agile software development methods?

Ajay: Gel with the team members and at the same time, do not forget your core skill: Testing - Providing information about the quality of the product and project to stakeholders who matter.

I have come to the realization that agile teams are an ideal place for a skilled software tester. With so little time, emphasis on finding critical information early and in a crisp manner is necessary. Who, other than a skilled tester can switch contexts, interact with multiple stakeholders, think critically and add value to the teams?

Hexawise: Do you have suggestions for how testers can be more effective if they are isolated from the software developers (either by practice in the organization or by geography)?

Ajay: If it is by practice, talk to the managers and see why it is the case. To be convincing, you might have to work on your reputation first. Work on it. People most of the times say "No" until they are convinced. Once you highlight the benefits of pairing and collaboration, people will listen. If they still don't listen, maybe its time to change teams or company.

If the distance is because of the location, it is easy to solve. How would you collaborate if your best friend was from a different city? You will find ways to get in touch. Today, there are multiple tools that help you have that seamless experience. Make use of those tools. At the end of the day, everyone is a human being. The moment we consider everyone as a human being and not as someone who has a developer/manager/tester role, most of the problems would just disappear.

Check out my book - 50+ tips to improve tester-programmer relationship which will help your learn how you can improve the relationship.


Communication tips graphic from Ajay's book

Hexawise: "Whenever you need to test the best combinations out of all possible combinations, I recommend Hexawise. It can help you create data quickly and in a format that you can directly use. Very cool tool. Use it to see the power of Hexawise." How did you learn about Hexawise. How does it help you?

Ajay: That was quite long back and I think I might not change the statement even now, though maybe I would edit it to say:

"Whenever you need to provide effective test coverage for multiple parameter combinations, I recommend Hexawise. It can help you create test plans quickly and in a format that you can directly use. Very cool tool. Use it to see the power of Hexawise."

Many testers use tools without understanding why they have to use the tool. Someone who understands the technique of combinatorial testing, will appreciate the ease of use Hexawise provides.

I learned about Hexawise through the Twitter world of #testing . There have been multiple instances in my testing career where I have helped the teams reduce the number of test cases with the help of Hexawise.

Staying Current / Learning

Hexawise: You have blogged about your career journey in software testing several times. What tips do you have for those looking to deepen their knowledge of software testing and move forward their careers?

Ajay: Many testers don't seem to have a learning path at all and that is sad. I really appreciate those who take time to get better at their craft. Today, we don't have shortage of sources of knowledge. People just have to spend time and dedicate themselves to learning.

Realize that there is a lot to learn. Pick what you want to learn, carefully. You will face challenges and you need to be motivated throughout the journey to excel in learning the subject. Set time limits, take help of mentors, take it slow if needed.

Each person has their own style of learning - some like it hands-on, some read books, some watch videos. Know your style and keep measuring your learning quotient - are you happy learning the subject. As long as you are happy learning it, continue or else modify the plan to suit your needs.

I am focusing on 1-2 quality criteria per year and have started with Security and Automation for the last few months. I am loving the journey and wish others too good luck in their journey. At this moment, I am reminded of the different pathways Katrina Clokie has blogged about.


Mind map of his learning by Ajay


Hexawise: What advice do you have for people attending software conferences so that they can get more out of the experience?

Ajay: Make notes, connect with the speakers even after the conference, do not sit with your team members in the conference, make new friends and spend time talking to people rather than spending your whole time inside at the talks. I created a mind map for anyone attending EuroSTAR 2011. It looks like the points apply even today.

Hexawise: What software testing-related books would you recommend should be on a tester’s bookshelf? What blogs would you recommend should be included in a software tester's RSS feed reader?

Ajay: There are many books that can be part of a tester's bookshelf (Huib Schoots' list). Check out those books which will help you achieve your immediate goals. My books can also be found here.

For blogs, check out the Ministry of Testing's testing feed everyday. You can also follow @JorisMeerts on Twitter.

Profile

Ajay considers being inducted into the 'Bach Brothers' Testing Legion of Merit, presenting his keynote at CAST 2015 and attending Problem Solving Leadership workshop by Jerry Weinberg and Esther Derby to be his biggest achievements to this date. He is also happy at the number of testers who have been influenced positively in interactions with him.

He started his career as a software tester and he continues to be a hands-on software tester along with training new testers, presenting at conferences, conducting workshops and sharing his thoughts through his blog and tweets.

Ajay started with testing standalone desktop applications and soon moved on to web applications and mobile applications. His journey was boosted by co-founding Weekend Testing, Test Maniac. His short books are popular with many testers for the practical, ready to use tips.

photo of Abinaya and Balamurugadas Ajay
Abinaya (his wife) and Balamurugadas Ajay

Links

Website: Test With Ajay

Blog: Enjoy Testing

Twitter: @ajay184f

By: John Hunter on May 1, 2017

Categories: Interview, Testing Smarter with..., Software Testing

This interview with Hans Buwalda is part of the Hexawise “Testing Smarter with…” software testing interview series. Our goal with these interviews is to highlight insights and experiences as told by many of the software testing field’s leading thinkers.


Hans Buwalda

Hans has gained experience as a developer, manager, and principal consultant for companies and organizations worldwide. His approaches to testing—action-based testing and soap opera testing—have helped a variety of customers achieve scalable and maintainable solutions for large and complex testing challenges.

Personal Background

Hexawise: What drew you into a career in software testing?

Hans: It was "accidental". I worked as a management consultant in my home country, the Netherlands and was sent on an assignment to a major customer. They had serious problems with both test design and automation. In particular it was difficult to involve the (very busy) domain experts, and the automation was virtually impossible to keep current. Using keywords turned out to solve both these problems, and I have been pioneering that ever since with the Action Based Testing Method.

Hexawise: You were an early pioneer and key contributor to the keyword-driven test automation framework which has stood the test of time and is widely adopted in the industry. What drove you and other early keyword-driven framework pioneers to create it and advocate for its broader adoption?

Hans: I believe the two main drivers for using keywords are readability for non-technical people, and long-term maintainability of the tests. I think my core message is not as much the keywords themselves, but more the importance of test design in achieving those goals. Without a good modularized organization of tests keywords will fail, and so will for example Behavior-Driven Development (BDD). To say it even stronger: the worst automation projects I have seen were keyword projects.

Hexawise: What one or two software testing-related experiences have you found to be most personally satisfying in your career?

Hans: I like it when people are successful. In my world that means being able to leverage automation to achieve better quality at a faster pace. Two of my colleagues recently showed me how they were able to help a company in the oil industry reduce time needed for testing dramatically (more than twenty times).

A nice detail there was that those tests were for equipment workflows, and did not involve any User Interface (UI). It is a common misunderstanding that keyword automation is for testing via the UI, but it can work just as well for non-UI testing, like Internet of Things (IoT), services, games, telecommunication, embedded software, etc..

Views on Software Testing

Hexawise: What testing practice(s) do you most wish the software testing community would embrace?

Hans: I believe that with Agile, DevOps and service oriented architectures, we're well on our way. Agile allows for good cooperation between the Quality Assurance (QA), developers, product owners, domain experts and stakeholders. DevOps allows testing to be incorporated into the build and deploy pipelines. And service oriented architectures gives more ways to test than just through the UI.

scalability in automation is not as much a technical challenge as it is a test design challenge. This gives a bit of a paradox: test designers influence the automation success, but are not developers, and do not necessarily have affinity with engineering concepts like structured programming and refactoring, which in turn are key elements of good maintainable automation.

Hexawise: Can you describe a view or opinion about software testing that you have that many or most smart experienced software testers probably disagree with? What led you to this belief?

Hans: "Disagree" is a big word. But I do like to focus on one particular area that many others tend not to address that much: It is my observation over the years that scalability in automation is not as much a technical challenge as it is a test design challenge. This gives a bit of a paradox: test designers influence the automation success, but are not developers, and do not necessarily have affinity with engineering concepts like structured programming and refactoring, which in turn are key elements of good maintainable automation. There are some more elements that can influence automation success, like "testability" of the application under test: to what extend is an application friendly to testing and automation.

Hexawise: You have extensive experience with automated testing. What interesting anecdotes would you be able to share about some of your earliest experiences automating tests?

Hans: An early experience was the very first project where action words (my term for keywords) were used. It was for a major screen trading system, and before I came in a substantial amount of tests had been created with record and playback. When I say that you can probably guess the upcoming/impending disaster... The development team made a slight change in a lay-out of the main screen and virtually all tests stopped working because of it. It was in the early days and people weren't expecting such problems.

Industry Observations / Industry Trends

Hexawise: One of my (Justin’s) favorite software testing articles of all time is “Soap Opera Testing.” What experiences led you to write it and why do you think it struck a chord with so many testers and managers?

Hans: Thanks for the kind note. It was based on a major project in a large financial institution. The project included re-architecting of all systems, and getting ready for the year 2000. Proper testing was essential, but the tests also needed to be developed and fully automated in a very short amount of time.

To get this done I introduced an approach based on real life stories from the every day practice of real users and good test design (test modules, keywords) to support successful automation. I guess the method appeals to teams and managers because it is an efficient way to translate domain knowledge into, aggressive, test cases, and to get them automated quickly as well.

I believe the two main drivers for using keywords are readability for non-technical people, and long-term maintainability of the tests. I think my core message is not as much the keywords themselves, but more the importance of test design in achieving those goals.

Hexawise: Large companies often discount the importance of software testing. What advice do you have for software testers to help their organizations understand the importance of expecting more from the software testing efforts in the organization?

Hans: You have to be serious about your craft. Know testing techniques, engineering principles and the domain of an application you're testing. Your attitude is important too. Be aggressive to the system under test, but cooperative as member of a team. Second, always keep an eye on the business side.

You should not be testing because you have read somewhere that testing is important. Testing costs time and money and there must be business reasons to invest in it. Not testing saves money, but also introduces risks that can cost money later on. Saving money and losing money are business considerations that well managed large companies tend to take very seriously.

You, or your QA management, must be ready to explain the value of their testing, and automation.

Staying Current / Learning

Hexawise: I see that you’ll be presenting at the StarEast conference in May. What could you share with us about those topics? What gave you the idea to talk about them?

Hans: I'm doing two tutorials, one on Better Test Design for Great Test Automation. The other focuses on what makes automated testing scalable. The classes are based on real-world experiences from various projects I’ve done, across many industries. With the many recent developments like cloud and DevOps, automation is constantly evolving, so it is fun to discuss about them.

Hexawise: How do you stay current on improvements in software testing practices; or how would you suggest testers stay current?

Hans: Reading, attending conferences and webinars and hands-on practice still work well. Don’t despair when current developments can be overwhelming. Concentrate on what you need, and be curious.

Profile

Hans Buwalda has been working with information technology since his high school years. In his career, Hans has gained experience as a developer, manager, and principal consultant for companies and organizations worldwide. He was a pioneer of the keyword approach to testing and automation, now widely used throughout the industry. His approaches to testing—action-based testing and soap opera testing—have helped a variety of customers achieve scalable and maintainable solutions for large and complex testing challenges. Hans is a frequent speaker at conferences and other events and is lead author of Integrated Test Design and Automation.

Links

Web site: Happy Tester

Twitter: @hansbuwalda

Previous Testing Smarter with... interviews: Testing Smarter with Mike Bland - Testing Smarter with Dorothy Graham - James Bach

Subscribe to the RSS feed for our blog.

By: John Hunter and Justin Hunter on Apr 17, 2017

Categories: Interview, Software Testing, Testing Smarter with...

Rikard Edgren has been a humanistic and technical tester since 1998. He currently works with testing healthcare software at Nordic Medtest in Karlstad, Sweden. Rikard enjoys the dynamics between people/machines, objective/subjective and whole/details.

photo of Rikard Edgren
Rikard Edgren

Personal Background

Hexawise: What drew you into a career in software testing?

Rikard: Like most people it happened by accident; I wanted to become a programmer, but had a chance to start at a company with testing as a stepping stone.

I realized I liked testing and was good at it, and wasn't upset that I never got the chance to be "promoted" to developer, and just went with it by doing a lot of testing.

I did a couple of years as project manager (good insights for a tester), but went back to what I enjoyed most.

I think I love testing because of the dynamics between the technical and the humanistic; the details and the whole; the objective and subjective.

And of course the thrill of being first to see a nasty problem!

Hexawise: You share authorship of the thoughts from the test eye blog with a few of your former colleagues. This joint author setup is fairly rare; what advantages do you see from this arrangement?

Rikard: Henrik Emilsson and Martin Jansson tricked me into this in 2008. In the beginning it was great to have two readers!

We also commented on each others posts, which made us learn more. We also did some joint efforts, e.g. a list of generic Software Quality Characteristics.

It is a benefit that the blog keeps going also when one or two are very occupied with other things. It also gives a healthy pressure to provide to our common project. The activity is lower nowadays, but still alive.

It has kept the three of us more in touch even though we have worked at different places and locations.

So I see only positive aspects from the joint setup, and encourage writing as a learning process, with sharing as a good side-effect. Ideas you discuss, try, discuss, are typically better than ideas you work out by yourself.

Views on Software Testing

Hexawise: In your book, The Little Black Book On Test Design, you write: "You want to create tests that are testworthy, and typical examples can be those tests where you don’t know if it is important or not, but you want to make sure." How often do you see software developers providing guidance on testing focus due to concerns about the code? My background is in software development and often we know when certain aspects of the code have higher potential for bugs - often due to complexity or novelty. But in my experience this insight (that could focus software testing) is not provided as often as would be useful.

cover image with the text, Little Black Book On Test Design, on a black background

Rikard: In my experience I haven't received important information like this on direct questions. However, when I have worked together with the developers for some time, and (hopefully) have earned respect by digging up interesting information, this sort of guidance come more and more often. I hope it is because they realize that my testing can act on vague and disparate information, and that they value the findings and want more of it. It doesn't seem to be enough to say that you can test the software well, you have to actually do it first in order to gain a healthy respect that everyone benefit from.

Once the trust is there, communication is more open and better, and it is easier also for me to admit areas where the testing wasn't the best.

Respect isn't given, it must be earned, and you do that by finding information others need, that they wouldn't have found by themselves.

Hexawise: Do you have any specific suggestions on how testers can gain respect from developers and encourage them to see a tester as someone to assist them in creating great software. Too often it seems software developers can view software testers as a bother rather than a help?

Rikard: Make sure you provide valuable information. Find bugs the developers want to fix. Give feedback on things that work well. Find bugs that other stakeholders want to fix. Listen to developers input on what needs testing. Collaborate. Be nice and do good work.

Hexawise: In your book you also mention the value of combinatorial testing to discover problems that don't appear in isolation but only with interaction between components of the software and different use cases. Do you have a favorite example of a combinatorial bug? How did that bug illuminate a challenge in software testing?

Rikard: I will never forget a dialog box that crashed when you clicked Cancel, but only if the dialog first had been moved. I don't think I would have come up with the idea of testing just this, it was something I stumbled on because I like to do things differently. It is still a mystery how someone managed to create this bug, and I don't know if it was important to fix it. This illuminates the challenge in software testing on which parameters to explicitly deal with, and which parameters to implicitly deal with by serendipity-enabling variations in our testing.

Hexawise: What is one thing you believe about software testing that many smart testers disagree with?

Rikard: I think many would disagree that it often is a good idea to run tests without no particular reason; that a random test can be better that one carefully picked; that using my subjectivity will help me test the product better.

These work (according to me) because testing is a sampling business, and we should have serendipity working for us.

Also, people have phenomenal capacity when motivated, so I would rather have you perform five tests you believe in, than one that I think is better.

Industry Observations / Industry Trends

Hexawise: Do you believe testing is becoming (or will become) more integrated with the software development process? And how do you see extending the view of the scope of testing to include all the way from understanding customer needs to reviewing actual customer experience to drive the testing efforts at an organization?

Rikard: I think testing have been integrated with the software development process for a long time, and I think good testing often needs to understand customer needs and actual experience. In my 19 years with testing this is how I have worked, unless there are circumstances that make it impossible.

But I have heard similar comments before, and I rather believe that there was some years where there were loud thoughts that testing must be totally objective, both in regards with who we work with, and with the sole objective to verify the documented requirements. Nowadays more people say that they look for more things than what was defined in advance, and that they collaborate with developers as much as possible. Testers have been doing this all the time, and it is a good thing that the idea of "test documentation over insights" is retiring.

Hexawise: What do you wish more developers, business analysts, and project managers understood about software testing?

Rikard: That it is difficult, but done well software testing provides a lot of value to any software where quality is important (but not every project needs testing!)

That they can influence us: tell us about what is important, and we will find out useful stuff about this (good and bad things.)

That automation is great, and exploratory testing too.

Staying Current / Learning

Hexawise: How do you stay current on improvements in software testing practices; or how would you suggest testers stay current?

Rikard: I think each testing situation is unique, so I try to stay current by doing the most effective testing in my current situation. This might include old stuff, recent stuff, brand new stuff, or at least a combination of "stuff" that makes sense for me and the project at the moment. I try to do small experiments with new tools, and disregard most of them, but maybe learned something in the process. By going deeper into the current context, I learn a lot in that area, but maybe not about what is the latest in general.

Many new ideas come right out of the blue when I do something else, so I think my subconscious is helpful to me.

My suggestion to testers would be to use whatever methods they think are fun (and valuable if it is work time), because that will keep up the motivation, and enhance the learning process.

Hexawise: Have you incorporated a new testing idea into your testing practices in the last year? Will you continue using it? Why? / Why not?

Rikard: I don't think I have incorporated any brand new testing idea the last year. But of course there are new combinations of old ideas for my specific purposes.

Recently our team created a "TimeBlaster" in SoapUI; the results from a request is analyzed for time elements, and then many requests are sent with different start-end times (some of them randomized) whereupon the new responses are analyzed for conformance to the specification. Since these rules for time is somewhat complicated, this automation makes us (and other users of the same services) test better and faster, so we will continue using it.

My ongoing, low-frequency, private research currently deals with mental models; how I and other testers think, how we come up with new ideas, how we act on certain perspectives, but not on other. How we handle the complexity of software and human interaction and figure out what we should be testing with limited time available.

But I don't have any answers yet, at least in my head it is a multitude of (invisible) mental models that interact in mysterious, and productive, ways.

Hexawise: What software testing-related books would you recommend should be on a tester’s bookshelf?

Rikard: The best testing book for me is Lessons Learned in Software Testing, by Kaner, Bach, Pettichord. Another favorite is Exploring Requirements by Weinberg/Gause, which (secretly) is about "test analysis" - finding out what is important.

The most brilliant free article is Heuristic Test Strategy Model by James Bach, because it is generic for any software project, and the reader has to do all the hard work by themselves when using the structure it gives for attacking a test project.

My favorite blogger/youtuber is Alan Richardson.

There is lots of good stuff available on the net, and a lot of bad stuff as well, so a healthy dose of skepticism is needed.

And to practice "a healthy dose of skepticism" is something we need to do as testers!

Profile

Rikard Edgren has been a humanistic and technical tester since 1998. He currently works with testing healthcare software at Nordic Medtest in Karlstad, Sweden. Rikard enjoys the dynamics between people/machines, objective/subjective and whole/details.

Rikard has been as a consultant with many education companies and higher vocational studies programs. Teaching is a learning experience, with the number one testing question: What is important?

He is a regular at international conferences, with many appearances at EuroSTAR. He is co-organizer of Swedish Workshop on Exploratory Testing (SWET).

Links:

Blog: thoughts from the test eye

Book: The Little Black Book On Test Design

Presentation: ExploratoryTestDesign

Previous Testing Smarter with... interviews: Testing Smarter with Mike Bland - Testing Smarter with Alan Page - Testing Smarter with Dorothy Graham

Subscribe to the RSS feed for our blog.

By: John Hunter on Apr 10, 2017

Categories: Interview, Software Testing, Testing Smarter with...

This interview with Michael Bolton is part of our series of “Testing Smarter with…” interviews. Our goal with these interviews is to highlight insights and experiences as told by many of the software testing field’s leading thinkers.

Michael Bolton is a consulting software tester and testing teacher who helps people solve testing problems that they didn’t realize they could solve. He is the co-author (with senior author James Bach) of Rapid Software Testing, a methodology and mindset for testing software expertly and credibly in uncertain conditions and under extreme time pressure.


Michael Bolton

Personal Background

Hexawise: Do your friends and relatives understand what you do? How do you explain it to them?

Michael: [laughs] A lot of my friends and relatives don’t really care what I do! But if they’re curious, and they want an explanation, I tell them this:

"All that software you’re using every day is hard to develop. Whenever someone wants software built, development groups try to figure out what to do and how to do it. But since people never completely understand one another, there is a risk that the proposed solution won’t solve the problem, or will introduce problems of its own.

"Then, even when the ideas about how to build something are pretty clear, we’re still building it for the first time. It’s new to us, so we have to learn how to build it. As that happens, people make mistakes, and they realize new problems along the way. It’s the tester’s job to help identify problems all the way through that process.

"And then, as people are trying to build the product, testers explore it and experiment with it, looking for problems that might threaten its value. Ideally, the important problems get found and get fixed before the software comes to you.

"In other words, testers help people to try to figure out whether the product they’ve got is the product they want. I help testers and teams learn how to do that effectively, as a consultant and as a teacher. And I do that worldwide; in classes, at client sites, at conferences, and in social networks."

If friends or family want to know more, I can tell them more. But usually they’re happy with that.

Friends and family are one thing; managers and teams are another. Some testers say their managers don’t understand them. Recognize this: managers need to know about problems that threaten the on-time, successful completion of the project, where “project” means “some burst of development work”. Think about what you’re doing and the problems you’re facing in terms of how they relate to that. Talk about your work that way, and things will become a whole lot clearer.

Hexawise: What one or two software testing-related experiences have you found to be most personally satisfying in your career?

Michael: That’s a tough call. On projects, I really like the detective work. In one specific case I investigated the roots of a nagging customer support problem, and found six or seven wildly different factors that, if addressed, could have eliminated that problem. That was satisfying. On another project, at a bank, I constructed a really elaborate oracle [Hexawise: "If you are not familiar with using oracles in software testing we strongly recommend following the link to learn more about this important concept."] that modeled consumer financial transactions through all of the different account flows. That involved a lot of programming, and using powerful features of Excel, and learning a ton about the business domain. That was fun. I found some cool bugs and weird behaviour, too.

From 2003 through 2008, I did pretty extensive study with Jerry Weinberg —several of the AYE Conferences, some consulting workshops, and the Problem Solving Leadership class. Those were really helpful for my own personal development.

But the most satisfying thing is teaching Rapid Software Testing, helping people develop their mindsets and skill sets, and reframing the ways they think and speak about testing. It’s rewarding to hear from the class participants about their renewed energy and focus—and to hear from their organizations about the value and power of testing, value they might not have seen clearly before.

Views on Software Testing

Hexawise: What testing concept(s) do you wish more software testers understood?

Michael: Oh dear. [laughs] That’s an awfully long list, and it varies from day to day.

I wish more testers were more articulate in describing their models—their models of the product and how to cover it; their models of oracles, and how they recognize problems; their models of testing itself.

I wish that more testers understood that test cases are not central to testing. To be an excellent tester means to explore, to experiment, to study, to model, to learn, to collaborate, to investigate. To discover. None of these activities, these performances, are captured in test cases. But testers keep talking about test cases as being central to the work, as though recipe books and written ingredient lists were the most important things about cooking.

I wish more testers understood that testing is not about “building confidence in the product”. That’s not our job at all. If you want confidence, go ask a developer or a marketer. It’s our job to find out what the product is and what it does, warts and all. To investigate. It’s our job to find out where confidence isn’t warranted; where there are problems in the product that threaten its value.

I wish more testers were more articulate in describing their models—their models of the product and how to cover it; their models of oracles, and how they recognize problems; their models of testing itself.

Of course, sometimes testers find those things hard to talk about because their intuitive models are vague and fuzzy. That’s one of the important tasks in the Rapid Software Testing space: to help people sharpen up their models so that they can sound like they know what they’re doing — while actually knowing what they’re doing. Then we can talk more precisely amongst ourselves about what we’re doing, and how we’re going to approach solving problems for our clients.

Some people are bothered by our focus on expressing things precisely. They complain “If we use jargon, the non-testers we’re talking to will get confused!” If you use jargon with people who don’t need to hear it, they may get confused, so don’t use jargon with them when there’s nothing at stake.

On the other hand, when non-testers hears “automated testing”, that affords the interpretation that testing can be automated. It can’t. That’s why, in Rapid Software Testing, we talk about automated checking: to alleviate confusion between things machines can do (checking) and things that you need clever humans to do (testing, which includes the process of developing and interpreting checks).

It’s okay for different testing and development communities to adopt and adapt their own ways of speaking. But amongst ourselves, within our communities and within our teams, we had damn well better learn to speak precisely, because imprecision is where lots of problems start, and where bugs live. Don’t jump on the newbies; help them learn and guide them along.

But I would say this to people who seek respect: getting straight on what things mean and how they matter is essential in real professions. [laughs] Doctors don’t say “virus, bacteria… who cares? The guy’s sick! He’s got… a… bad thing! And he needs… stuff to make him better! Pharmacist, give him… some... better-making-stuff and tell him to take it… whenever.” Fail to make appropriate distinctions, and the attempt to help the patient will fail. And then society gets antibiotic-resistant bacteria as a byproduct.

I wish more testers recognized that testing can be applied to anything—a running product, of course, but also to documents, diagrams, models, ideas. I often hear people saying “testers should do more than just testing,” and when I ask them what they mean by “just testing”, it turns out that their notion of testing is really impoverished.

Testing isn’t just operating a product and looking for bugs. Testing involves modeling and questioning and studying, making inferences, challenging beliefs. Generating, describing, refining, revising, abandoning, and recovering ideas, all the way through the project. Designing and performing experiments, not just by interacting with running code, but also by performing thought experiments on whatever we might have in front of us, from an idea to a document to a diagram to a program. Review is testing a story or some code. Critical thinking is testing an idea — helping people sharpen their understanding. Developing tools to help test is testing. [laughs] What’s with the “just testing” business? All that isn’t enough?

Hexawise: Can you describe a view or opinion about software testing that you have changed your mind about in the last few years? What caused you to change your mind?

Michael: Many years ago, when I first started out as a program manager, I thought it would be a good idea to write lots of things down, formally, in advance, and hand those instructions to programmers. After that, building the product and testing it would be simple.

Well, that sure sounded appealing, but it didn’t work very well most of the time. Some programmers really liked very specific instructions, and occasionally we could give them those when we had a really good idea of what we wanted.

What I was ignoring was all the work, all of the trial and error, all of the collective tacit knowledge that gets you to the point where you can make a lot of stuff explicit… but by then you don’t need most of it to be explicit, so making it explicit is a waste of time!

Excessive or premature formality is really dangerous for testing. Notice those adjectives: excessive, premature! Sometimes, in certain contexts, there are specific things that must be tested formally, in specific ways, or to check specific facts. That formality might be important because of technical risk, business risk, legal risk, risk to finances or to human health and safety.

Much of the time, though, your testing doesn’t need to be very formal. Even when you’re in a context that requires formal testing, you need plenty of excellent informal testing in order to get to excellent formal testing.

I see a lot of people investing in formality and documentation really early in the project. But I think it’s a mistake to do that before we’ve learned about the product and the problem space. That learning is a fundamentally exploratory process, one that can’t be formalized unless you want to suppress it or damage it or slow it down needlessly.

Testing isn’t just operating a product and looking for bugs. Testing involves modeling and questioning and studying, making inferences, challenging beliefs.

Hexawise: You have clearly articulated the inherent limitations of testing coverage metrics. For example, from your blog post: 100% Coverage is Possible, you state:

To claim 100% coverage is essentially the same as saying “We’ve looked for bugs everywhere!” For a skilled tester, any “100%” claim about coverage should prompt critical thinking: “How much” compared to what? 100% of what, specifically? Some model of X—which one? Whose model? How well does the “model of X” model reality? What does the model of X leave out of the universe of possible ways of thinking about X? And what non-X things should we also be considering when we’re testing?

What advice do you give to teams you work with who use quantitative coverage metrics?

Michael: [laughs] That’s a little like asking “What advice do you give to journalists who use spreadsheets?”

Start from the default premise that we don’t need them. Don’t start from the premise that we must hang a number on our coverage. Assume that we will describe our coverage. If there’s some way in which numbers will help, assume that whatever we decide to quantify will be based on some model of the software. All models focus on something and leave everything else out, and at best we’re only aware of some of that “everything else”.

After all, what might we be covering? We could decide to model our coverage in terms of how many lines of code, or branches, or conditions have been covered. We could count them. But not all lines of code (or branches, or conditions) are equally significant. And code coverage tools report on our code, but not necessarily the code in third-party libraries and frameworks, in the operating system, in the hardware platform.

Some people model coverage in terms of “requirements”. What they really mean by that is specific statements in requirements documents. Documents model ideas about the product.

So here’s one statement: “the site shall provide scheduling information for all flights on the airline’s network”. Here’s another: “when the customer is logged in, each page shall display the customer’s first and last names in the upper-right corner of the screen”. Are those statements of equivalent significance? How would we test them?

[laughs] Don’t say something silly, like you’ll fully test your product by having “one positive and one negative test case for each sub-requirement”! (The Wikipedia article on “test cases”, as of today, says exactly that.) Your testing of a product is a performance.

Sometimes a journalist will find it useful to illustrate a point with numbers, or a table, a spreadsheet. Baseball is a great example of a game where statistics help us to evaluate performances. Bill James, the guy at the centre of Moneyball, has shown how the numbers you choose can support arguments about a player’s value. But the magic of the Baseball Abstract, which was a kind of journal he wrote for many years, was that he told great stories, using stats to illustrate them. He also showed how stats don’t tell the whole story, and how they can mislead, too.

Sometimes journalists will quote figures from researchers, or use poll numbers to back a story. But the history of polls—particularly recent history—provides a great example of the ways we can be fooled by numbers.

I wrote these articles on talking about coverage a while ago, but I think they still stand up:

You’ll find useful stuff on measurement in Quality Software Management, Vol. 2: First Order Measurement (Weinberg) or the two e-books that make up that book, How to Observe Software Systems and Responding to Significant Software Events.

But if you want to quantify coverage, be skeptical. Be professionally unsure. Read "Software Engineering Metrics: What Do They Measure and How Do We Know" (Kaner and Bond). Read How Not to Be Wrong (Ellenberg). Read Proofiness (Seife). Read How to Lie with Statistics (Huff).

Industry Observations / Industry Trends

Hexawise: What software testing industry trends make you optimistic about the future? And which software testing industry trends make you concerned?

Michael: There’s been some degree of progress over the years in refining how people think about testing, but the people who are pushing for that are in a tiny minority. We’re a gaggle of hobbits facing an army with entrenched ideas that might as well have been handed down from orcs. Those ideas didn’t work 30 years ago and are crazily inefficient now. People still don’t think very critically about testing folklore and mythodology. (Yes, I said mythodology.)

Many people treat certain ideas as Grand Truths about testing. But many of the claims that people make—especially some of the testing tool vendors—are myths, or folklore, or simply invalid. People often say things that they haven’t thought about very deeply, and some of those things don’t stand up very well to critical scrutiny.

People don’t seem to notice how rickety everything is at the best of times. Have you tried to print something lately, using the new printer software that came out yesterday? Tried to buy a plane ticket? Tried to set up or top up a pay-as-you-go mobile hotspot when you’re travelling? I do all that stuff regularly, and it’s usually time-critical, and almost every time I enter a world of pain. Technology is complex, and our lives are complex, and the least little bump in the road can make the wheels fall off. Every day, as I try to use software, I feel like I’m being pecked to death by ducklings.

We want to do stuff like move money between accounts with our smart phones, securely, but the global financial system is at the mercy of institutions that are getting testing services from the lowest bidder, trying to find ways to make testing cheaper instead of more powerful and more risk focused. That life-critical medical software comes from the same larger community that brought us JavaScript and CSS. Yikes. [laughs]

I’m concerned that we’re still overly focused on testing the clockwork, the functional aspects of the product without thinking about how people will respond to it. As Harry Collins would say, machines are social prostheses. Like insulin pumps or artificial legs, they’re being plugged into human life and human purposes to do work that humans once did (or where humans with superpowers could do that work). But protheses do don’t what the real thing does, and the surrounding body has to adapt to that fact. Software doesn’t do what humans do, so humans often must adapt to it. We should all be asking: “how does the technology change us—for good and for ill?”

Staying Current / Learning

Hexawise: I see that you’ll be presenting at the StarEAST conference in May. What could you share with us about what you’ll be talking about? / What gave you the idea to talk about it?

Michael: I’ll be giving two tutorials. The first is about critical thinking for testers. We describe critical thinking like this: “thinking about thinking, with the intention of avoiding being fooled”. That’s central to our work as testers. Testers think critically about software to help our clients make informed decisions about whether the product they’ve got is the product they want.

Many people treat certain ideas as Grand Truths about testing. But many of the claims that people make—especially some of the testing tool vendors—are myths, or folklore, or simply invalid. People often say things that they haven’t thought about very deeply, and some of those things don’t stand up very well to critical scrutiny. That’s one of the reasons I started developing and teaching this class in 2008. I was dismayed that testers and other people in software development were accepting certain myths about testing unskeptically.

Not only that, but testers and teams allow themselves to be fooled by focusing on confirmation, rather than challenging the software. So, in the class, we talk about the ways in which words and models can fool us. In a safe environment, it’s okay—and even fun—to be fooled, and to figure out how not to be fooled quite so easily the next time.

On Tuesday, I present on one-day class called “A Rapid Introduction to Rapid Software Testing.” RST (the methodology) is focused on the mindset and the skill set of the individual tester. It’s about focusing testing on the mission, rather than on bureaucracy and paperwork. We sometimes joke that RST (the class) is a three day class in which we attempt to cover about nine days of material. [laughs] In “A Rapid Introduction to Rapid Software Testing”, I try to do the three-day class in one day. It is a rapid introduction, but we’ll be able to explore some of the central ideas.

Hexawise: What software testing-related books would you recommend should be on a tester’s bookshelf?

Michael: The two that I can recommend that are explicitly about software testing are Lessons Learned in Software Testing and Perfect Software and Other Illusions about Testing.

But you asked about testing-related books, and there’s an absurd number of those. Thinking Fast and Slow is about critical thinking and how easy it is for us to get fooled. The Shape of Actions (Collins and Kusch) is about what machines can and cannot do, and the Golem series (Collins and Pinch) is about the nature of science, technology, and medicine. Code (Petzold) is about how data is represented and processed on digital computers. Everyday Scripting in Ruby is a decent, tester-focused book on creating little tools and doing work with scripts. And the list of Jerry Weinberg books is plenty long on its own: Exploring Requirements, and An Introduction to General Systems Thinking, and Errors, and Agile Impressions.

James Bach and I have not written a book on software testing together. But you could read our blogs, which would be like reading a long-ish book of essays on testing.

Profile

Michael Bolton is a consulting software tester and testing teacher who helps people solve testing problems that they didn’t realize they could solve. He is the co-author (with senior author James Bach) of Rapid Software Testing, a methodology and mindset for testing software expertly and credibly in uncertain conditions and under extreme time pressure.

With twenty-five years of experience testing, developing, managing, and writing about software, Michael has led DevelopSense, a Toronto-based testing and development consultancy, for the past fifteen years. Previously, he was with Quarterdeck Corporation where he managed the company’s flagship products and directed project and testing teams both in-house and worldwide.

Links:

Blog: DevelopSense

Twitter: @michaelbolton

Related posts: Testing Smarter with Alan Page - Testing Smarter with Dorothy Graham

By: John Hunter and Justin Hunter on Apr 3, 2017

Categories: Interview, Software Testing, Testing Smarter with...

This interview with Mike Bland is part of our series of “Testing Smarter with…” interviews. Our goal with these interviews is to highlight insights and experiences as told by many of the software testing field’s leading thinkers.

Mike Bland aims to produce a culture of transparency, autonomy, and collaboration, in which “Instigators” are inspired and encouraged to make creative use of existing systems to drive improvement throughout an organization. The ultimate goal of such efforts is to make the right thing the easy thing. He's followed this path since 2005, when he helped drive adoption of automated testing throughout Google as part of the Testing Grouplet, the Test Mercenaries, and the Fixit Grouplet.


Mike Bland

This post includes highlights from our full interview with Mike Bland. The full interview is long and packed with great thoughts.

Personal Background

Hexawise: If you could write a letter and send it back in time to yourself when you were first getting into software testing, what advice would you include in it?

Mike: When I first started practicing automated testing and had a lot of success with it, I couldn’t understand why people on my team wouldn’t adopt it despite its “obvious” benefits. One of the biggest things that experience, reading, and reflection has afforded me is the perspective to realize now that different people adopt change differently, at different rates and for different reasons, and that you’ve got to create the space for everyone to adapt accordingly.

As I say in my most recent presentation, “The Rainbow of Death”, metrics and arguments are far from sufficient to inspire action in either the skeptical or the powerless, and the greater challenge is to create the cultural space necessary for lasting change.

Oh, and you’ve got to repeat yourself and say the same thing different ways multiple times—a lot.

Hexawise: What change management lessons did you learn while driving adoption of test automation methods at Google between 2005-2010? Which of those lessons were applicable when you were involved in the recent U.S. federal government effort to bring in talented tech people to bring new ways of working with technology into the government? Which of those lessons were not?

Mike: The top objective is to make the right thing the easy thing. Once people have the knowledge and power to do the right thing the right way, they won’t require regulation, manipulation, or coercion—doing things any other way will cease to make any sense.

Most of what I learned in terms of specific approaches to supplying the necessary knowledge and power has come from trying different things and seeing what sticks—and I’m still working to make sense of why certain things stuck, years after the fact. The most important insight, as mentioned earlier, is that different people adopt change differently, for different reasons, and as a result of different stimuli. Geoffrey A. Moore’s Crossing the Chasm was the biggest eye-opener in this regard. Then, years later, when I saw fellow ex-Googler Albert Wong present his “Framework for Helping” to describe his first experience in the U.S. Digital Service, I instantly saw it snapping in place across the chasm, describing how the Innovators and Early Adopters from Moore’s model—who I like to call “Instigators”—need to fulfill an array of functions in order to connect with and empower the Early Majority on the other side of the chasm. Of course, through the filter of my own twisted sense of humor, I thought “Rainbow of Death” might make the model stick in people’s brains a little better.

So these models helped provide context for why the specific things the Testing Grouplet did worked; and how, despite the fact that there were many scattered, parallel efforts underway, they ultimately served to reinforce one another, rather than creating confusion and chaos. Of course, Google’s open communication channels and the Testing Grouplet’s shared vision—that emerged two years into our five year run—helped keep everything aligned. The point being, don’t wait for the clear vision and perfect plan up-front—start doing things and pay attention to what’s working, and why, and develop your plans as you go. That’s just good Agile practice, isn’t it?

And did I mention that you have to reiterate things you’ve already said in different language over and over—like, a lot?

Every lesson applied, in that the real lessons were about human nature, not technology. Google disabused me of the notion that one metric, one tool, or one method of persuasion would suffice to change an entire population's behavior. In other words, there's no silver bullet.

See the full interview for useful details.

The top objective is to make the right thing the easy thing. Once people have the knowledge and power to do the right thing the right way, they won’t require regulation, manipulation, or coercion—doing things any other way will cease to make any sense.

Hexawise: Describe a testing experience you are especially proud of. What discovery did you make while testing and how did you share this information so improvements could be made to the software?

Mike: Probably like many folks, I remember my first time the most vividly. Immediately on the heels of a death march—when my team barely got a steaming pile of other people’s code to meet a critical spec by a harsh deadline and very nearly would’ve killed one another were it not for Strongbad’s Emails to keep us one hair’s breadth away from going completely insane—we got some time and freedom to try to make the program faster.

I’d gotten the idea that we needed to rewrite a particular subsystem to take advantage of data we weren’t even using, and at about the same time, I happened to read an issue of the C/C++ Users Journal that had an article on using CppUnitLite, I believe. Unit testing sounded like a neat idea, so I practiced it at the same time I started rewriting this subsystem from scratch.

In the end, my new subsystem was rock-solid and improved performance by a factor of 18x. When a couple bugs came up, I diagnosed and fixed them very, very quickly, when the norm was on the order of weeks or months. It totally transformed our relationship with our client.

See the full interview for additional useful details.

Hexawise: In watching your videos and reading your content online your ideas resonate with those of W. Edwards Deming, Russell Ackoff and Peter Senge from management, culture change and systems thinking perspectives. Who are your greatest influence in this area?

Mike: I’m a little ashamed to admit I haven’t read any of their stuff, or at least not much. Certainly what little I’ve gleaned of Deming resonates with my experience. I’ve begun reading Senge’s The Fifth Discipline, and while the introduction resonated very clearly, I’ve not yet read further. Ackoff is a new name for me (and thanks for the tip!). That said, it is gratifying when I do read an established author and find that, yep, more learned minds than mine have clearly articulated widely-accepted concepts that I’ve only figured out due to trial, error, and intuition.

In fact, one of the things I’m trying to do moving forward is to go back through the literature and connect it to the experiences I’ve had—not just for my own validation, but to reassure my audience and clients that the things I’ve done and the things I recommend aren’t all crazy talk. I’ve got Geoffrey A. Moore’s Crossing the Chasm model combined with Albert Wong’s model to form the Rainbow of Death, which comprises the core of my narrative now; and I’ve also recently added a very high-level view of Kurt Lewin’s theory of social change, which someone only recently suggested to me.

Views on Software Testing

Hexawise: In your online presentation, Making the Right Thing the Easy Thing, you note: [Use] "amplifying feedback loops to make sure knowledge is shared where needed as quickly and clearly as possible." How do you suggest this idea be applied by those involved with software testing?

Mike: Heh, that’s a paraphrase of the Second Way of Devops (out of Three), originally articulated by Gene Kim. Clearly the extent to which you can automate testable cases, to make it easy and fast to do, the better. People need to know that there are different kinds of automated tests for different levels of the software—they need to learn how to do the right thing the right way. Once developers in particular have gotten some traction with writing automated tests, then folks performing manual or system-level automated testing won’t waste their time catching (and re-catching!) bugs the developers could’ve easily caught, and can focus on truly pushing the limits of the software—reporting not just on whether it meets functional and nonfunctional requirements, but on the overall quality of the product.

In other words, a healthy balance of automated testing and manual testing plays to the strengths of all the humans and machines involved. When you’ve got optimal resource utilization happening, you eliminate a lot of both physical (in terms of slowness) and human friction, and a feeling of true partnership can take hold. Testers aren’t just the people reminding you that your code isn’t perfect—you’ve already reminded yourself of that through your own automated tests!—they’re the ones helping you make it even better.

it’s not about defects; it’s about feedback and collaboration. If you arrange incentives to produce an adversarial relationship between team members, e.g. if developers are incentivized to minimize defects and testers are incentivized to report defects, then that’s a house divided against itself.

Hexawise: What do you wish more developers, business analysts, and project managers understood about software testing?

Mike: Oh my. For one, it’s not about defects; it’s about feedback and collaboration. If you arrange incentives to produce an adversarial relationship between team members, e.g. if developers are incentivized to minimize defects and testers are incentivized to report defects, then that’s a house divided against itself. Some people think a degree of competition and/or adversarialism is a good thing, but when it comes to producing a product as a team—i.e. achieving a mission—you should keep it to a minimum in favor of fostering a spirit of collaboration.

Collaboration doesn’t mean blind consensus; it means communicating honestly in an environment in which we feel safe to do so, in which we share criticism in a spirit of mutual self-interest, not cutthroat competition.

One test type does not fit all. First, in terms of automated tests, unit testing can find a truly large number of errors, very quickly and cheaply, and tends to encourage better code quality (i.e. readability, maintainability, extensibility) overall. Integration tests can shake out errors and ambiguities between component contracts. High-level, developer-written system tests (as opposed to more extensive system tests developed by a dedicated tester) can quickly affirm that the entire product is in a buildable, runnable state. All of this “white box” testing by the developers is essential to giving the testers as high-quality a product as possible, so they can apply their “black box” techniques to push the product to its limits, rather than waste time alerting developers to defects they could’ve much more quickly, easily, and cheaply discovered themselves.

To this last point, I like to point to the examples of goto fail and Heartbleed. So many Internet “experts” threw up their hands and claimed that bugs like these were “too hard to test”. In both cases, after 2.5 years out of the industry (another story), I spent an evening diving into code I’d never seen before and wrote a test to reproduce each bug and validate its fix. After that, some liked to say, “Oh well, lots of other tools and techniques could’ve found these bugs.”

My claim isn’t that automated testing would’ve been the only way; my claim is that the discipline of automated testing likely would’ve prevented these bugs from ever existing even before writing a single test. With goto fail, the offending block of code was copied and pasted throughout the file six times! It was just that one of the six contained the errant “goto fail” line. But as I demonstrated with my version of the “fix”, extracting a common function and testing that six ways from Sunday likely would’ve avoided the problem entirely. In the case of Heartbleed, it was a failure to validate that an input buffer was actually as long as the user-supplied length indicated. Testing that kind of corner condition is unit testing 101, and the kind of thing you become more sensitive to every time you write a line of code once you’re in the habit of testing.

Hence, as difficult as it would’ve been for manual testing to discover these errors, and as long as it took for them to get shaken out months or years after their widespread deployment—Heartbleed via third-party fuzz testing, goto fail who knows how—both very, very likely could’ve been stopped dead in their tracks (or never would’ve existed!) if the developers were in the everyday habit of unit testing their code.

Hexawise: Our CTO, Sean Johnson, shared your memorably-named "Rainbow of Death" presentation with our management team. We absolutely loved it. In your presentation, you describe a series of concrete, practical steps you and your colleagues at Google took over the course of 5+ years to overcome resistance to change, educate teams, and successfully achieve broad adoption of automated testing efforts at Google across many teams, including lots of teams that were initially very change resistant. Can you please describe for our readers 2 or 3 noteworthy aspects of that change management journey?

Mike: What I hope the Rainbow of Death model, in combination with Geoffrey A. Moore’s Crossing the Chasm model, make apparent is that different people adopt change differently. There are many needs that need to be met by and for many different people, and the chances of figuring out the perfect plan to execute before taking any action are practically zero. After all, don’t the Agile and DevOps models that are all the rage comprise tools and practices for adapting to change, for performing experiments and adjusting course based on feedback? Organizational change is no different, yet many people remain conditioned to expect waterfall-like solutions to their social problems.

image showing items supporting change in the organization


Also, I mention in the talk that “The problem you want to solve may not be the problem you have to solve first.” In our case, we wanted to solve the problem of developers not writing enough automated tests. But first, we had to solve two other problems: People back then had very little exposure to or experience with automated testing, leading to the “My code is too hard to test” excuse, because they had no idea how to test it, or to write testable code to begin with.

The second problem was that the tools at the time couldn’t keep up with the growth of the company, its products, and its code base. It was growing ever more painful to write any code to begin with, yet delivery pressure was intense and Imposter Syndrome was rampant—on top of the fear of admitting your code might contain flaws, how could you make any time to learn how to write automated tests to begin with? Hence the “I don’t have time to test” excuse.

So we couldn’t just say “Testing is good! Yay testing! Please write moar testz!”

I think this mix of perspective, empathy, creativity, collaboration, tenacity, and patience is crucial to changing not only tech organizations, but society at large. I hope to put this notion, and the Rainbow of Death model in particular, to the test continuously throughout the remainder of my career.

my advice to both developers and testers is to identify the priorities, the social structures and dynamics at play in the organization. How can you work with these structures and dynamics instead of against them—or do you need to create a culture of open communication and collaboration in parallel with (or even before) communicating the testing message?

Industry Observations / Industry Trends

Hexawise: Large companies often discount the importance of software testing. What advice do you have for software testers to help their organizations understand the importance of expecting more from the software testing efforts in the organization?

Mike: Sadly, there’s no one message that works for every company, every culture, everywhere. It’s up to the Instigators in each environment to take the timeless principles I believe are essential—that testing is about feedback and collaboration, that different types of tests all catch different and important bugs, that developers and testers have different and mutually-reinforcing roles to play—and find the right cultural hooks to hang those messages on. In the case of Google, it took the Testing Grouplet five years to figure out and successfully implement, and it took an array of parallel efforts across multiple groups to saturate the culture with the message, not just one magical tool or technique or team to bind them all.

So my advice to both developers and testers is to identify the priorities, the social structures and dynamics at play in the organization. How can you work with these structures and dynamics instead of against them—or do you need to create a culture of open communication and collaboration in parallel with (or even before) communicating the testing message? This is the punchline of my Rainbow of Death presentation: The problem you want to solve may not be the problem you have to solve first, and the Standard Narrative from which all the problems emerged will not produce any solutions—though it may provide the keys necessary to unlock effective solutions.

At Google, it was the Testing Grouplet’s Test Certified program and all the other education and tooling efforts supporting it that provided the right hook—after two years of experimentation and reflection! But don’t focus on what Test Certified was comprised of: focus on why that approach worked for us, and see if that reflection inspires an approach that will work for your company.

In addition to that, it probably wouldn’t hurt to remind anyone who’ll listen of goto fail and Heartbleed, and how basic unit testing practices and the coding habits they encourage could’ve prevented these potentially catastrophic defects from even being written in the first place.

Hexawise: The story of your team's journey is fantastic. We highly recommend it to IT organizations embarking on any large improvement effort. Thank you for sharing it. We've recently started using elements of your approach to help our clients successfully adopt test optimization approaches at scale in their organizations.

Mike: That’s incredibly gratifying to hear! Please keep me in the loop of how well it’s working for you and your clients. Just as the impact of Testing Grouplet’s efforts were far greater than the sum of any individual part, and as my Rainbow of Death presentation benefited enormously from the input of trusted fellow Instigators to help illustrate it, I’m sure there are many more insights and improvements waiting to emerge from the model once more people have applied it farther and wider than I ever could on my own!

Hexawise: Do you believe the DevOps movement is resulting in better software testing within organizations? Do you see any other trends that software testers could leverage to promote improved application of software testing practices?

See full interview for Mike’s response.

Staying Current / Learning

Hexawise: What software testing-related books would you recommend should be on a tester’s bookshelf?

Mike: Sadly, I’m not current enough to make any solid recommendations. In my career, I’ve moved more into the culture change space than being a 100% in-the-trenches practitioner. That said, I certainly did years of quality time with my old “library” of programming and algorithms books, and was fortunate to be part of a culture that itself generated a broad swath of automated testing knowledge. Though I don’t keep up with the details of the latest developments, that time spent internalizing the core principles has served me very well throughout my career.

That said, I’m sure that great books exist, and people dedicated to the craft would do themselves a great service by discovering them and spending years plumbing their depths, rather than trying to read every book on the subject forevermore. That’s the model that worked for me, at least; but perhaps a more voracious reading regimen suits you better. Everybody’s different.

See the full interview for more questions and answers.

Profile

Mike aims to produce a culture of transparency, autonomy, and collaboration, in which “Instigators” are inspired and encouraged to make creative use of existing systems to drive improvement throughout an organization. The ultimate goal of such efforts is to make the right thing the easy thing. He's followed this path since 2005, when he helped drive adoption of automated testing throughout Google as part of the Testing Grouplet, the Test Mercenaries, and the Fixit Grouplet. He was instrumental in the execution of Test Certified and Testing on the Toilet, and the four company-wide Fixits he organized led to the development and rollout of the Test Automation Platform. His account of Google’s automated testing adoption also appears as a case study in The DevOps Handbook by Gene Kim, et. al.

He also served as a member of the Websearch Infrastructure team, which practiced DevOps before he was aware it had a name. Frequently working in concert with other indexing infrastructure teams, he also worked closely with Release Engineers and Site Reliability Engineers to package, release, deploy, and monitor multiple indexing services.

Most recently he served as Practice Director at 18F, a technology team within the U.S. General Services Administration, where he personally launched and drove several initiatives to increase 18F’s capability as a learning organization, including the Pages platform, the Guides series, and the Handbook.

Links:

This post includes highlights from our full interview with Mike Bland. The full interview is long and packed with great thoughts.

By: John Hunter on Mar 27, 2017

Categories: Interview, Software Testing, Testing Smarter with..., Testing Strategies

This interview with Alan Page is part of our series of “Testing Smarter with…” interviews. Our goal with these interviews is to highlight insights and experiences as told by many of the software testing field’s leading thinkers.

Alan Page has been a software tester (among other roles) since 1993 and is currently the Director of Quality for Services at Unity. Alan spent over twenty years at Microsoft working on a variety of operating systems and applications in nearly every Microsoft division.


Alan Page

Personal Background

Hexawise: Looking back on over 20 years in software testing at Microsoft can you describe a testing experience you are especially proud of?

Alan: There are a lot of great experiences to choose from - but I'm probably most proud of my experiences on the Xbox One team. My role was as much about building a community of testers across the the Xbox ecosystem (console, services, game development) as it was about testing and test strategy for the console. We had a great team of testers, including a handful of us with a lot of testing experience under our belts. We worked really well together, leveraged each others strengths, and delivered a product that has had nearly zero quality issues since the day of release.

Hexawise: What did you enjoy about working in such a large software company, Microsoft, for so long?

Alan: The only way I survived at Microsoft so long was that I could change jobs within the company and get new experiences and challenges whenever I felt I needed them. I love to take on new challenges and find things that are hard to do - and I always had that opportunity.

The biggest thing I look for in testers is a passion and ability to learn. I've interviewed hundreds of testers... The testers who really impress me are those who love to learn - not just about testing, but about many different things. Critical thinking and problem solving are also quite important.

Hexawise: What new challenges are you looking forward to tackling in your new role at Unity?

Alan: I'm looking forward to any and all challenges I can find. Specifically, I want to build a services testing community at Unity. Building community is something I feel really strongly about, and from what I've seen so far, I think there are a lot of opportunities for Unity testers to learn a lot from each other while we play our part in the coming growth of Unity services.

Views on Software Testing

Hexawise: What thoughts do you have in involving testers in A/B and multivariate testing? That stretches the bounds of how many people categorize testers but it could be a good use of the skills and knowledge some testers process.

Alan: My approach to experimentation (A/B testing) is that there are three roles needed. Someone needs to design the experiment. A product owner / program manager often plays this role and will attempt to figure out the variation (or variations) for the experiment as well as how to measure the business or customer value of both the control (original implementation) and treatment / experiment.

The second role is the implementer - this is typically a developer or designer depending on the nature of the experiment. Implementing an experiment is really no different than implementing any other product functionality.

The final role is the analyst role - someone needs to look at the data from the experiment, as well as related data and attempt to prove whether the experiment is a success (i.e. the new idea / treatment is an improvement) or not. I've seen a lot of testers be successful in this third role, and statistics and data science in general, are great skills for testers to learn as they expand their skill set.

image of book cover for How We Test Software at Microsoft

Hexawise: During your 20 years at Microsoft you have been involved in hiring many software testers. What do you look for when choosing software testers? What suggestions do you have for those looking to advance in their in software testing career?

Alan: The biggest thing I look for in testers is a passion and ability to learn. I've interviewed hundreds of testers, including many who came from top universities with advanced degrees who just weren't excited about learning. For them, maybe learning was a means to an end, but not something they were passionate about.

The testers who really impress me are those who love to learn - not just about testing, but about many different things. Critical thinking and problem solving are also quite important.

As far as suggestions go, keep building your tool box. As long as you're willing to try new things, you'll always be able to find challenging, fun work. As soon as you think you know it all, you will be stuck in your career.

Combinatorial testing is actually pretty useful in game testing. For example, consider a role-playing game with six races, ten character classes, four different factions, plus a choice for gender. That's 480 unique combinations to test! Fortunately, this has been proven to be an area where isolating pairs (or triples) of variations makes testing possible, while still finding critical bugs.

Hexawise: Do you have a favorite example of a combinatorial bug and how it illuminated a challenge with software testing?

Alan: I was only indirectly involved in discovering this one, but the ridiculously complex font-picker dialog from the Office apps was a mess to test - but it was tested extensively (at least according the person in charge of testing it). A colleague of mine showed them the all pairs technique, and they used it to massively decrease their test suite - and found a few bugs that had existed for years.

Hexawise: It seems to me that testing games would have significant challenges not found in testing fairly straightforward business applications. Could you share some strategies for coping with those challenges?

Alan: Combinatorial testing is actually pretty useful in game testing. For example, consider a role-playing game with six races, ten character classes, four different factions, plus a choice for gender. That's 480 unique combinations to test! Fortunately, this has been proven to be an area where isolating pairs (or triples) of variations makes testing possible, while still finding critical bugs.

Beyond that, testing games requires a lot of human eyeballs and critical thinking to ensure gameplay makes sense, objects are in the right places, etc. I've never seen a case where automating gameplay, for example, has been successful. I have, however, seen some really innovative tools written by testers to help make game testing much easier, and much more effective.

Hexawise: That sounds fascinating, could you describe one or more of those tools?

Alan: The games test organization at Microsoft wrote a few pretty remarkable tools. One linked the bug tracking system to a "teleportation" system in the game under test, including a protocol to communicate between a windows PC and an xbox console.

This enabled a cool two-way system, where a tester could log a bug directly from the game, and the world coordinates of where the bug occurred were stored automatically in the bug report. Then, during triage or debugging, a developer / designer could click on a link in the bug report, and automatically transport to the exact place on the map where the issue was occurring.

Hexawise: What type of efforts to automate gameplay came close to providing useful feedback? What makes automating gameplay for testing purposes ineffective?

Alan: I would never automate gameplay. There are too many human elements needed for a game to be successful - it has to be fun to play - ideally right at that point between too challenging, and too easy. If the game has a story line, a tester needs to experience that story line, evaluate it, and use it as a reference when evaluating gameplay.

Tools to evaluate cpu load or framerate during gameplay are usually a much better investment than trying to simulate gameplay via automation.

That said, there are some "what if" scenarios in games that may provide interesting bugs. For example simulating a user action (e.g. adding and removing an item) thousands of time may reveal memory leaks in the application. It really comes down to test design and being smart about choosing what makes sense to automate (and what doesn't make sense).

Hexawise: What is one thing you believe about software testing that many smart testers disagree with?

Alan: I don't believe there's any value from distinguishing "checks" from tests. I know a lot of people really like the distinction, but I don't see the value. Of course, I recognize that some testing is purely binary validation, but this is a(nother) example of where choosing more exact words in order to discern meaning leads to more confusion and weird looks than it benefits the craft of testing.

Industry Observations / Industry Trends

Hexawise: Do you believe testing is becoming (or will become) more integrated with the software development process? And how do you see extending the view of the scope of testing to include all the way from understanding customer needs to reviewing actual customer experience to drive the testing efforts at an organization.

Alan: I believe testing has become more integrated into software development. In fact, I believe that testing must be integrated into software development. Long ship cycles are over for most organizations, and test-last approaches, or approaches that throw code to test to find-all-the-bugs are horribly inefficient. The role of a tester is to provide testing expertise to the rest of the team and accelerate the entire team's ability to ship high-quality software. Full integration is mandatory for this to occur.

It's also important for testers to use data from customer usage to help them develop new tests and prioritize existing bugs. We've all had conversations before about whether or not the really cool bug we found was "anything a customer would ever do" - but with sufficient diagnostic data in our applications or services, we can use data to prove exactly how many customers could (or would) hit the issue. We can also use data to discover unexpected usage patterns from customers, and use that knowledge to explore new tests and create new test ideas.

There's an important shift in product development that many companies have recognized. They've moved from "let's make something we think is awesome and that you'll love" - i.e. we-make-it-you-take-it to "we want to understand what you need so we can make you happy". This shift cannot happen without understanding (and wallowing in) customer data.

Hexawise: In your blog post, Watch out for the HiPPO, you stated: "What I’ve discovered is that no matter how strongly someone feels they “know what’s best for the customer”, without data, they’re almost always wrong." What advice do you have for testers for learning what customers actually care about? To me, this points out one of the challenges software testers (and everyone else actually) faces, which is the extent to which they are dependent on the management system they work within. Clayton Christensen's Theory of Jobs to Be Done is very relevant to this topic in my opinion. But many software testers would have difficulty achieving this level of customer understanding without a management system already very consistent with this idea.

Alan: Honestly, I don't know how a product can be successful without analysis and understanding of how customers use the product. The idea of a tester "pretending to be the customer" based purely on their tester-intuition is an incomplete solution to the software quality problem. If you hear, "No customer would ever do that", you can either argue based on your intuition, or goo look at the data and prove yourself right (or wrong).

There's an important shift in product development that many companies have recognized. They've moved from "let's make something we think is awesome and that you'll love" - i.e. we-make-it-you-take-it to "we want to understand what you need so we can make you happy". This shift cannot happen without understanding (and wallowing in) customer data. Testers may not (and probably won't) create this system, but any product team interested in remaining in business should have a system for collecting and analyzing how customers use their product.

Hexawise: I agree, this shift is extremely important. Do you have an example of how using such a deep understanding of users influenced testing or how it was used to shift the software development focus. Since software testing is meant to help make sure the company provides software that users want it certainly seems important to make sure software testers understand what users want (and don't want).

Alan: First off, I prefer to think of the role of test as accelerating the achievement of shipping quality software; which is similar to making sure the company provides software customers want, but (IMO), is a more focused goal.

As far as examples go...how many do you want? Here a few to ponder:

  • We tracked how long people ran our app. 45% ran it continuously - all the time. This is the way we ran the app internally. But another 45% ran it for 10 minutes or less. We had a lot of tests that tried to mimic a week of usage and look for memory leaks and other weirdness, but after seeing the data, we ended up doing a lot more testing (and improving) of start up and shut down scenarios.
  • We once canceled a feature after seeing that the data told us that it was barely used. In this case, we didn't add the tracking data until late (a mistake on our part). Ideally, we'd discover something like this earlier, and than redesign (rather than cancel)
  • We saw that a brand new feature was being used a lot by our early adopters. This was pretty exciting...but as testers, we always validate our findings. It turned out after a bit more investigation, that the command following usage of the brand new feature was almost always undo. Users were trying the feature... but they didn't like what it was doing.

Staying Current / Learning

Hexawise: What advice do you have for people attending software conferences so that they can get more out of the experience?

Alan: Number one bit of advice is to talk to people. In fact, I could argue that you get negative value (when balanced against the time investment) if you show up and only attend talks. If there's a talk you like in particular, get to know the speaker (even us introverts are highly approachable - especially if you bring beer). Make connections, talk about work, talk about challenges you have, and things you're proud of. Take advantage of the fact that there are a whole lot of people around you that you can learn from (or who can learn from you).

Additionally, if you're in a multi-track conference, feel free to jump from talk to talk if you're not getting what you need - or if it just happens that two talks that interest you are happening at the same time.

Hexawise: How do you stay current on improvements in software testing practices; or how would you suggest testers stay current?

Alan: I read a lot of testing blog posts (I use feedly for aggregation). I subscribe to at least fifty software development and test related blogs. I skim and discard liberally, but I usually find an idea or two every week that encourages me to dig deeper.

Biggest tip I have, however, is to know how essential learning is to your success. Learning is as important to the success of a knowledge worker as food is to human life. Anyone who thinks they're an "expert" and that learning isn't important anymore is someone on a career death spiral.

Profile

image of book cover for The A Word by Alan Page

Alan has been a software tester (among other roles) since 1993 and is currently the Director of Quality for Services at Unity. Alan spent over twenty years at Microsoft working on a variety of operating systems and applications in nearly every Microsoft division. Alan blogs at angryweasel.com, hosts a podcast (w/ Brent Jensen) at angryweasel.com/ABTesting, and on occasion, he speaks at industry testing and software engineering conferences.

Links Books: How We Test Software at Microsoft, The A Word

Blog: Tooth of the Weasel

Podcast: AB Testing

Twitter: @alanpage


Related posts: Testing Smarter with Dorothy Graham - Testing Smarter with James Bach

By: John Hunter on Mar 16, 2017

Categories: Combinatorial Software Testing, Customer Success, Software Development, Software Testing, Testing Smarter with..., Interview

This interview with Dorothy Graham is part of our series of “Testing Smarter with…” interviews. Our goal with these interviews is to highlight insights and experiences as told by many of the software testing field’s leading thinkers.

Dorothy Graham has been in software testing for over 40 years, and is co-author of 4 books: Software Inspection, Software Test Automation, Foundations of Software Testing and Experiences of Test Automation.


Dorothy Graham

Personal Background

Hexawise: Do your friends and relatives understand what you do? How do you explain it to them?

Dot: Some of them do! Sometimes I try to explain a bit, but others are just simply mystified that I get invited to speak all over the world!

Hexawise: If you could write a letter and send it back in time to yourself when you were first getting into software testing, what advice would you include in it?

Dot: I would tell myself to go for it. When I started in testing it was definitely not a popular or highly-regarded area, so I accidentally made the right career choice (or maybe Someone up there planned my career much better than I could have).

Hexawise: Which person or people have had the greatest influence on your understanding and practice of software testing?

Dot: Tom Gilb was (and is) a big influence, although in software engineering in general rather than testing. The early testing books (Glenford Myers, Bill Hetzel, Cem Kaner, Boris Beizer) were very influential in helping me to understand testing. I attended Bill Hetzel’s course when I was just beginning to specialize in testing, and he was a big influence on my view of testing back then. Many discussions about testing with friends in the Specialist Group In Software Testing (SIGIST), and colleagues at Grove, especially Mark Fewster.

Hexawise: Failures can often lead to interesting lessons learned. Do you have any noteworthy failure stories that you’d be willing to share?

Dot: I was able to give a talk about a failure story in Inspection, where a very successful Inspection initiative had been “killed off” by a management re-organisation that failed to realise what they had killed. There were a lot of lessons about communicating the value of Inspection (which also apply to testing and test automation). The most interesting presentation was when I gave it at the company where it had happened a number of years before; one (only one) person realized it was them.

One fallacy is in thinking that the automation does only what is done by manual tests. There are often many things that are impossible or infeasible in manual testing that can easily be done automatically.

Hexawise: Describe a testing experience you are especially proud of. What discovery did you make while testing and how did you share this information so improvements could be made to the software?

Dot: One of the most interesting experiences I had was when I was doing a training course in Software Inspection for General Motors in Michigan, and in the afternoon, when people were working on their own documents, several people left the room quite abruptly. I later found out that they had gone to stop the assembly line! They were Inspecting either braking systems or cruise control systems.

When I was working at Ferranti on Police Command and Control Systems, I discovered that the operating system’s timing routines were not working correctly. This was critical for our system, as we had to produce reminders in 2 minutes and 5 minutes if certain actions had not been taken on critical incidents. I wrote a test program that clearly highlighted the problems in the OS and sent it to the OS developers, wondering what their reaction would be. In fact they were delighted and found it very useful (and they did fix the bugs). It did make me wonder why they hadn’t thought to test it themselves though.

Views on Software Testing

Hexawise: In That's No Reason to Automate, you state "Testing and automation are different and distinct activities." Could you elaborate on that distinction?

Dot: By “automation”, I am referring to tools that execute tests that they have been set up to run. Execution of tests is one part of testing but testing is far more than just running tests. First what it is that needs to be tested? Then how best can that be tested - what specific examples would make good test cases? Then how are these tests constructed to be run? Then they are run, and afterwards, what was the outcome from the test compared to what was expected, i.e. did the software being tested do what it should have done? This is why tools don’t replace testers - they don’t do testing, they just run stuff.


Diagram from That's No Reason to Automate

Hexawise: Test automation allows for great efficiency. But there are limits to what can be effectively automated. How do you suggest testers determine what to automate and what to devote tester's precious time toward investigating personally?

Dot: There are two fallacies when people say “automate all the manual tests” (which is unfortunately quite common). The first one is that all manual tests should be automated - this is not true. Some tests that are now manual would be good candidates for automation, but other tests should always be manual. For example, “do these colours look nice?” or “is this what users really do?” or even “Captcha” where it should only work if a person not a machine is filling in a form. Think about it - if your automated test can do the Captcha, then the Captcha is broken.

Automating the manual tests “as is" is not the best way to automate, since manual tests are optimized for human testers. Machines are different to people, and tests optimized for automation may be quite different. And finally, a test that takes an inordinate amount of time to automate, if the test isn’t needed often, is not worth automating.

The other fallacy is in thinking that the automation does only what is done by manual tests. There are often many things that are impossible or infeasible in manual testing that can easily be done automatically. For example if you are testing manually, you see what is on the screen, but you don’t know if all the GUI (graphical user interface) objects are in the correct state - this could be checked by a tool. There are also types of testing that can only be done automatically, such as Monkey testing or High Volume Automated Testing.

Hexawise: Can you describe a view or opinion about software testing that you have that many or most smart experienced software testers probably disagree with? What led you to this belief?

Dot: One view that I feel strongly about, and many testers seem to disagree with this, is that I don’t think that all testers should be able to write code, or, what seems to be the prevailing view, that the only good testers are testers who can code. I have no objection to people who do want to do both, and I think this is great, especially in Agile teams. But what I object to is the idea that this has to apply to all testers.

There are many testers who came into testing perhaps through the business side, or who got into testing because they didn’t want to be developers. They have great skills in testing, and don’t want to learn to code. Not only would they not enjoy it, they probably wouldn’t be very good at it either! As Hans Buwalda says, “you lose a good tester and gain a poor programmer.” The ultimate effect of this attitude is that testing skills are being devalued, and I don’t think that is right or good for our industry!

The best way to have fewer bugs in the software is not to put them in in the first place. I would like to see testers become bug prevention advisors! And I think this is what happens on good teams.

Hexawise: What do you wish more developers, business analysts, and project managers understood about software testing?

Dot: The limitations of testing. Testing is only sampling, and can find useful problems (bugs) but there aren’t enough resources or time in the universe to “test everything”.

The best way to have fewer bugs in the software is not to put them in in the first place. I would like to see testers become bug prevention advisors! And I think this is what happens on good teams.

Hexawise: I agree that becoming bug prevention advisors is a very powerful concept. Software development organizations should be seeking to improve the entire system (don't create bugs, then try to catch them and then fix them - stop creating them). I recently discussed these ideas in Software Code Reviews from a Deming Perspective.

Dot: I had a quick look at the blog and like it - especially the emphasis on it being a learning experience. It seems to be viewing “inspection” in the manufacturing sense as looking at what comes off the line at the end. In our book “Software Inspection” (which is actually a review process), it is very much an up-front process. In fact applying inspection/reviews to things like requirements, contracts, architecture design etc is often more valuable than reviewing code.

I also fully agree with selecting what to look at - in our book we recommend a sampling process to identify the types of serious bug, then try to remove all other of that type - that way you can actually remove more bugs than you found.

Hexawise: In what ways would you see "becoming bug prevention advisors" happening? Done in the wrong way it can make people defensive about others advising them how to do their jobs in order to avoid creating bugs. What advice do you have for how software testers can make this happen and do it well?

Dot: I see the essential mind-set of the tester as asking “what could go wrong? or “what if it isn’t” [as it is supposed to be]. If you have a Agile team that includes a tester, then those questions are asked perhaps as part of pair working - that way the potential problems are identified and thought about before or as the code is written. The best way is close collaboration with developers - developers who respect and value the tester’s perspective.

Industry Observations / Industry Trends:

Hexawise: Your latest book, Experiences of Test Automation, includes an essay on Exploratory Test Automation. How are better tools making the ideas in this essay more applicable now than they were previously?

Dot: Actually this is now called High-Volume Automated Testing, which is a much better name for it. HiVAT uses massive amounts of test inputs, with automated partial oracles. The tests are checking that the results are reasonable, not exact results, and any that are outside those bounds are reported to humans to investigate. Harry Robinson described using this technique to test Bing - quite a large thing to test!

Hexawise: When looking to automate tests one thing people sometimes overlook is that given the new process it may well be wise to add more tests than existed before. If all test cases were manually completed that list of cases would naturally have been limited by the cost of repeatedly manually checking so many test cases in regression testing. If you automate the tests it may well be wise to expand the breadth of variations in order to catch bugs caused by the interaction of various parameter values. What are your thoughts on this idea?

Dot: Nice analogy - I like the term “grapefruit juice bugs”. Using some of the combinatorial techniques is a good way to cover more combinations in a reasonably rigorous way. Automation can help to run more tests (provided that expected results are available for them) and may be a good way to implement the combination tests, using pair-wise and/or orthogonal arrays.

Hexawise: In your experience how big of a problem is automation decay (automated tests not being maintained properly)? How do recommend companies avoid this pitfall?

Dot: This seems to be the most common way that test automation efforts get abandoned (and tools become shelfware), particularly for system-level test automation. It can be avoided by having a well-structured testware architecture in the first place, where maintenance concerns are thought about beforehand, using good programming practices for the automation code. Although fixing this later is possible, preventing it in the first place is far better.

Hexawise: Large companies often discount the importance of software testing. What advice do you have for software testers to help their organizations understand the importance of software testing efforts in the organization?

Dot: Unfortunately often the best way to get an organisation to appreciate testing is to have a small disaster, which adequate testing could have prevented. I’m not exactly suggesting that people should do this, but testing, like insurance, is best appreciated by those who suffer the consequences of not having had it. Perhaps just try to make people aware of the risks they are taking (pretend headlines in the paper about failed systems?)

Staying Current / Learning

Hexawise: I see that you’ll be presenting at the StarEast conference in May. What could you share with us about what you’ll be talking about?

Dot: I have two tutorials on test automation. One is aimed mainly at managers who want to ensure that a new (or renewed) automation effort gets off “on the right foot”. There are a number of things that should be done right at the start to give a much better chance of lasting success in automation.

My other tutorial is about Test Automation Patterns (using the wiki). Here we will look at common problems on the technical side (rather than Management issues), and how people have solved these problems - ideas that you can apply. This also a code-free tutorial - we are talking about generic technical issues and patterns, whatever test execution tool you use.

Hexawise: What advice do you have for people attending software conferences so that they can get more out of the experience?

Dot: Think about what you want to find out about before you come, and look through the programme to decide which presentations will be most relevant and helpful for you. It’s very frustrating to realise at the end of the conference that you missed the one presentation that would have given you the best value, so do your homework first!

But also realise that the conference is more than just attending sessions. Do talk to other delegates - find out what problems you might share (especially if you are in the same session). You may want to skip a session now and then to have a more thorough look around the Expo, or to have a deeper conversation with someone you have met or one of the speakers. The friends you make at the conference can be your “support group” if you keep in touch afterwards.

I don’t think that all testers should be able to write code, or, what seems to be the prevailing view, that the only good testers are testers who can code. I have no objection to people who do want to do both, and I think this is great, especially in Agile teams. But what I object to is the idea that this has to apply to all testers.

Hexawise: How do you stay current on improvements in software testing practices; or how would you suggest testers stay current?

Dot: I am fortunate to be invited to quite a few conferences and events. I find them stimulating and I always learn new things. There are also many blogs, webinars and online resources to help us all try to keep up to some extent.

Hexawise: What software testing-related books would you recommend should be on a tester’s bookshelf?

Mine, of course! ;-) I do hope that testers have a bookshelf! For testers, I recommend Lee Copeland’s book “A Practitioner’s Guide to Software Test Design” as a great introduction to many techniques. For a deep understanding, get Kem Caner’s book “The Domain Testing Workbook”. I also like “Lessons Learned in Software Testing” by Kaner, Bach and Pettichord, and “Perfect Software and other illusions about testing” by the great Gerry Weinberg. But there are many other books about testing too!

Hexawise: What are you working on now?

Dot: I am working on a wiki called TestAutomationPatterns.org with Seretta Gamba. This is a free resource for system level automation, with lists of different issues or problems and resolving patterns to help address them. There are four categories: Process, Management, Design and Execution. There is a “Diagnostic” to help people identify their most pressing issue, which then leads them to the patterns to help. We are looking for people to contribute their own short experiences to the issues or patterns.

Dorothy Graham has been in software testing for over 40 years, and is co-author of 4 books: Software Inspection, Software Test Automation, Foundations of Software Testing and Experiences of Test Automation, and currently helps develop TestAutomationPatterns.org.

Dot has been on the boards of conferences and publications in software testing, including programme chair for EuroStar (twice). She was a founder member of the ISEB Software Testing Board and helped develop the first ISTQB Foundation Syllabus. She has attended Star conferences since the first one in 1992. She was awarded the European Excellence Award in Software Testing in 1999 and the first ISTQB Excellence Award in 2012.

Website

Twitter: @DorothyGraham

By: John Hunter on Mar 10, 2017

Categories: Software Testing, Testing Smarter with..., Interview

This interview with James Bach is the first our series of “Testing Smarter with…” interviews. Our goal with these interviews is to highlight insights and experiences as told by many of the software testing field’s leading thinkers.

James Bach, one of the most well-known and controversial leaders in the software testing community, challenges himself and others to continually develop their software testing approaches. James believes that excellent testing is a craft that requires many skills and ongoing practice and focus to develop and maintain those skills. The skills of testing include general systems analysis and critical thinking, but also social skills. In some sense, any child can test. But children and other amateurs cannot test systematically, nor can they provide professional self-assessment and reporting on the testing they do.

photo of James Bach
James Bach, Founder and CEO of Satisfice Inc

Personal Background

Hexawise: What one or two software testing-related experiences have you found to be most personally satisfying in your career?

James Bach: In 2002, Microsoft complained to a federal judge that I hadn’t given it a power cord. Yes, an ordinary power cord of the kind you can pull out of the back of any standard desktop computer. Yes, to a federal judge. No, I am not making this up. Yes, I also thought it was bizarre-- bizarre but kind of satisfying.

This happened during the Microsoft Remedies Trial, wherein nine American states were suing Microsoft and the government because they wanted a tougher punishment for Microsoft after it lost its big antitrust case. The states hired me as an expert witness to find out if Microsoft was telling the truth when it claimed it was “technically infeasible” to remove IE and the Windows Media Player. I gathered a team and went to work. I soon discovered that it was possible to remove these things-- using only public information and Microsoft’s own helpful tech support people to set up my testing to prove it. (The tech support people did not realize the purpose of my questions, and cheerily gave me all that I needed to know.)

When I revealed my results, Microsoft demanded that I turn over all the materials necessary to reproduce my results. So I gave them one of my test systems. At the last moment, I pulled out the power cord, thinking that I was causing some low-level techie 30 seconds of annoyance. But the next day Microsoft was in court acting like I had withheld the Golden Power Cord of Truth.

It was satisfying because Microsoft never even attempted to proved me wrong on the facts. They used lawyer tricks to stop my truth bombs, instead. Way to go, Bill Gates.

Another satisfying moment was watching my son find a catastrophic bug in a life-critical piece of medical equipment. He didn’t ask for a spec or a test case specification. He used video-gamer techniques to confuse the system until it overrode its own safety features and melted itself inside the simulated patient. Yeah. Melted. That was a $3000 piece of equipment he ruined. I’ve rarely been prouder of myself for having such foresight as to create a son like him: he is such a good test tool.

people who don’t “embrace exploratory testing” are, to me, not even testers. They are fact checkers, maybe. I think that’s not good enough.

Hexawise: Failures can often lead to interesting lessons learned. Do you have any noteworthy failure stories that you’d be willing to share?

James: How about the time I tried to set up my corporate server. I got it all working. Then I moved it from the conference room to the server room. I couldn’t get it to come online after that. For 12 hours I worked on it, all through the night. At long last I relented to my brother’s suggestion-- that we move it back to the conference room. I had refused to do that because it didn’t make any sense. The conference room simply connected to the server room. How would adding an extraneous variable like that change anything. But, zoom, we were back online.

After a moment of “wha??” the solution flashed into my mind: we must have two feeds to the Internet instead of one. It turned out that the conference room was patched into the open net, but the other port I had been using in the server room was routed through a firewall which in turn connected to the net. Nobody told me this during the buildout of my office space. It was a completely missing possibility in my mind.

So what did I learn? I learned about the importance of de-focusing, which includes trying apparently silly things to solve problems. I’m more open to that now.

Here is another interesting failure. I recently wrote a report involving the calculation of percentages. A non-technical person (a lawyer I worked with) checked my math and found it to be wrong. In fact, every one of her calculations was wrong. But in the process of refuting her claims, I discovered a different error in one of my own numbers. So, isn’t that interesting? Even if a critique of your work is incorrect, it could still be a useful stimulant to help you find your own problems.

Hexawise: What kinds of activities do you enjoy when you’re not at work?

James: I run a business, so I feel like I’m always at work. But I guess I do take little bits of time off each day. What do I do? I daydream. I read science news. I solve math and logic puzzles. I try to walk each day. And I watch videos with my wife. We binge on English television series, mostly.

Hexawise: Describe a testing experience you are especially proud of. What discovery did you make while testing and how did you share this information so improvements could be made to the software?

James: Well, many of those things I now use as testing exercises for my students, so I don’t want to spoil them. But, hmm, here’s one. I was given one day to break into an invoicing system for a large pharmaceutical company. I found three ways to do it. One of the methods I used was to get one of the sales engineers to sit with me while I tested. I asked him to demo the system to me and then he hung around while I tried to break-in. The first time I broke in (using a traversal attack if you follow such things) I didn’t even know I had done it until the sales engineer said “hey you aren’t supposed to see that data.” Good thing he was there, huh? So part of testing can be charming people into helping you, and you never know what that help will bring.

Views on Software Testing

Hexawise: Some of the thought-provoking ideas you and Michael Bolton have come up with, like the important distinction between Testing vs. Checking have received a great deal of attention within the community. Other intellectual contributions to the community you have made are not as well known but are arguably equally important and insightful. One such contribution that comes to mind which really resonates with us at Hexawise is the exploratory-scripted (or formality) continuum you and your brother Jon described.

image of software testing formality continuum

Do you have one or two intellectual contributions to the community that you wish were more widely known?

James: I wish that more people understood the folly, the sheer silliness, of counting test cases and calculating pass rates.

I don’t care if you have 80 test cases or 8 million of them. That number tells me nothing about you or your testing. It tells me nothing by itself, and it tells me nothing in conjuction with other information (except in rare cases not worth talking about). It’s like telling me that you have broken your day into 27 tasks, of 1353 tasks, or whatever. Just stop. Instead of fake science smoke rings, tell me what you actually did. Here’s a simple suggestion: instead of giving me a number, give me a list: a list of test ideas, test cases, test activities, bugs, features, people… I can do something with a list. But if you give me a number I just have to say “show me the things you are counting.”

Hexawise: Can you describe a view or opinion about software testing that you have changed your mind about in the last few years? What caused you to change your mind?

James: I used to think it was useful to talk about exploratory testing. But now I think it’s more helpful to say that all testing is exploratory. To say “exploratory testing” is the same as saying “testing.” Instead, I speak of how testing can be more or less formal, but it is always informal to some degree or else it ceases to be testing.

Also, in the last few years I have concluded that term “test automation” is toxic and should be avoided. It deposits a little poison in the mind whenever it is uttered. An angel loses its wings every time someone calls himself an “automated tester.” I changed my mind on both things as the result of ongoing attempts to teach students and hearing their questions and seeing where they get confused. That, and deep conversations with Michael Bolton.

Hexawise: How do/would you test very complex systems such as genetic algorithm systems and evolutionary systems? How do you test systems when we don't understand how they work? It seems kind of like medical differential diagnosis: poke, observe, learn, hypothesize, poke again. Or is there a better way?

James: I test them using social science methods. That, after all, is how scientists attempt to test their theories about social life. That means an emphasis on qualitative analysis, but bringing in statistical methods whenever applicable.

I agree that the medical world is a good example of where statistical methods and heuristic approaches are also needed. In testing complex things, some of what you need to do includes:

  • You must use time to your advantage-- observing systems over time the way primatologists observe chimps in the wild.
  • You must use Grounded Theory, beginning with immersion and observation, until patterns begin to reveal themselves.
  • You must focus on testability. To create an environment where you can control and observe more of what is there.
  • You must pay attention to clues. Many, many clues. Stop looking for simplistic “test cases” that will “prove” that the software works.
  • You must become expert at data wrangling, since these systems usually involve huge amounts of data.
  • Let other people help you.
  • Forge partnerships with users.

If it’s a training gig then my objective is to show them what testing can be, show them a path to get there, and encourage them to walk that path. A lot of that is about removing the obstacles to moving along.

Hexawise: It is clear from your writings and frequent presentations, that you feel passionately that the software testing community would greatly benefit if far more testers embraced Exploratory Testing. It’s a deeply held conviction. What particular testing practice(s) do you most wish the software testing community would embrace?

James: Testing is exploratory. So, people who don’t “embrace exploratory testing” are, to me, not even testers. They are fact checkers, maybe. I think that’s not good enough.

I wish more testers were mathematically inclined. You must see this, too, at Hexawise-- the widespread math-phobia in our field. I want to talk about Karnaugh maps and the value of de Bruijn sequences. But I have to keep that stuff out of my classes or I will freak most people out. It’s not that I am a mathematics expert. I’m just an enthusiast who wants to be held to a higher standard. But even my dalliances in Bayesian belief nets sound like high elf incantations to most testers. Mathematical disabilities, in general, make our craft prey to quackery and fraud of all kinds.

At the same time, I want to be inclusive. Mathophobes have a lot of offer and I don’t want them to think I don’t welcome them. But must they necessarily be the majority of testers? I guess I’m saying I want a cure for mathophobia, please.

Hexawise: What do you wish more developers, business analysts, and project managers understood about software testing?

James: I wish they understood that it benefits from specialization. When software people get heart disease they don’t limit themselves to a GP, do they? If they need surgery they don’t insist on being operated on by a rotating team of generic medical people who took a three day class in “Agile medicine” do they?

I get to be a specialist tester mainly because I do it on my own time. My clients are paying me, usually, for training, not for doing testing. So I usually test in my “free time” to sharpen my skills. I recently did have a wonderful and lucrative testing-related gig (because this particular project knew that it needed the best tester and analyst it could possibly get and became convinced I was the Chosen One), but those gigs are few and far between for someone like me.

Hexawise: When individual companies hire you for consulting engagements, how would you describe what it is that you usually seek to provide to them?

James: If it’s a training gig then my objective is to show them what testing can be, show them a path to get there, and encourage them to walk that path. A lot of that is about removing the obstacles to moving along. Chief among those obstacles is lack of confidence. So I do a lot of pep talking. Another obstacle is the very primitive, mechanical way that people think about testing. I have to replace that with systems thinking.

If it’s a testing gig then my objective is usually to provide deep, exemplary testing, that is transparent to my client. I want them to feel that they see their product in a beautiful focus. The danger I am always in when I test is that I will get too deep (and therefore be too slow and expensive). But for me, deep testing is the most fun, so it’s a constant struggle to hold myself back from using my most penetrating methods and tools.

Industry Observations / Industry Trends

Hexawise: As Artificial Intelligence increases in capability (for example the strides made by Watson) do you foresee an increase in the capabilities of computer checking? I am thinking, not of an elimination of the difference between human lead testing and what can be done without people but to what extent you see the possibility for AI to do a progressively much better job of checking.

James: I foresee a collapse of critical thinking about these complex systems, followed by some sort of disaster, followed by a new realization of the risks of surrendering human judgment to a machine. I foresee that this will be an ongoing cycle. This collapse of critical thinking will lead to more shallow testing and perfunctory checking, presented as if it were deep. For an example of what I mean, see this old computer commercial:

Pay attention starting at 2:05. Oh look, the computer is assuring us that it has no errors. Everyone relax! It’s “electronic brain” can be trusted!

Hexawise: Do you have any predictions about how large an impact Artificial Intelligence and Machine Learning will have on software testing in the next 5-10 years?

James: I don’t think it will have any impact on testing as such, except inasmuch as many people (not skeptical testers, but people who might otherwise hire testers) will trust black boxes when they should be challenging them.

I suppose as machine learning become more available to the masses, someone might try to train one to recognize bad output of some kind. That’s a sort of test tool. But it would only apply to well-established kinds of badness.

If you think about it, anti-spam systems are a sort of test system. Machine learning is used in spam filters. So, I guess testing is already using machine learning in that sense. But I don’t see the average tester applying machine learning methods to testing. I don’t see a developer doing that, either. It’s too involved and complicated; too narrow in application.

I hear that people at Google are going to “put coders out of business” with a system that writes code based on people just talking. You know what that’s called? A compiler. They are inventing a high level compiler. Now the people who talk will be called developers and will have to learn to talk properly, because it will emerge that normal people can’t say what they actually want.

Hexawise: Have you seen a particularly effective process where the software testing team was integrated into the feedback from a deployed software application (getting feedback from users on problems, exploring issues the software noted as possible bugs...)? What was so effective about that instance?

James: Not really. What I see is developers ignoring feedback. It’s too overwhelming. I suspect there are people who are really good at doing that. But I haven’t run across any.

One of the things that has happened with DevOps is a de-emphasis on testing and more of an emphasis on overall risk management. That’s a valid strategy, of course, but it has interesting blind spots. Whenever I hear a developer speak about wanting feedback from users I immediately think about how abusive and incompetent most users are about reporting problems. No, my developer friends, you really don’t want to read all those Internet comments on your software. You will be demoralized. But testers? We love reading that stuff. It’s our wheelhouse. We get clues and then we can reproduce the problems and make them sensible for the devs.

My brother, at eBay, with his testing mentality, loves going over the user feedback and bringing it to the teams there. But he will tell you it’s a constant struggle to get the attention of the dev teams.

attend a conference. Don’t bother to go to the talks, though. Most of the talks are full of fluff. Instead, find people and talk to them. Compare notes, make friends. Go to the testing lab.

Hexawise: Often one of the major roadblocks to software testers is their own management. Do you think this is a fair statement? Do you have suggestions for how testers can attempt to improve the situation. My background is strongly influenced by W. Edwards Deming so I have a tendency to look at the organization as a system and see room to improve the management system. It seems to me often the biggest gains are not possible if we keep departments separate (software development, software testing, marketing, customer service...). We can make improvements in software testing even if it is largely seen as separate from the organization but in doing so we miss much greater potential improvement.

James: The collapse of the test management industry is a terrible problem. It’s getting harder to find any kind of test manager out there. Do they even exist in Silicon Valley any more or have they all been hunted down by parasitic wasps who lay “scrum master” eggs in their living carcasses?

People who seem to know little about management or testing tell me that test managers are not needed. Okay, that means a whole lot of things that test managers do will not get done. This includes: providing a protected place for testers to work, free of harassment; negotiating for testability; negotiating for resources; assuring that schedules are reasonable; assuring that testing gets the respect it requires in order to attract and keep talented people; assuring that deadlines are met; explaining testing to management; assuring that testers are properly trained. When those things aren’t happening, testers tend to become more zombie-like and reactive (I’m not speaking of those fast zombies); or they become cheerleaders for the devs, instead of critical thinkers.

Staying Current / Learning

Hexawise: How do you stay current on improvements in software testing practices?

James: I’m not convinced there are improvements in testing practices in the absence of improvements in the thinking and social systems that drive practices-- and those things don’t improve much, as I’m sure you’ve noticed. Seems to me that the current nonsense in our craft is very similar to the old nonsense. Maybe some of the buzzwords have changed, but not much else.

The landscape of testing has definitely changed. Agile and Lean have aggressively colonized a lot of the testing space. Since most testers are young people (and test management has been eviscerated) they are easy pickings for the Agile Universalists (the people who think that we don’t need testers because we can just all test whenever we feel like it).

What this means is that testing remains rather primitive wherever I go (with a few interesting exceptions, driven inevitably by a single enlightened Elrond-like or Galadriel-like manager, who always seems to disappear off into the Grey Havens within a couple of years of me meeting him or her).

How I become aware of new and interesting ideas is through my community. For instance, a student told me about Karnaugh maps the other day and now I am trying to find a use for them.

Hexawise: How would you suggest testers stay current?

James: I don’t think currency is a thing in testing, except with respect to learning about certain emerging technologies and buzzwords.

The bigger thing in testing is to push us forward, which not enough testers are trying to do. Don’t worry about currency, worry about whether you truly understand testing, and keep working on that study.

Read widely about science. Get ideas from that. And play with the ideas. For instance, I read on Hacker News about 350,000 free images being released by the Metropolitan Museum of Art. I decided to experiment with turning that into a practical resource for test data. This led to playing with data wrangling and image analysis tools.

Also, attend a conference. Don’t bother to go to the talks, though. Most of the talks are full of fluff. Instead, find people and talk to them. Compare notes, make friends. Go to the testing lab. Or host a little conference. Invite testers to a small gathering where you can share experience reports.

Hexawise: What software testing-related books would you recommend should be on a tester’s bookshelf?

James:

Profile

James Bach has authored two books and consulted and presented on software testing worldwide. It is difficult to put into words how unique and insightful James is. In order to get a feel, we suggest listening to his presentations yourself and reading his excellent blog.

Some Career Highlights

Links

By: John Hunter and Justin Hunter on Mar 2, 2017

Categories: Interesting People , Software Testing, Testing Strategies, Interview, Testing Smarter with...

I read a great post by Aleksis Tulonen Brainstorming Test Ideas With Developers and wanted to share it with you.

The ideas in the post provide a very efficient way to:

  1. increase the robustness and thoroughness of testing,
  2. prioritize among different testing ideas (both from a relative importance standpoint and from a relative timing perspective)
  3. surface new testing ideas that would not otherwise be considered
  4. provide some useful risk mitigation protection against the reality that "all models are wrong; some models are useful" problem

I'd suggest also scheduling a debrief with the same testers and developers a few weeks or months after the code was deployed.

At that time take a look at photos of the pre-testing Post It notes and a list of defects that slipped past testing and ask

  1. "what extra tests would we need to have added to have found these defects?"
  2. "what specific inputs did we forget to include?",
  3. would techniques such as pairwise testing have been beneficial?
  4. what areas did we spent a lot of effort on that did not result in learning much?
  5. what lessons learned should we incorporate for next time?

The idea of delibrately examining your software development and testing practices will be familar to those using agile retrospectives. The power of continually improving the development practices used withing the organization is hard to appreciate but it is immense. The gains compound over time so the initial benefits are only a glimpse of what can be achieve by continuing to iterate and improve.

Related: Pairwise and Combinatorial Software Testing in Agile Projects - Applying Toyota Kata to Agile Retrospectives - Create a Risk-based Testing Plan With Extra Coverage on Higher Priority Areas

By: John Hunter and Justin Hunter on Dec 15, 2016

Categories: Agile, Software Testing, Testing Strategies

Performance testing examines how the software performs (normally "how fast") in various situations.

Performance testing does not just result in one value. You normally performance test various aspects of the software in differing conditions to learn about the overall performance characteristics. It can well be that certain changes will improve the performance results for some conditions (say a powerful laptop with a fiber connection) and greatly degrade the performance for other use cases. And often the software can be coded to attempt to provide different solutions under different conditions.

All this makes performance testing complex. But trying to over-simplify performance testing removes much of its value.

Another form of performance testing is done on sub components of a system to determine what solutions may be best. These often are server based issues. These likely don't depend on individual user conditions but can be impacted by other things. Such as under normal usage option 1 provides great performance but under larger load option 1 slows down a great deal and option 2 is better.

Focusing on these tests of sub components run the risk of sub-optimization, where optimizing individual sub-components result in a less than optimal overall performance. Performance testing sub-components is important but it is most important is testing the performance of the overall system. Performance testing should always place a priority on overall system performance and not fall into the trap of creating a system with components that perform well individually but when combined do not work well together.

Load testing, stress testing and configuration testing are all part of performance testing.

Continue reading about performance testing in the Hexawise software testing glossary.

By: John Hunter on Sep 27, 2016

Categories: User Experience, Software Testing Testing Strategies, Software Testing

Software testing concepts help us compartmentalize the complexity that we face in testing software. Breaking the testing domain into various areas (such as usability testing, performance testing, functional testing, etc.) helps us organize and focus our efforts.

But those concepts are constructs that often have fuzzy boundries. What matters isn't where we should place certain software testing efforts. What matters is helping create software that users find worthwhile and hopeful enjoyable.

One of the frustration I have faced in using internet based software in the last few years is that it often seems to be tested without considering that some users will not have fiber connections (and might have high latency connections). I am not certain latency (combined maybe with lower bandwidth) is the issue but I have often found websites either actually physically unusable or mentally unusable (it is way too frustrating to use).

It might be the user experience I face (on the poorly performing sites) is as bad for all users, but my guess is it is a decent user experience on the fiber connections that the managers have when they decide this is an OK solution. It is a usbility issue but it is also a performance issue in my opinion.

It is certainly possible to test performance results on powerful laptops with great internet connections and get good performance results for web applications that will provide bad performance results on smart phones via wifi or less than ideal cell connections. This failure to understand the real user conditions is a significant problem and an area of testing that should be improved.

I consider this an interaction between performance testing and user-experience testing (I use "user-experience" to distinguish it from "usability testing", since I can test aspects of the user experience without users testing the software). The page may load in under 1 second on a laptop with a fiber connection but that isn't the only measure of performance. What about your users that are connecting via a wifi connection with high latency? What if the performance in that case is that it takes 8 seconds to load and your various interactive features either barely work or won't work at all given the high latency.

In some cases ignoring the performance for some users may be OK. But if you care about a system that delivers fast load times to users you need to consider the performance not just for a subset of users but consider how it performs for users overall. The extent you will prioritize various use cases will depend on your specific situation.

I have a large bias for keeping the basic experience very good for all users. If I add fancy features that are useful I do not like to accept meaningful degradation to any user's experience - graceful degradation is very important to me. That is less important to many of sites that I use, unfortunately. What priority you place on it is a decision that impacts your software development and software testing process.

Hexawise attempts to add features that are useful while at the same time paying close attention to making sure we don't make things worse for users that don't care about the new feature. Making sure the interface remains clear and easy to use is very important to us. It is also a challenge when you have powerful and fairly complex software to keep the usability high. It is very easy to slip and degrade the users experience. Sean Johnson does a great job making sure we avoid doing that.

Maintaining the responsiveness of Hexawise is a huge effort on our part given the heavy computation required in generating tests in large test case scenarios.

You also have to realize where you cannot be all things to all people. Using Hexawise on a smart phone is just not going to be a great experience. Hexawise is just not suited to that use case at all and therefore we wouldn't test such a use case.

For important performance characteristics it may well be that you should create a separate Hexawise test plan to test the performance under server different conditions (relating to latency, bandwidth and perhaps phone operating system). It could be done within a test plan it just seems to me more likely separate test plans would be more effective most of the time. It may well be that you have the primary test plan to cover many functional aspects and have a much smaller test plan just to check that several things work fine in a high latency and smart phone use case).

Within that plan you may well want to test out various parameter values for certain parameters operating system iOS Android 7 Android 6 Android 5

latency ...

Of course, what should be tested depends on the software being tested. If none of the items above matter in your case they shouldn't be used. If you are concerned about a large user base you may well be concerned about performance on various Android versions since the upgrade cycle to new versions is so slow (while most iOS users are on the latest version fairly quickly).

If latency has a big impact on performance then including a parameter on latency would be worthwhile and testing various parameter values for it could be sensible (maybe high, medium and low). And the same with testing various levels of bandwidth (again, depending on your situation).

My view is always very user focused so the way I naturally think is relating pretty much everything I do to how it impacts the end user's experience.

Related: 10 Questions to Ask When Designing Software Tests - Don’t Do What Your Users Say - Software Testers Are Test Pilots

By: John Hunter on Jul 26, 2016

Categories: User Experience, Testing Strategies, Software Testing

Usability testing is the practice of having actual users try the software. Outcomes include the data of the tasks given to the user to complete (successful completion, time to complete, etc.), comments the users make and expert evaluation of their use of the software (noticing for example that none of the users follow the intended path to complete the task, or that many users looked for a different way to complete a task but failing to find it eventually found a way to succeed).

Usability testing involves systemic evaluation of real people using the software. This can be done in a testing lab where an expert can watch the user but this is expensive. Remote monitoring (watching the screen of the user; communication via voice by the user and expert; and viewing a webcam showing the user) is also commonly used.

In these setting the user will be given specific tasks to complete and the testing expert will watch what the user does. The expert will also ask the user questions about what they found difficult and confusing (in addition to what they liked) about the software.

The concept of usability testing is to have feedback from real users. In the event you can't test with the real users of a system it is important to consider if you are fairly accuratately representing that population with your usability testers. If the users of the system of fairly unsophisticated users if you use usability testers that are very computer savy they may well not provide good feedback (as their use of the software may be very different from the actually users).

"Usability testing" does not encompass experts evaluating the software based on known usability best practices and common problems. This form of expert knowledge of wise usability practices is important but it is not considered part of "usability testing."

Find more exploration of software testing terms in the Hexawise software testing glossary.

Related: Usability Testing Demystified - Why You Only Need to Test with 5 Users (this is not a complete answer, it does provide insite into the value of quick testing to run during the development of the software) - Streamlining Usability Testing by Avoiding the Lab - Quick and Dirty Remote User Testing - 4 ways to combat usability testing avoidance

By: John Hunter on Jul 4, 2016

Categories: User Experience, User Interface, Testing Strategies, Software Testing

At Hexawise we aim to improve the way software is tested. Achieving that aim requires not only providing our clients with a wonderful software tool (which our customers say we’re succeeding at) but also a commitment from the users of our tool to adopt new ways of thinking about software testing.

We have written previously about our focus on the importance of the values Bill Hunter (our founder's father) to Hexawise. That has led us to constantly focus on how maximize the benefits our customers gain using Hexawise. This focus has led us to realize that our customers that take advantage of the high-touch training services and ongoing expert test design support on demand that we offer often realize unusually large benefits and roll out usage of Hexawise more quickly and broadly than our customers who acquire licenses to Hexawise and try to “get the tool and make it available to the team.”

We are now looking for someone to take on the challenge of helping our clients succeed. The principles behind our decision to put so much focus on helping our customers succeed are obvious to those that understand the thinking of Bill Hunter, W. Edwards Deming, Russel Ackoff etc. but they may seem a bit odd to others. The focus of this senior-level position really is to help our customers improve their software testing results. It isn't just a happy sounding title that has no bearing on what the job actually entails.

The person holding this position will report to the CEO and work with other executives at Hexawise who all share a commitment to delighting our customers and improving the practice of software testing.

Hexawise is an innovative SaaS firm focused on helping large companies use smarter approaches to test their enterprise software systems. Teams using Hexawise get to market faster with higher quality products. We are the world’s leading firm in our niche market and have a growing client base of highly satisfied customers. Since we launched in 2009, we have grown both revenues and profits every year. Hexawise is changing the way that large companies test software. More than 100 Fortune 500 companies and hundreds of other smaller firms use our industry leading software.

Join our journey to transform how companies test their software systems.

Hexawise office

Description: VP of Customer Success

In the Weeks Prior to a Sale Closing

  • Partner with sales representatives to conduct virtual technical presentations and demonstrations of our Hexawise test design solution.

  • Clearly explain the benefits and limitations of combinatorial test design to potential customers using language and concepts relevant to their context by drawing upon your own “been there, done that” experiences of having successfully introduced combinatorial test design methods in multiple similar situations.

  • Identify and assess business and technical requirements, and position Hexawise solutions accordingly.

Immediately Upon a New Sale Closing

  • Assess a new client’s existing testing-related processes, tools, and methods (as well as their organizational structure) in order to provide the client with customized, actionable recommendations about how they can best incorporate Hexawise.

  • Collaborate with client stakeholders to proactively identify potential barriers to successful adoption and put plans in place to mitigate / overcome such barriers.

  • Provide remote, instructor-led training sessions via webinars.

  • Provide multi-day onsite instructor-led training sessions that: cover basic software test design concepts (such as Equivalence Class Partitioning, the definition of Pairwise-Testing coverage, etc.) as well as how to use the specific features of Hexawise.

  • Include industry-specific and customer-specific customized training modules and hands-on test design exercises to help make the sessions relevant to the testers and BA’s who attend the training sessions.

  • Collaborate with new users and help them iterate, improve, and finalize their first few sets of Hexawise-generated software tests.

  • Set rollout and adoption success criteria with clients and put plans in place to help them achieve their goals.

Months After a New Sale Closing

  • Continue to engage with customers onsite and virtually to understand their needs, answer their test design questions, and help them achieve large benefits from test optimization.

  • Monitor usage statistics of Hexawise clients and proactively reach out to clients, as appropriate, to provide proactive assistance at the first sign that they might be facing any potential adoption/rollout challenges.

  • Collaborate with stakeholders and end users at our clients to identify opportunities to improve the features and capabilities of Hexawise and then collaborate with our development team to share that feedback and implement improvements.

Required Skills and Experience

We are looking for a highly-experienced combinatorial test design expert with outstanding analytical and communication skills to provide these high touch on-boarding services and partner with our sales team with prospective clients.

Education and Experience

  • Bachelor’s or technical university degree.

  • Deep experience successfully introducing combinatorial test design methods on multiple different kinds of projects to several different groups of testers.

  • Set rollout and adoption success criteria with multiple teams and put plans in place to achieve them.

  • Minimum 5 years in software testing, preferably at a IT consulting firm or large financial services firm.

Knowledge and Skills

  • Ability to present and demonstrate capabilities of the Hexawise tool, and the additional services we provides to our clients beyond our tool.
  • Exhibit excellent communication and presentation skills, including questioning techniques.
  • Demonstrate passion regarding consulting with customers.
  • Understand how IT and enterprise software is used to address the business and technical needs of customers.
  • Demonstrate hands-on level skills with relevant and/or related software technology domains.
  • Communicate the value of products and solutions in terms of financial return and impact on customer business goals.
  • Possess a solid level of industry acumen; keeping current with software testing trends and able to converse with customers at a detailed level on pertinent issues and challenges.
  • Represents Hexawise knowledgeably, based on a solid understanding of Hexawise’s business direction, portfolio and capabilities
  • Understand the competitive landscape for Hexawise and position Hexawise effectively.
  • A cover letter that describes who you are, what you've done, and why you want to join Hexawise.
  • Ability to work and learn independently and as part of a team
  • Desire to work in a fast-paced, challenging start-up environment

Why join Hexawise?

salary + bonus; medical and dental, 401(k) plans; free parking and very slick Chapel Hill office! Opportunity to experience work with a fast-growing, innovative technology company that is changing the way software is tested.

Key Benefits:

Salary: Negotiable, but minimum of $100,000 + Commissions based upon client license renewals Benefits: Health, dental included, 401k plan Travel: Average of no more than 2-3 days onsite per week Location: Chapel Hill, NC*

*Working from our offices would be highly preferable. We might consider remote working arrangements for an exceptional candidate based in the US.

Apply for the VP of Customer Success position at Hexawise.

By: John Hunter on May 12, 2016

Categories: Hexawise, Career, Software Testing, Lean, Customer Success, Agile

It has been quite a long time since we last posted a roundup of great content on software testing from around the web.

  • Mistakes We Make in Testing by Michael Sowers - "Not being involved in development at the beginning and reviewing and providing feedback on the artifacts, such as user stories, requirements, and so forth. Not collaborating and developing a positive relationship with the development team members..."
  • Changing the conversation about change in testing by Katrina Clokie - "I'm planning to stop talking about bugs, time and money. Instead I'm going to start framing my reasoning in terms of corporate image, increasing quality awareness among all disciplines, and improving end-user satisfaction."
  • How to Pack More Coverage Into Fewer Software Tests by Justin Hunter - "There are simply too many interactions for a tester to keep track of on their own. As a result, manually-selected test sets almost always fail to test for a rather large number of potentially important interactions."
  • Building Quality by Alan Page - "your best shot at creating a product that customers like crave is to get quantitative and qualitative feedback directly from your users... Get it in their hands, listen to them, and make it better."
  • Dr. StrangeCareer or: How I Learned to Stop Worrying and Love the Software Testing Industry by Keith Klain - "Testing is hard. Doing it right is very hard. An ambiguous, unchartered journey into a sea of bias and experimentation, but as the line from the movie goes, 'the hard is what makes it great'."
  • Exploratory Testing 3.0 by James Bach and Michael Bolton - "Because we now define all testing as exploratory. Our definition of testing is now this: 'Testing is the process of evaluating a product by learning about it through exploration and experimentation, which includes: questioning, study, modeling, observation and inference, output checking, etc.'"
  • Coverage Is Not Strongly Correlated With Test Suite Effectiveness by Laura Inozemtseva and Reid Holmes - "Our results suggest that coverage, while useful for identifying under-tested parts of a program, should not be used as a quality target because it is not a good indicator of test suite effectiveness. "
  • How Software Testers Can Teach, Share and Play with Others by Justin Rohrman - Software testers "bring a varied skill set to software development organizations — math, experiment design, modeling, and critical thinking."
  • Who Killed My Battery: Analyzing Mobile Browser Energy Consumption - " dynamic Javascript requests can greatly increase the cost of rendering the page... by modifying scripts on the Wikipedia mobile site we reduced by 30% the energy needed to download and render Wikipedia pages with no change to the user experience."

By: John Hunter on Apr 27, 2016

Categories: Roundup, Software Testing

When you outsource testing responsibilities to a third party, there are several key points to consider beyond the three most frequently asked questions of:

  • What is the cost per hour of your services?
  • What evidence can you provide of your firm’s level of expertise in our industry and/or the technical skills we feel you’ll need? and
  • What is your plan for increasing testing automation? In particular, it is worth developing a thorough understanding how the vendor will identify what kinds of things should be tested, how those things should be tested, and how that information will be communicated to you.
  1. How do you ensure testing is performed thoroughly - beyond just “happy path”/ confirmatory tests?
  2. Do you use Exploratory Testing broadly? Would you recommend we use it? Exploratory Testing is a well-established testing approach.
    • Champions of exploratory testing include James Bach and Michael Bolton.
    • If your vendor hasn’t heard of exploratory testing and starts to explain the disadvantages of undisciplined “ad hoc testing” it’s a warning sign that they might have rigid tunnel vision about what good testing entails. Ad hoc testing is not what exploratory testing is but those that haven't learned about exploratory testing may get the two ideas confused.
  3. How do you communicate the diminishing marginal returns that exist when you execute sets of test scripts?
    • In Hexawise, analyze tests screen is how we help show this concept. That screen shows both pecentage of parameter value pairs tested and the coverage matrix showing the parameter values pairs tested and untested. This view can seen for any specific number of tests along the test plan (using the adjustable bar at the top of the image). This image shows 80% of the paired values have been tested after 10 test cases. Using the slider this view lets you see exactly which parameter value pairs are untested at each point. In this example after 15 tests there is 96% pairwise coverage and 19 tests covers all pairwise combinations.
    • Be conscious that your vendor has a strong conflict of interest here. 80% 2-way coverage might be sufficient for your needs on a given project but testing to that level of thoroughness (instead of the complete set of tests) could represent 50% of their billable hours in which case they may go to 100% 2-way coverage (to increase billable hours). Of course, 100% 2-way coverage may be called for, these decisions have to be made with an understanding of the software being tested.
    • Implications: ideally, you would like to have a testing vendor that is happy to openly share and discuss the tradeoffs that exist and there strategy to get you the best results based on your situation.
  4. Do your testers use a structured test design approach such as pairwise test design?
    • If so, how many of your testers regularly use pairwise test design or similar methods to design tests?
    • During the process of selling you their testing services, vendors have a strong incentive to highlight their efficiency-enhancing capabilities. As soon as the ink dries on the contract though they have an incentive to keep billable hours high.
  5. Why (and in what types of testing) do they use it?
    • Pairwise and combinatorial test design approaches can be widely used in virtually all kinds of software testing.
    • If your vendor indicates that they use these approaches only in narrow areas (e.g., most likely functional testing or configuration testing), that is a large red flag that they don’t understand these test design approaches very well at all.
    • If your vendor does not understand how to apply these test design methods broadly (including in systems integration testing, performance testing, etc.) then you can safely assume that there will be considerable wasteful redundancy in any areas where these test design approaches are not used.
  6. If we gave you 50 test scripts we already have, could you generate a set of pairwise tests for us that could test the same system more thoroughly in significantly fewer tests?
    • A knowledgeable vendor should be able to analyze your tests and provide a draft set for your review within 24 hours.
    • If the vendor takes multiple days and they need to search far and wide for an available expert to create a set of optimized set of tests for you, that indicates that the actual number of testers capable of performing this kind of test design is probably pretty low.
    • Another practical way to double-check vendor claims is to search sites like LinkedIn for keywords such as pairwise testing, orthogonal array testing, and/or Hexawise.

Making sure your software testing process is staying current with the best ideas in software testing is an important factor in creating great software solutions that your customers love. Often companies understand the need to stay current on software coding practices that are successful but fewer organizations pay attention to good practices in software testing. This often means there is a good deal to be gained by spending some time to examine and improve your software testing practices.

By: John Hunter and Justin Hunter on Mar 29, 2016

Categories: Testing Strategies, Software Testing

When thinking about how to test software these questions can help you think of ideas you might otherwise miss. These tips are useful for anyone testing software; we do integrate tips that are specfically relevant to those using the Hexawise software testing application.

  • What additional things could I add that probably won't matter (but might?)
  • What are the two parameters in the plan with the most values? And should I shrink the number of parameter values for either case?). If you have a parameter or two with many more parameter values than other parameters have that can greatly increase the number of test cases that must be run to explore every pairwise (or higher) combination. If those are critical to test, then they should be, but often a high number of parameter values is an indication that they really could be reduced. Using value expantions may well be a wise choice.
  • Do the tests cover all the high priority scenarios that stakeholders want covered? Once you generate the tests think about high priority scenarios and if they are missing add them as required test cases. It may well be worth adding tests for the most common scenarios. Often this can be done without requiring extra tests when using Hexawise. Hexawise will just consider those and then create test cases to match your criteria which often won't require an extra test. Sometimes it will add a test or a few tests to the total, in which case you can decide the benefit is worth the added cost of those tests.
  • If you're testing a process, what are the if/then points where the process diverges? Make sure the alternative processes are being tested.
  • If the plan is heavily constrained, would it make sense to split it into multiple separate plans?
  • Might it make a difference to how the system operates if you were to include additional variation ideas related to: who, what, when, where, why, how and how many? Those questions are often helpful in identifying parameters that should be tested (and occassionally in identifying parameter values to test).
  • Have you accidentally included values in your plan that everyone is confident won't impact the system? You want to test everything that is important but if you test things that are not going to impact whether the software works or not it adds cost with no benefit.
  • Have you accidentally included "script killing" negative tests in your plan? See details on this point in our training file, different types of negative tests need to be handled differently when using Hexawise
  • Have you clearly thought about the implications of the distinction between impossible to test for combinations vs combinations that will lead the system to display error messages? Impossible to test combinations can be avoided using the invalid pair feature. But situations where users can enter values that should results in error messages should be tested to validate those error messages appear.
  • Have you considered the cost/benefit trade-off of executing, say, 50% of the Hexawise generated test set vs 75% vs 100%? It is best to think about what is the most sensible business decision instead of just automatically going for some specfic percentage. Depending on the softare being tested and the impact on your business and customers different levels of testing will make sense.

Too often software testing fails to emphasis the importance of experienced software testers using their brains, insight, experience and knowledge to make judgements about what is the best course of action. Hexawise is a wonderful tool to help software testers but the tool needs a smart and thoughtful person to wield it and produce the results we all want to see.

Related: Software Testers Are Test Pilots - Create a Risk-based Testing Plan With Extra Coverage on Higher Priority Areas - How Much Detail to Include in Test Cases?

By: John Hunter and Justin Hunter on Mar 14, 2016

Categories: Testing Strategies, Software Testing, Hexawise test case generating tool, Combinatorial Software Testing, Hexawise tips

Michael Bolton is one of the software testing industry's deep thinkers. He has an impressive ability to logically analyze testing problems and clearly explain complex issues.

I like how Michael summarizes what people often really mean when they say "it works"*

It works really means...

There's a lot of truth in those words, isn't there? I've shared these words from Michael in test design trainings I've done recently and found that they immediately resonate with quite a few testers. It seems that anyone who has been in testing for more than a while has seen teams of testers test a feature or function a bit, declare that "it works," only to discover later that the feature/function works in some situations but does not work in other situations.

What's a tester to do? We recommend testers use two deliberate strategies: use a rich oracle and cover critical interactions.

First, use a "rich oracle" to enage your brain more actively and train your eye to better recognize potential issues. Imagine the following scenario. 3 people are in a room. The first person, a guy plucked from the street outside at random, is given a set of 10 written test scripts to execute and told to follow the test scripts, step-by-step in return for a six pack of beer. Being a fan of beer, and endowed with the dual-abilities of being able to both read instructions and follow instructions, he performs what is asked of him dutifully.

In the room are two testers who are allowed to observe the tests being executed but who are not allowed to communicate.

  • The first tester has a list of ten numbers, each with three boxes for checkmarks. He operates in a world of black and white where if the documented "Expected Result" is consistent with what they observe, they write a green check mark. If the "Expected Result" is inconsistent, they write a green "X."
  • The second tester operates differently. She goes beyond. She goes deeper. She notices subtle things along the way that look unexpected, or not quite right. She makes notes of those things. In doing so, she thinks of new test ideas that have not been executed yet. She documents those test ideas to explore further later, provided there is time.

I think you see where I'm going with this. My point is that the more curious approach adopted by the second tester is a far more valuable one to people who care about software quality. Why is this? Here too, Michael Bolton has words of wisdom to share that resonate well:

If you insist you need written requirements to find bugs

Second, testers should adopt test case design approaches in order to avoid the "under some conditions... once" risk. One of the most important benefits of using our Hexawise test case design tool is that, even with very basic pairwise test sets, every feature or function you test will automatically be tested multiple times. And under as many different conditions as possible in the time you have available.

After close to 10 years of introducing new groups of software testers to these types of test design approaches, people have a hard time believing how big efficiency and thoroughness improvements in this area are. That's why we always strongly encourage teams using our optimized test case selection approach to do apples-to-apples comparisons of coverage and defect-finding effectiveness. We work with teams to compare the coverage gaps of their existing "business as usual" test sets to how thorough they are when Hexawise is used to generate an optimized set of tests. Images of one recent coverage analysis is shown below. Data on defect-finding effectiveness and defect finding efficiency improvements resulting from optimized test case selection can also be found here.

Testing Coverage Analysis Hexawise Pairwise Combinatorial Testing OATS

 

*With thanks to Jon Bach for sharing this on his blog.

By: Justin Hunter on Sep 25, 2014

Categories: Pairwise Software Testing, Pairwise Testing, Combinatorial Testing, Software Testing

Begin with the “Goldilocks Rule” in mind to identify how much detail is appropriate.

unnamed

If your tests cover a large scope, as in a set of end-to-end tests of a process, you focus first on entering only the most important elements of that process into Hexawise. If your tests cover a small scope, as in tests that focus on a few items on a single screen, the amount of detail you will want to include in Hexawise is higher. Learn more about the Goldilocks Rule.

 

Imagine explaining your System Under Test to someone’s mother. Start with 5 things that may change in each test

Suggestion: do not start this process with detailed Requirements or Technical Specifications. Instead, start with your basic common sense description of some things that would change from test to test.

  • If you were explaining the application to someone’s mother, how would you explain what it does in 2 minutes?
  • What kinds of things would be important to vary from test to test?
    • Hardware and software configurations?
    • User types?
    • Different actions that a user might take?
  • Identify 5 things that change from test to test and turn those 5 things into your first Parameters.
    • How might those things change? Add one or more Values for each Parameter.
    • At this point general descriptions might be fine; (e.g., SUV’s or Economy cars vs. Toyota Corolla).
    • Remember that, where possible, you should avoid creating long lists of values.

 

Create a draft set tests, assess obvious big gaps in the tests, and start filling them.

What obvious types of scenarios are missing? Add parameters and values as necessary to fill those big, obvious holes.

 

Create tests again, assess whether you’re covering necessary business rules and requirements.

If your tests are not yet testing a business rule or Requirement that you want to test:

  • Consider adding a Parameter or Value
  • Consider adding a specific combination of values to be tested together in the requirements tab

 

Reduce scope if the test plan is too complex

  • Consider cutting the scope of the plan (e.g., create two different plans with largely similar parameters – one for regular users and one for special users – instead of one big plan which tries to “do it all.”)
  • Consider changing the way you are describing “hard coded” values. Instead, of “iPhone 4S with International roaming” (which might not be a valid option after the first part of the draft test case suggests a transaction for a phone for corporate customers from a Northern location responding to the special holiday offer…) consider using descriptive parameters and values along the lines of “from the available options of phones available at this point in the test case, select an option that meets as many of these conditions as you can…

 

Consider adding additional details into your plan from other sources.

Other sources for test ideas could be:

J Michael Hunter’s “You Are Not Done Yet”
Elisabeth Hendrikson’s Test Heuristics Cheat Sheet
Hans Buwalda’s Soap Opera Testing

By: John Hunter on Jun 17, 2014

Categories: Combinatorial Software Testing, Software Testing

The Hexawise Software Testing blog carnival focuses on sharing interesting and useful blogposts related to software testing.

 

  • The Zen of Application Test Suites by Curtis “Ovid” Poe – “This document is about testing applications — it’s not about how to write tests. Application test suites require a different, more disciplined approach than library test suites. I describe common misfeatures experienced in large application test suites and follow with recommendations on best practices.”

  • The Software Tester’s Easter Egg Hunt by Ben Austin – “The testing industry will benefit greatly if more people follow Holland’s example and prioritize critical thinking over the ability to write test scripts. That said, it is equally important to recognize that scripting tools, when used in the situations they’re intended for, can be great time-savers that can enable more thoughtful, context-driven exploratory testing.

  • “Grapefruit Juice Bugs” – A New Term for a Surprisingly Common Type of Surprising Bugs by Justin Hunter – “Like grapefruit juice’s impact on prescription drugs, software testing involves critical interactions between different parts of the system. And risks exist when these different parts interact with one another.”

  • Testing at Airbnb by Lou Kosak – “Building good habits around testing is hard, especially at an established company. One of the biggest lessons I’ve learned this year is that as long as you have a team that’s open to the idea, it’s always possible to start testing (even if you’ve got a six year old monolithic Rails app to contend with). Once you have a decent test suite in place, you can actually start refactoring your legacy code, and from there anything is possible. The trick is just to get started.”

  • Disruptive Testing – James Bach Interviewed by Rodney Urquhart – “the future I am helping to build is about systematically training up skilled testers, some of whom but not all with coding skills, so that a small number of testers can do– or coordinate to be done– all the testing that a large project might need. A good future for testing would be one with a lot fewer “testers” but each one of those testers being passionate about his craft.”

  • The role of a Test Manager in agile by Katrina Clokie – “In a cross-skilled team, the agile tester must ensure that the quality of testing is good regardless of who in the team is completing it. The tester becomes the spokesperson for collaborative testing practices, and provides coaching via peer reviews or workshops to those without a testing background.”

  • Speaking to Your Business Using Measurements by Justin Rohrman – “In my experience, no one measure did a great job of telling the story about my software ecosystem. I’ve been deceived by groups of measures, too, because I misunderstood their weaknesses. If we are so easily deceived by measurements, imagine what happens when we send them off to others who need quick, high-level information.”

 

unnamed

Photo in Ranakpur India by Justin Hunter.

 

  • Shine a light by Rob Lambert – “Tours, personas and a variety of other test ideas give you a way of re-shining your light. Use these ideas to see the product in different ways, but never forget that it’s often time that is against you. And time is one of the hardest commodities to argue for during your testing phase.”

  • Using mind-mapping software as a visual test management tool by Aaron Hodder – “I like to employ a style of software testing that emphasises the personal freedom and responsibility of the individual tester to continually optimise the quality of his/her work by treating test-related learning, test design, test execution and test result interpretation as mutually supportive activities that run in parallel throughout the project. When performed by a skilled tester, this approach yields valuable and consistent results.”

  • Helpful Tips for Hiring Better Testers by Isaac Howard – “I looked at the good testers around me and tried to identify the “whys” of their success. All of them were driven to learn and capable of adapting to change. If they didn’t know a tool or a tech, they learned it. Because under the hood, testing is learning and relearning software everyday. The following are seven changes I made to my interviewing process.”

By: John Hunter on Jun 10, 2014

Categories: Software Testing

Justin posted this to the Hexawise Twitter feed

cave-question

It sparked some interesting and sometimes humorous discussion.

cave-exploration

The parallels to software testing are easy to see. In order to learn useful information we need to understand what the purpose of the visit to the cave is (or what we are seeking to learn in software testing). Normally the most insight can be gained by adjusting what you seek to learn based on what you find. As George Box said

Always remember the process of discovery is iterative. The results of each stage of investigation generating new questions to answered during the next.

Some "software testing" amounts to software checking or confirming that a specific known result is as we expect. This is important for confirming that the software continues to work as expected as we make changes to the code (due to the complexity of software unintended consequences of changes could lead to problems if we didn't check). This can, as described in the tweet, take a form such as 0:00 Step 1- Turn on flashlight with pre-determined steps all the way to 4:59 Step N- Exit But checking is only a portion of what needs to be done. Exploratory testing relies on the brain, experience and insight of a software tester to learn about the state of the software; exploratory testing seeks to go beyond what pre-defined checking can illuminate. In order to explore you need to understand the aim of the mission (what is important to learn) and you need to be flexible as you learn to adjust based on your understanding of the mission and the details you learn as you take your journey through the cave or through the software. Exploratory software testing will begin with ideas of what areas you wish to gain an understanding of, but it will provide quite a bit of flexibility for the path that learning takes. The explorer will adjust based on what they learn and may well find themselves following paths they had not thought of prior to starting their exploration.

 

Related: Maximizing Software Tester Value by Letting Them Spend More Time Thinking - Rapid Software Testing Overview - Software Testers Are Test Pilots - What is Exploratory Testing? by James Bach

By: John Hunter on Apr 8, 2014

Categories: Exploratory Testing, Scripted Software Testing, Software Testing, Testing Checklists

Coining a New Term

I'm coining a new term today, "grapefruit juice bugs."

My inspiration for this term is a blog post in the New York Times that David Pogue wrote. I was fascinated by the post and it got me to thinking about a particular kind of bugs in software that are more common than most people may realize. You could say that these bugs are surprisingly common. In fact, if you wanted to be more precise, you could even say that this term applies to a specific type of "surprisingly common type of surprising bugs." Let me explain.

There's something about the chemical makeup of grapefruit juice that makes it interact with our biology and a large number of different drugs in ways which result in dangerous conditions. For example, certain drugs lose their effectiveness dramatically when interacting with grapefruit juice which can have life-threatening consequences. Other times, the interactions with grapefruit juice can dramatically increase a drug's potency. This can result in "safe doses" becoming very unsafe.

Grapefruit Is a Culprit in More Drug Reactions

The 42-year-old was barely responding when her husband brought her to the emergency room. Her heart rate was slowing, and her blood pressure was falling. Doctors had to insert a breathing tube, and then a pacemaker, to revive her.

They were mystified: The patient’s husband said she suffered from migraines and was taking a blood pressure drug called verapamil to help prevent the headaches. But blood tests showed she had an alarming amount of the drug in her system, five times the safe level.

Did she overdose? Was she trying to commit suicide? It was only after she recovered that doctors were able to piece the story together.

“The culprit was grapefruit juice,” said Dr. Unni Pillai, a nephrologist in St. Louis, Mo. ...

The previous week, she had been subsisting mainly on grapefruit juice. Then she took verapamil, one of dozens of drugs whose potency is dramatically increased if taken with grapefruit. In her case, the interaction was life-threatening.

Last month, Dr. David Bailey, a Canadian researcher who first described this interaction more than two decades ago, released an updated list of medications affected by grapefruit. There are now 85 such drugs on the market, he noted, including common cholesterol-lowering drugs, new anticancer agents, and some synthetic opiates and psychiatric drugs, as well as certain immunosuppressant medications taken by organ transplant patients, some AIDS medications, and some birth control pills and estrogen treatments. ... Under normal circumstances, the drugs are metabolized in the gastrointestinal tract, and relatively little is absorbed, because an enzyme in the gut called CYP3A4 deactivates them. But grapefruit contains natural chemicals called furanocoumarins, that inhibit the enzyme, and without it the gut absorbs much more of a drug and blood levels rise dramatically.

For example, someone taking simvastatin (brand name Zocor) who also drinks a small 200-milliliter, or 6.7 ounces, glass of grapefruit juice once a day for three days could see blood levels of the drug triple, increasing the risk for rhabdomyolysis, a breakdown of muscle that can cause kidney damage.

 

So what do interactions between grapefruit juice and drugs have to do with software testing?

Like grapefruit juice's impact on prescription drugs, software testing involves critical interactions between different parts of the system. And risks exist when these different parts interact with one another. This is true whether you're talking about "large parts" interact in System Testing or "small parts" interact in Unit Testing.

Interactions between things are a very rich source of bugs in software. As anyone who has heard the infernal phrase "works on my machine" can tell you, software features and functions often work perfectly fine in many usage scenarios, hardware and software configurations , etc. - only to fail to work in ever-so-slightly different situations.

 

The difference between plain old every-day "Dual-Mode Faults" and "Grapefruit Juice Bugs"

A dual-mode fault occurs whenever two test inputs must both be present to trigger a defect. Most software testers start encountering them quite frequently within days of starting their jobs. Some examples:

  • This "buy" button works fine. Except when the customer is a "new user." (First, action = "click on the buy button" and Second, customer = "new user")

  • Transaction prices for share purchases are calculated correctly. Except when denominated in Japanese Yen. (First, Action = "sell shares" and Second, Currency = "Japanese Yen")

Like grapefruit juice's impact on prescription drugs, software testing involves critical interactions between different parts of the system. And risks exist when these different parts interact with one another. This is true whether you're talking about "large parts" interact in System Testing or "small parts" interact in Unit Testing.

While all grapefruit juice bugs are dual-mode faults, not all dual-mode faults are Grapefruit Juice Bugs:

  • Grapefruit juice bugs have got to have a little of the element of surprise in them. When you explain them to a developer, their first reaction should be "Huh? How is that even possible?" or at least "Hmmm... That's odd. Let me investigate."

  • Anything along the lines of "This feature usually works, except in IE6, when..." is almost definitely not a grapefruit juice bug. Problematic interactions with IE6 are an incredibly common type of dual-mode fault, not a surprising one.

Whenever you hear "works on my machine" replies to your bug reports, and it takes a while for the issue to be replicated, odds are pretty good that a grapefruit juice bug might be involved.

Here's an example of an especially surprising grapefruit juice bug. This excerpt from Apple's online help files that the company posted after users of the original iPad complained about problems with Wi-Fi connectivity. Certain screen brightness settings were causing problems with the Wi-Fi signals. I'm not even to begin to guess how one would have anything to do with the other.

Auto-Scripting-Exercises-at-1.30.13-PM1

How to identify grapefruit juice bugs during your testing?

What is a tester to do when faced with more possible potential grapefruit juice bugs than he can handle using traditional methods?

If you're a software tester trying to do your best to determine whether a feature or function in your System Under Test will work "on everyone's machine," you've got a nightmare on your hands . Really nasty combinatorial explosions arise when you consider all of the possible combinations that would be required to test multiple hardware options, multiple software options, multiple usage scenarios, multiple test data inputs (and multiple combinations of the test data itself), multiple ways in which users enter data, and all of the rest of the "stuff that could vary" when people use your application. If you take the time to think expansively about the possible variations in a medium-sized applications, Quadrillions of possible tests often result.

While not eating grapefruit and not drinking grapefruit juice might be wise if you are taking drugs, there is rarely, if ever, such an easy method for eliminating the possibility of negative results due to software interactions. Refusing to support IE 6 in order to avoid the disproportionate number of grapefruit juice-like problematic interactions associated with IE6 would be as close as you could come in the world of software.

Design of Experiments-based test design methods can help testers come to grips with this challenge. Orthogonal array software testing (often referred to as OATS or simply OA testing) is a test design strategy that allows us to efficiently detect bugs created by interactions within the system. Orthogonal array software testing is based on the principles of multifactor designed experiments as first explored by Sir RA Fisher.

Design of Experiments-based test design methods are very-closely related to pairwise testing (AKA allpairs testing, all pairs testing, and pairwise-testing). Any of these test design strategies will allow a software tester to quickly generate a set of tests that includes tests for every single pair of test inputs.

This approach to test design often has multiple advantages, including faster test creation, more varied test scenarios, 100% coverage of all potential dual-mode faults (including hard-to-predict grape-fruit juice bugs), and often a smaller resulting set of tests that will be quicker to execute. Having said that, it is by no means a magical silver bullet. This approach to test design requires test designers with above average analytical abilities to identify the appropriate Parameters and Values for their system under test; this is sometimes easier said than done because it requires a new mindset from test designers.

Software testers can take solace that the challenges of software testing, while significant, are simple when compared to trying to understand the effects of drug interactions in people.

Combinatorial testing can look at bugs created by the interaction between multiple (3, 4, 5, 6...) variables. So if there was a bug that didn't get triggered just by using Chrome on Windows but it would get triggered if you also tried to replace an existing photo in your profile with a new profile photo into your profile (test idea number 3), then pairwise testing might not catch it. Pairwise test design would create a set of tests that would include at least one test for each of these pairs:

  • Chrome & Windows and

  • Chrome & replace photo and

  • Windows & replace photo, but...

A set of pairwise might not fail to test for the specific combination of all three of those test inputs in the same test. With the use of combinatorial test design approaches, you could create test plans with 100% coverage for 3 way interactions and be sure that all 3-way interactions or 4-way interactions are covered. When you create sets of 3-way tests, 4-way tests, 5-way tests, and 6-way tests though, you'll quickly discover that the number of tests required starts to balloon.

Hexawise allows you to create test plans with the coverage interactions you desire. This allows you to create sets of tests from 2-way up all the way up to phenomenally-thorough 6-way sets of tests. In fact, it even lets you generate clever sets of risk-based tests that will, say, prioritize comprehensive 4-way coverage on 4 sets of Parameter Values while ensuring only pairwise coverage of the other, lower-priority, interactions in your system under tests. Hexawise also lets you create mixed strength test plans so if you have certain factors that you are very concerned about and want to provide coverage for more possible interactions you can set the interaction levels for those at a higher level.

 

Related: Hexawise Tip: Using Value Expansions and Value Pairs to Handle Dependent Values - Maximize Test Coverage Efficiency And Minimize the Number of Tests Needed - How to Model and Test CRUD Functionality - 25 Great Quotes for Software Testers

By: Justin Hunter on Feb 11, 2014

Categories: Bugs, Combinatorial Testing, Design of Experiments, Multi-variate Testing, Multi-variate Testing, Pairwise Software Testing, Software Testing, Testing Strategies

The process used to hire employees is inefficient in general and even more inefficient for knowledge work. Justin Hunter, Hexawise CEO, posted the following tweet:

The labor market is highly inefficient for software testers. Many excellent testers are undervalued relative to average testers. Agree?

 

The tweet sparked quite a few responses:

inefficient-job-market-tweet

I think there are several reasons for why the job market is inefficient in general, and for why it is even more inefficient for software testing than for most jobs.

 

  • Often, how companies go about hiring people is less about finding the best people for the organization and more about following a process that the organization has created. Without intending to, people can become more more concerned about following procedural rules than in finding the best people.

  • The hiring process is often created much like software checking, a bunch of simple things to check - not because doing so is actually useful but because simple procedural checks are easy to verify. So organizations require a college degree (and maybe even require a specific major). And they will use keywords to select or reject applicants. Or require certification or experience with a specific tool. Often the checklist used to disqualify people contains items that might be useful but shouldn't be used as barriers but it is really easy for people that don't understand the work to apply the rules in the checklist to filter the list of applicants.

  • It is very hard to hire software testers well when those doing the hiring don't understand the role software testing should play. Most organizations don't understand, so they hire for software checkers. They, of course, don't value people that could provide much more value (software testers that go far beyond checking). The weakness of hiring without understanding the work is common for knowledge work positions and likely even more problematic for software testing due to the even worse understanding of what they should be doing compared to most knowledge workers.

 

And there are plenty more reasons for the inefficient market.

Here are few ideas that can help improve the process:

  • Spend time to understand and document what your organization seeks to gain from new hires.

  • Deemphasize HR's role in the talent evaluation process and eliminate dysfunctional processes that HR may have instituted. Talent evaluation should be done by people that understand the work that needs to get done. HR can be useful in providing guidance on legal and company-decided policies for hiring. Don't have people that can't evaluate the difference between great testers and good testers decide who should be hired or what salary is acceptable. Incidentally, years of experience, certifications, degrees, past salary and most anything else HR departments routinely use are often not correlated to the value a potential employee brings.

  • A wonderful idea, though a bit of a challenge in most organizations, is to use job auditions. Have the people actually do the job to figure out if they can do what you need or not (work on a project for a couple weeks, for example). This has become more common in the last 10 years but is still rare.

  • I also believe you are better off hiring for somewhat loose job descriptions, if possible, and then adjusting the job to who you hire. That way you can maximize the benefit to the organization based on the people you have. At Hexawise, for example, most of the people we hire have strengths in more than one "job description" area. Developers with strong UI skills, for instance, are encouraged to make regular contributions in both areas.

  • Creating a rewarding work environment helps (this is a long term process). One of the challenges in getting great people is they are often not interested in working for dysfunctional organizations. If you build up a strong testing reputation great testers will seek out opportunities to work for you and when you approach great testers they will be more likely to listen. This also reduces turnover and while that may not seem to relate to the hiring process is does (one reason we hire so poorly is we don't have time to do it right, which is partly because we have to do so much of it).

  • Having employees participate in user groups and attending conferences can help your organization network in the testing community. And this can help when you need to hire. But if your organization isn't a great one for testers to work in, they may well leave for more attractive organizations. The "solution" to this risk is not to stunt the development of your staff, but to improve the work environment so testers want to work for your organization.

 

Great quote from Dee Hock, founder of Visa:

Hire and promote first on the basis of integrity; second, motivation; third, capacity; fourth, understanding; fifth, knowledge; and last and least, experience. Without integrity, motivation is dangerous; without motivation, capacity is impotent; without capacity, understanding is limited; without understanding, knowledge is meaningless; without knowledge, experience is blind. Experience is easy to provide and quickly put to good use by people with all the other qualities.

Please share your thoughts and suggestions on how to improve the hiring process.

 

Related: Finding, and Keeping, Good IT People - Improving the Recruitment Process - Six Tips for Your Software Testing Career - Understanding How to Manage Geeks - People: Team Members or Costs - Scores of Workers at Amazon are Deputized to Vet Job Candidates and Ensure Cultural Fit

By: John Hunter on Jan 14, 2014

Categories: Checklists, Software Testing, Career

The Hexawise Software Testing blog carnival focuses on sharing interesting and useful blog posts related to software testing.

 

  • Using mind-mapping software as a visual test management tool by Aaron Hodder - "I want to be able to give and receive as much information as I can in the limited amount of time I have and communicate it in a way that is respectful of others' time and resources. These are my values and what I think constitutes responsible testing."

  • Healthcare.gov and the Tyranny of the Innocents by James Bach - "Management created the conditions whereby this project was 'delivered' in a non-working state. Not like the Dreamliner. The 787 had some serious glitches, and Boeing needs to shape that up. What I’m talking about is boarding an aircraft for a long trip only to be told by the captain 'Well, folks it looks like we will be stuck here at the gate for a little while. Maintenance needs to install our wings and engines. I don’t know much about aircraft building, but I promise we will be flying by November 30th. Have some pretzels while you wait.'"

 

jungle-bridge

Rope bridge in the jungle by Justin Hunter

 

  • Software Testers Are Test Pilots by John Hunter - "Software testers should be test pilots. Too many people think software testing is the pre-flight checklist an airline pilot uses."

  • Where to begin? by Katrina Clokie - "Then you need to grab your Product Owner and anyone else with an interest in testing (perhaps architect, project manager or business analyst, dependent on your team). I'm not sure what your environment is like, usually I'd book an hour meeting to do this, print out my mind map on an A3 page and take it in to a meeting room with sticky notes and pens. First tackle anything that you've left a question mark next to, so that you've fleshed out the entire model, then get them to prioritise their top 5 things that they want you to test based on everything that you could do."

  • Being a Software Tester in Scrum by Dave McNulla - "Pairing on development and testing strengthens both team members. With people crossing disciplines, they improve understanding of the product, the code, and what other stakeholders find important."

  • Stop Writing Code You Can’t Yet Test by Dennis Stevens - "The goal is not to write code faster. The goal is to produce valuable, working, testing, remediated code faster. The most expensive thing developers can do is write code that doesn’t produce something needed by anyone (product, learning, etc). The second most expensive thing developers can do is write code that can’t be tested right away."

  • Is Healthcare.gov security now fixed? by Ben Simo - "I am very happy that the most egregious issue was immediately fixed. Others issues remain. The vulnerabilities I've listed above are defects that should not make it to production. It doesn't take a security expert or “super hacker” to exploit these vulnerabilities. This is basic web security. Most of these are the kinds of issues that competent web developers try to avoid; and in the rare case that they are created, are usually found by competent testers."

  • Embracing Chaos Testing Helps Create Near-Perfect Clouds - "Chaos Monkey works on the simple premise that if we need to design for high availability, we should design for failure. To design for failure, there should be ways to simulate failures as they would happen in real-world situations. This is exactly what a Chaos Monkey helps achieve in a cloud setup.
    Netflix recently made the source code of Chaos Monkey (and other Simian Army services) open source and announced that more such monkeys will be made available to the community."

  • Bugs in UK Post Office System had Dire Consequences - "A vocal minority of sub-postmasters have claimed for years that they were wrongly accused of theft after their Post Office computers apparently notified them of shortages that sometimes amounted to tens of thousands of pounds. They were forced to pay in the missing amounts themselves, lost their contracts and in some cases went to jail. Second Sight said the Post Office's initial investigation failed at first to identify the root cause of the problems. The report says more help should have been given to sub-postmasters, who had no way of defending themselves."

  • Traceability Matrix: Myth and Tricks by Adam Howard - "And this is where we get to the crux of the problem with traceability matrices. They are too simplistic a representation of an impossibly complex thing. They reduce testing to a series of one to one relationships between intangible ideas. They allow you to place a number against testing. A percentage complete figure. What they do not do is convey the story of the testing."

  • Six Tips for Your Software Testing Career by John Hunter - "Read what software testing experts have written. It’s surprising how few software testers have read books and articles about software testing.Here are some authors (of books, articles and blogs) that I've found particularly useful..."

By: John Hunter on Dec 16, 2013

Categories: Software Testing

Since creating Hexawise, I've worked with executives at companies around the world who have found themselves convinced in the value of pairwise testing. And then they need to convince their organization of the value.

They often follow the following path: first thinking "pairwise testing is a nice method in theory, but not applicable in our case" then "pairwise is nice in theory and might be applicable in our case" to "pairwise is applicable in our case" and finally "how do I convince my organization."

In this post I review my history helping convince organizations to try and then adopt pairwise, and combinatorial, software testing methods.

About 8 years ago, I was working at a large systems integration firm and was asked to help the firm differentiate its testing offerings from the testing services provided by other firms.

While I admittedly did not know much about software testing then but by happy coincidence, my father was a leading expert in the field of Design of Experiments. Design of Experiments is a field that has applicability in many areas (including agriculture, advertising, manufacturing, etc.) The purpose of Design of Experiments is to provide people with tools and approaches to help people learn as much actionable information as possible in as few tests as possible.

I Googled "Design of Experiments Software Testing." That search led me to Dr. Madhav Phadke (who, by coincidence, had been a former student of my father). More than 20 years ago now, Dr. Phadke and his colleagues at ATT Bell Labs had asked the question you're asking now. They did an experiment using pairwise test design / orthogonal array test design to identify a subset of tests for ATT's StarMail system. The results of that testing effort were extraordinarily successful and well-documented.

Shortly after doing that, while working at that systems integration firm, I began to advocate to anyone and everyone who would listen that designing approach to designing tests promised to be both (a) more thorough and (b) require (in most but not all cases) significantly fewer tests. Results from 17 straight projects confirmed that both of these statements were true. Consistently.

 

Repeatable Steps to Confirm Whether This Approach Delivers Efficiency and Thoroughness Improvement (and/or document a business case/ROI calculation)

How did we demonstrate that this test design approach led to both more thorough testing and much more efficient testing? We followed these steps:

  1. Take an existing set of 30 - 100 existing tests that had already been created, reviewed, and approved for testing (but which had not yet been executed).

  2. Using the test ideas included in those existing tests, design a set of pairwise tests (often approximately half as many tests as were in the original set). When putting your tests together, if there are particular, known, high-priority scenarios that stakeholders believe are important to test, it is important to make sure that that you "force" your pairwise test generator to include such high-priority scenarios.

  3. Have two different testers execute both sets of tests at the same time (e.g., before developers start fixing any defects that are uncovered by testers executing either set of tests)

Document the following:

  • How long did it take to execute each set of tests?

  • How many unique defects were identified by each set of tests?

  • How long did it take to create and document each set of tests?*

 

*This third measurement was usually an estimate because a significant number of teams had not tracked the amount of time it took to create the original set of tests.

The results in 17 different pairwise testing "bake-off" projects conducted at my old firm included:

  • Defects found per tester hour during test execution: when pairwise tests were used, more than twice as many defects were found per tester hour

  • Total defects found: pairwise tests as many or more defects in every single project (despite the fact that in almost every case there were significantly more tests in the each original set of tests)

  • Defects found by pairwise tests but missed by traditional tests: a lot (I forget the exact number)

  • Defects found by traditional tests but missed by pairwise tests: zero

  • Amount of time to select and document tests: much less time required when a pairwise test generator was used (As mentioned above, precise measurements were difficult to gather here)

 

More recent project benefits have included these:

bcbs1

bcbs2

Those experiences - combined with the realization that many Fortune 500 firms were starting to try to implement smarter test design methods to achieve these kinds of benefits but were struggling to find a test design tool that was user-friendly and would integrate into their other tools - led me to the decision to create Hexawise.

 

Additional Advice and Lessons Learned Based on My Experiences

Once the testing the value of pairwise software testing at a specific organization it is very common to find the proponent of taking advantage of pairwise testing advantages to find themselves saying:

I have already elaborated some test plans that would save us up to 50% effort with that method. But now my boss and other colleagues are asking me for a proof that these pairwise test cases suffice to make sure our software is running well.

In that case, my advice is three-fold:

First, appreciate how your own thinking has evolved and understand that other people will need to follow a similar journey (and that others won't have as much time to devote as you have had to experience learnings first-hand).

When I was creating Hexawise, George Box, a Design of Experiments expert with decades of experience explaining to skeptical executives how Design of Experiments could deliver transformational improvements to their organizations' efficiency and effectiveness, told me "Justin, you'll experience three phases of acceptance and acceptance will happen more gradually than you would expect it to happen. First, people will tell you 'It won't work.' Next, they'll say "It won't work here." Eventually, he said with a smile, they'll tell you 'Of course this works. I thought of it first!'

When people hear that you can double their test execution productivity with a new approach, they won't initially believe you. They'll be skeptical. Most people you're explaining this approach to will start with the thought that "it is nice in theory but not applicable to what I'm doing." It will take some time and experience for people to understand and appreciate how dramatic the benefits are.

Second, people will instinctively be dismissive of pairwise testing case study after case study after case study that show how effective this approach has been for testers in virtually all types of industries and all types and phases of testing. George Box was right when he predicted that people will often respond with 'It won't work here.' Sometimes it is hard not to smile when people take that position.

Case in point: I will be talking to a senior executive at a large capital markets firm soon about how our tool can help them transform the efficiency and effectiveness of their testing group. And I can introduce them to a client of ours that is using our test design tool extensively in every single one of their most important IT projects. Will that executive take me up on my offer? I hope so, but based on past experience, I suspect odds are good that he'll instead react with 'Yes, yes, sure, if companies were people, that company would be our company's identical twin, but still... It won't work here.'

Third, at the end of the day, the most effective approach I have found to address that understandable skepticism and to secure organizational-level buy-in and commitment is through gathering hard, indisputable evidence on multiple projects that the approach works at the company itself through a bake-off approach (e.g., following those four steps outlined above. A few words of advice though.

My proposed approach isn't for the faint of heart. If you're working at a large company with established approaches, you'll need patience and persistence.

Even after you gather evidence that this approach works in Business Unit A, and B and C, someone from Business Unit D will be unconvinced with the compelling and irrefutable evidence you have gathered and tell you 'It won't work here. Business Unit D is unique.' The same objections may likely arise with results from "Type of Testing" A, B, and C. As powerful and widely-applicable as this test design approach is, always remember (and be clear with stakeholders) that it is not a magical silver bullet.

James Bach raises several valid limitations with using this approach. In particular, this approach won't work unless you have testers who have relatively strong analytical skills driving the test design process. Since pairwise test case generating tools are dependent upon thoughtful test designers to identify appropriate test inputs to vary, this approach (like all test design approaches) is subject to a "garbage in / garbage out" risk.

Project leads will resist "duplicating effort." But unless you do an actual bake-off stakeholders won't appreciate how broken their existing process is. There's inevitably far more wasteful repetition hidden away in standard tests than people realize. When you start reporting a doubling of tester productivity on several projects, smart managers will take notice and want to get involved. At that point - hopefully - your perseverance should be rewarded.

 

Some benefits data and case studies that you might find useful:

 

If you can't change your company, consider changing companies

Lastly, remember that your new-found skills are in high demand whether or not they're valued at your current company. And know that, despite your best efforts and intentions, your efforts might not convince skeptics. Some people inevitably won't be willing to take the time to understand. If you find yourself in a situation where you want to use this test design approach (because you know that these approaches are powerful, practical, and widely-applicable) but that you don't have management buy-in, then consider whether or not it would be worth leaving your current employer to join a company that will let you use your new-found skills.

Most of our clients, for example, are actively looking for software test designers with well developed pairwise and combinatorial test design skills. And they're even willing to pay a salary premium for highly analytical test designers who are able to design sets of powerful tests. (We publicize such job openings in the LinkedIn Hexawise Guru group for testers who have achieved "Guru" level status in the self-paced computer-based-training modules in the tool).

 

Related: Looking at the Empirical Evidence for Using Pairwise and Combinatorial Software Testing - Systematic Approaches to Selection of Test Data - Getting Known Good Ideas Adopted

By: Justin Hunter on Nov 21, 2013

Categories: Pairwise Software Testing, Software Testing, Software Testing Efficiency, Testing Case Studies, Testing Strategies

Software testers should be test pilots. Too many people think software testing is the pre-flight checklist an airline pilot uses.

 

The checklists airline pilots use before each flight are critical. Checklists are extremely valuable tools that help assure steps in a process are followed. Checklists are valuable in many professions. The Checklist – If something so simple can transform intensive care, what else can it do? by Atul Gawande

 

Sick people are phenomenally more various than airplanes. A study of forty-one thousand trauma patients—just trauma patients—found that they had 1,224 different injury-related diagnoses in 32,261 unique combinations for teams to attend to. That’s like having 32,261 kinds of airplane to land. Mapping out the proper steps for each is not possible, and physicians have been skeptical that a piece of paper with a bunch of little boxes would improve matters much. In 2001, though, a critical-care specialist at Johns Hopkins Hospital named Peter Pronovost decided to give it a try. … Pronovost and his colleagues monitored what happened for a year afterward. The results were so dramatic that they weren’t sure whether to believe them: the ten-day line-infection rate went from eleven per cent to zero. So they followed patients for fifteen more months. Only two line infections occurred during the entire period. They calculated that, in this one hospital, the checklist had prevented forty-three infections and eight deaths, and saved two million dollars in costs.

 

Checklists are extremely useful in software development. And using checklist-type automated tests is a valuable part of maintaining and developing software. But those pass-fail tests are equivalent to checklists - they provide a standardized way to check that planned checks pass. They are not equivalent to thoughtful testing by a software testing professional.

I have been learning about software testing for the last few years. This distinction between testing and checking software was not one I had before. Reading experts in the field, especially James Bach and Michael Bolton is where I learned about this idea.

 

Testing is the process of evaluating a product by learning about it through experimentation, which includes to some degree: questioning, study, modeling, observation and inference.

(A test is an instance of testing.)

Checking is the process of making evaluations by applying algorithmic decision rules to specific observations of a product.

 

I think this is a valuable distinction to understand when looking to produce reliable and useful software. Both are necessary. Both are done too little in practice. But testing (as defined above) is especially underused - in the last 5 years checking has been increasing significantly, which is good. But now we really need to focus on software testing - thoughtful experimenting.

 

Related: Mistake Proofing the Deployment of Software Code - Improving Software Development with Automated Tests - Rapid Software Testing Overview Webcast by James Bach - Checklists in Software Development

By: John Hunter on Nov 13, 2013

Categories: Checklists, Software Testing, Testing Checklists, Testing Strategies

Recently, the following defect made the news and was one of the most widely-shared articles on the New York Times web site. Here's what the article, Computer Snag Limits Insurance Penalties on Smokers said:

A computer glitch involving the new health care law may mean that some smokers won’t bear the full brunt of tobacco-user penalties that would have made their premiums much higher — at least, not for next year.

The Obama administration has quietly notified insurers that a computer system problem will limit penalties that the law says the companies may charge smokers, The Associated Press reported Tuesday. A fix will take at least a year.

 

Tip of the Iceberg

This defect was entirely avoidable and predictable. Its safe to expect that hundreds (if not thousands) of similar defects related to Obamacare IT projects will emerge in the weeks and months to come. Had testers used straightforward software test design prioritization techniques, bugs like these would have been easily found. Let me explain.

 

There's no Way to Test Everything

If the developers and/or testers were asked how could this bug could sneak past testing, they might at first say something defensive, along the lines of: "We can't test everything! Do you know how many possible combinations there are?" If you include 40 variables (demographic information, pre-existing conditions, etc.) in the scope of this software application, there would be:

41,231,686,041,600,000

possible scenarios to test. That's not a typo: 41 QUADRILLION possible combinations. As in it would take 13 million years to execute those tests if we could execute 100 tests every second. There's no way we can test all possible combinations. So bugs like these are inevitably going to sneak through testing undetected.

 

The Wrong Question

When the developers and testers of a system say there is no way they could realistically test all the possible scenarios, they're addressing the wrong challenge. "How long would it take to execute every test we can think of?" is the wrong question. It is interesting but ultimately irrelevant that it would take 13 million years to execute those tests.

 

The Right Question

A much more important question is "Given the limited time and resources we have available for testing, how can we test this system as thoroughly as possible?" Most teams of developers and software testers are extremely bad at addressing this question. And they don't realize nearly how bad they are. The Dunning Kruger effect often prevents people from understanding the extent of their incompetence; that's a different post for a different day. After documenting a few thousand tests designed to cover all of the significant business rules and requirements they can think of, testers will run out of ideas, shrug their shoulders in the face of the overwhelming number of total possible scenarios and declare their testing strategy to be sufficiently comprehensive. Whenever you're talking about hundreds or thousands of tests, that test selection strategy is a recipe for incredibly inefficient testing that both misses large numbers of easily avoidable defects and wastes time by testing certain things again and again. There's a better way.

 

The Straightforward, Effective Solution to this Common Testing Challenge: Testers Should Use Intelligent Test Prioritization Strategies

If you create a well-designed test plan using scientific prioritization approaches, you can reduce the number of individual tests to test tremendously. It comes down to testing the system as thoroughly as possible in the time that's available for testing. There are well-proven methods for doing just that.

 

There are Two Kinds of Software Bugs in the World

Bugs that don't get found by testers sneak into production for one of two main reasons, namely:

  • "We never thought about testing that" - An example that illustrates this type of defect is one James Bach told me about. Faulty calculations were being caused by an overheated server that got that way because of a blocked vent. You can't really blame a tester who doesn't think of including a test involving a scenario with a blocked vent.

  • "We tested A; it worked. We tested B; it worked too.... But we never tested A and B together." This type of bug sneaks by testers all too often. Bugs like this should not sneak past testers. They are often very quick and easy to find. And they're so common as to be highly predictable.

 

Let's revisit the high-profile bug Obamacare bug that will impact millions of people and take more than a year to fix. Here's all that would have been required to find it:

  • Include an applicant with a relatively high (pre-Medicare) age. Oh, and they smoke.

 

Was the system tested with a scenario involving an applicant who had a relatively high age? I'm assuming it must have been.

Was the system tested with a scenario involving an applicant who smoked? Again, I'm assuming it must have been.

Was the system tested with a scenario involving an applicant who had a relatively high age who also smoked? That's what triggers this important bug; apparently it wasn't found during testing (or found early enough).

 

If You Have Limited Time, Test All Pairs

Let's revisit the claim of "we can't execute all 13 million-years-worth of tests. Combinations like these are bound to sneak through, untested. How could we be expected to test all 13 million-years-worth of tests?" The second two sentences are preposterous.

  • "Combinations like these are bound to sneak through, untested." Nonsense. In a system like this, at a minimum, every pair of test inputs should be tested together. Why? The vast majority of defects in production today would be found simply by testing every possible pair of test inputs together at least once.

  • "How could we be expected to test all 13 million-years-worth of tests?" Wrong question. Start by testing all possible pairs of test inputs you've identified. Time-wise, that's easily achievable; its also a proven way to cover a system quite thoroughly in a very limited amount of time.

 

Design of Experiments is an Established Field that was Created to Solve Problems Exactly Like This; Testers are Crazy Not to Use Design of Experiments-Based Prioritization Approaches

The almost 100 year-old field of Design of Experiments is focused on finding out as much actionable information as possible in as few experiments as possible. These prioritization approaches have been very widely used with great success in many industries, including advertising, manufacturing, drug development, agriculture, and many more. While Design of Experiments test design techniques (such as pairwise testing and orthogonal array testing / OA testing) are increasingly becoming used by software testing teams but far more teams could benefit from using these smart test prioritization approaches. We've written posts about how Design of Experiments methods are highly applicable to software testing here and here, and put an "Intro to Pairwise Testing" video here. Perhaps the reason this powerful and practical test prioritization strategy remains woefully underutilized by the software testing industry at large is that there are too few real-world examples explaining "this is what inevitably happens when this approach is not used... And here's how easy it would be to avoid this from happening to you in your next project." Hopefully this post helps raise awareness.

 

Let's Imagine We've Got One Second for Testing, Not 13 Million Years; Which Tests Should We Execute?

Remember how we said it would take 13 million years to execute all of the 41 quadrillion possible tests? That calculation assumed we could execute 100 tests a second. Let's assume we only have one second to execute tests from those 13 million years worth of tests. How should we use that second? Which 100 tests should we execute if our goal is to find as many defects as possible?

If you have a Hexawise account, you can to your Hexawise account to view the test plan details and follow along in this worked example. To create a new account in a few seconds for free, go to hexawise.com/free.

By setting the 40 different parameter values intelligently, we can maximize the testing coverage achieved in a very small number of tests. In fact, in our example, you would only need to execute only 90 tests to cover every single pairwise combination.

The number of total possible combinations (or "tests") that are generated will depend on how many parameters (items/factors) and how many options (parameter values) there are for each parameter. In this case, the number of total possible combinations of parameters and values equal 41 quadrillion.

 

insurance-bug-1

This screen shot shows a portion of the test conditions that would be included the first 4 tests of the 90 tests that are needed to provide full pairwise coverage. Sometimes people are not clear about what "test every pair" means. To make this more concrete, by way of a few specific examples, pairs of values tested together in the first part of test number 1 include:

  • Plan Type = A tested together with Deductible Amount = High

  • Plan Type = tested together with Gender = Male

  • Plan Type = A tested together with Spouse = Yes

  • Gender = Male tested together with State = California

  • Spouse = Yes tested together with Yes (and over 5 years)

  • And lots of other pairs not listed here

 

insurance-bug-2

This screen shot shows a portion of the later tests. You'll notice that the values are shown in purple italics. Those values listed in purple italics are not providing new pairwise coverage. You will note in the first tests every single parameter value is providing new pairwise coverage value, toward the end few parameter value settings are providing new pairwise coverage. Once a specific pair has been tested, retesting it doesn't provide additional pairwise coverage. Sets of Hexawise tests are "front loaded for coverage." In other words, if you need to stop testing at any point before the end of the complete set of tests, you will have achieved as much coverage as possible in the limited time you have to execute your tests (whether that is 10 tests or 30 tests or 83). The pairwise coverage chart below makes this point visually; the decreasing number of newly tested pairs of values that appear in each test accounts for the diminishing marginal returns per test.

 

You Can Even Prioritize Your First "Half Second" of Tests To Cover As Much As Possible!

insurance-bug-3

This graph shows how Hexawise orders the test plan to provide the greatest coverage quickly. So if you get through 37 of the 90 tests needed for full pairwise coverage you have already tested over 90% of all the pairwise test coverage. The implication? Even if just 37 tests were covered, there would be a 90% chance that any given pair of values that you might select at random would be tested together in the same test case by that point.

 

Was Missing This Serious Defect an Understandable Oversight (Because of Quadrillions of Possible Combinations Exist) or was it Negligent (Because Only 90 Intelligently Selected Tests Would Have Detected it)?

A generous interpretation of this situation would be that it was "unwise" for testers to fail to execute the 90 tests that would have uncovered this defect.

A less generous interpretation would be that it was idiotic not to conduct this kind of testing.

The health care reform act will introduce many such changes as this. At an absolute minimum, health insurance firms should be conducting pairwise tests of their systems. Given the defect finding effectiveness of pairwise testing coverage, testing systems any less thoroughly is patently irresponsible. And for health insurance software testing it is often wiser to expand to test all triples or all quadruples given the interaction between many variables in health insurance software.

Incidentally, to get full 3 way test coverage (using the same example as above) would require 2,090 tests.

 

Related: Getting Started with a Test Plan When Faced with a Combinatorial Explosion - How Not to Design Pairwise Software Tests - Efficient and Effective Test Design

By: Justin Hunter on Sep 26, 2013

Categories: Combinatorial Software Testing, Hexawise test case generating tool, Multi-variate Testing, Pairwise Software Testing, Software Testing, Testing Strategies

Software testing is becoming more critical as the proportion of our economic activity that is dependent on software (and often very complex software) continues to grow. There seems to be a slowly growing understanding of the importance of software testing (though it still remains an under-appreciated area). My belief is that skilled software testers will be appreciated more with time and that the opportunities for expert software testers will expand.

Here are a few career tips for software testers.

 

  • Read what software testing experts have written. It's surprising how few software testers have read books and articles about software testing.Here are some authors (of books, articles and blogs) that I've found particularly useful. Feel free to suggest other software testing authors in the comments.

James Bach

Cem Kaner

Lisa Crispin

Michael Bolton

Lanette Creamer

Jerry Weinberg

Keith Klain

James Whittaker - posts while at Google and his posts with his new employer, Microsoft

Pradeep Soundararajan

Elisabeth Hendrickson

Ajay Balamurugadas

Matt Heusser

our own, Justin Hunter

 

  • Join the community You'll learn a lot as a lurker, and even more if you interact with others. Software Testing Club is one good option. Again, following those experts (listed above) is useful; engaging with them on their blogs and on Twitter is better. The software testing community is open and welcoming. In addition, see: 29 Testers to Follow on Twitter and Top 20 Software Testing Tweeps. Interacting with other testers who truly care about experimenting, learning, and sharing lessons learned is energizing.

 

  • Develop your communication skills Communication is critical to a career in software testing as it is to most professional careers. Your success will be determined by how well you communicate with others. Four critical relationship are with: software developers, your supervisors, users of the software and other software testers.Unfortunately in many organizations, managers and corporate structures restrict your communication with some of these groups. You may have to work with what you are allowed, but if you don't have frequent, direct communication with those four groups, you won't be able to be as effective.
    Work on developing your communication skills. Given the nature of software testing two particular types of communication - describing problems ("what is it?") and explaining how significant problem is ("why should we fix it?") - are much more common than in many other fields. Learning how to communicate these things clearly and without making people defensive is of extra importance for software testers.
    A great deal of your communication will be in writing and developing your ability to communicate clearly will prove valuable to your continued growth in your career.
    Writing your own blog is a great way to further you career. You will learn a great deal by writing about software testing. You will also develop your writing ability merely by writing more. And you will create a personal brand that will grown your network of contacts. It also provides a way for those possibly interested in hiring you to learn more. We all know hiring is often done outside the job announcement - job application process. By making a name for yourself in the software testing community you will increase the chances of being recruited for jobs.

 

  • Do what you love You career will be much more rewarding if you find something you love to do. You will most likely have parts of your job you could do without but finding something you have a passion for makes all the difference in the world. If you don't have a passion for software testing, you are likely better off finding something you are passionate about and pursuing a career in that field.

Your work is going to fill a large part of your life, and the only way to be truly satisfied is to do what you believe is great work. And the only way to do great work is to love what you do. If you haven’t found it yet, keep looking. Don’t settle.

Steve Jobs

 

  • Practice and learn new techniques and ideas. James Bach provides such opportunities. Also the Hexawise tool, lets you easily and quickly try out different scenarios and learn by doing. We also provide guided training, help and webcasts explaining software testing concepts and how to apply them using Hexawise.We include a pathway that guides you through the process of learning about software testing, with learning modules, practical experience and tests along the way to make sure you learn what is important. And once you reach the highest level and become a Hexawise guru we have offer a community experience to allow all those reaching that level to share their experience and learn from each other.

 

  • Go to good software testing conferences.I've heard great things about CAST (by the Association for Software Testing), Test Bash, and Let's Test in particular. Going to conferences is energizing because while you're there, you're surrounded by the minority of people in this industry who are passionate about software testing. Justin, Hexawise founder and CEO, will be presenting at CAST next week, August 26-28 in Madison, Wisconsin.

 

Related: Software Testing and Career Advice by Keith Klain - Looking at the Empirical Evidence for Using Pairwise and Combinatorial Software Testing - Maximizing Software Tester Value by Letting Them Spend More Time Thinking

By: John Hunter on Aug 22, 2013

Categories: Software Testing

The Hexawise Software Testing blog carnival focuses on sharing interesting and useful blog posts related to software testing.

 

  • T-Shaped Testers and their role in a team by Rob Lambert - "I believe that testers, actually – anyone, can contribute a lot more to the business than their standard role traditionally dictates. The tester’s critical and skeptical thinking can be used earlier in the process. Their other skills can be used to solve other problems within the business. Their role can stretch to include other aspects that intrigue them and keep them interested."

  • Testing triangles, pyramids and circles, and UAT (the link was broken so it was removed) by Allan Kelly - "Thus: UAT and Beta testing can only truly be performed by USERS (yes I am shouting). If you have professional testers performing it then it is in effect a form of System Testing.
    This also means UAT/Beta cannot be automated because it is about getting real life users to user the software and get their feedback. If users delegate the task to a machine then it is some other form of testing."

 

flower-justin

Photo by Justin Hunter, taken in South Africa.

 

  • Software Testing in Distributed Teams by Lisa Crispin - "Remote pairing is a great way to promote constant communication among multiple offices and to keep people working from home 'in the loop'.
    I often remote pair with a developer to specify test scenarios, do exploratory testing, or write automated test scripts. We use screen-sharing software that allows either person to control the mouse and keyboard. Video chat is best, but if bandwidth is a problem, audio will do. Make sure everyone has good quality headphones and microphone, and camera if possible."

  • Seven Kinds of Testers by James Bach - "I propose that there are at least seven different types of testers: administrative tester, technical tester, analytical tester, social tester, empathic tester, user, and developer. As I explain each type, I want you to understand this: These types are patterns, not prisons. They are clusters of heuristics; or in some cases, roles. Your style or situation may fit more than one of these patterns."

  • Which is Better, Orthogonal Array or Pairwise Software Testing? by John Hunter and Justin Hunter - "After more study I have concluded that: Pairwise is more efficient and effective than orthogonal arrays for software testing. Orthogonal Arrays are more efficient and effective for manufacturing, and agriculture, and advertising, and many other settings."

  • Experience Report: Pairing with a Programmer by Erik Brickarp - "We have different investigation methods. The programmer did low level investigations really well adding debug printouts, investigating code etc. while I did high level investigations really well checking general patterns, running additional scenarios etc. Not only did this make us avoid getting stuck by changing 'method' but also, my high level investigations benefited from his low level additions and vice versa."

  • I decided to evolve a faster test case by Ben Tilly - "I first wrote a script to run the program twice, and report how many things it found wrong on the second run. I wrote a second script that would take a record set, sample half of it randomly, and then run the first script.
    I wrote a third script that would take all records, run 4 copies of my test program, and then save the smaller record set that best demonstrated the bug. Wash, rinse, and repeat..."

  • Tacit and Explicit Knowledge and Exploratory Testing by John Stevenson - "It is time we started to recognise that testing is a tacit activity and requires testers to think both creativity and critically."

  • Tear Down the Wall by Alan Page - "There will always be a place for people who know testing to be valuable contributors to software development – but perhaps it’s time for all testing titles to go away?"

By: John Hunter on Jul 10, 2013

Categories: Software Testing

Here is a wonderful webcast that provides a very quick, and informative, overview of rapid software testing.

Software testing is when a person is winding around a space searching that space for important information.

 

James Bach starts by providing a definition of software testing to set the proper thinking for the overview.

Rapid software testing is a set of heuristics [and a set of skills]. Heuristics live at the border of explicit and tacit knowledge... Heuristics solve problems when they are under the control of a skilled human... It takes skill to use the heuristics effectively - to solve the problems of testing. Rapid software testing focuses on the tester... Tacit skills are developed through practice.

 

Automated software tests are useful but limited. In the context of rapid software testing only a human tester can do software testing (automated checks are defined as "software checking"). See his blog post: Testing and Checking Refined.

 

Related: People are better at figuring out interesting ideas to test. Generating a highly efficient, maximally varied, minimally repetitive set of tests based on a given set of test inputs is something computer algorithms are more effective at than a person. - Hexawise Lets Software Testers Spend More Time Focused on Important Testing Issues - 3 Strategies to Maximize Effectiveness of Your Tests

By: John Hunter on Jul 2, 2013

Categories: Software Testing, Testing Strategies

We have created the Hexawise Software Testing Glossary. The purpose is to provide some background for both general software testing tool and some Hexawise specific details.

While we continue to develop, expand and improve the Hexawise software itself we also realize that users success with using Hexawise rests not only on the tool itself but on the knowledge of those using the tool. The software testing glossary adds to the list of offerings we provide to help users. Other sources of help include sample test plans, training material (on software testing in general and Hexawise in particular), detailed help on specific topics, webcast tutorials on using Hexawise and this blog.

In Hexawise 2.0 we integrated the idea of building your expertise directly into the Hexawise application with specific actions, reading and practicums that guide users from novice to practitioner to expert to guru.

achievements

View for a Hexawise user that is currently a practitioner and is on their way to becoming an expert.

 

We aim to provide software services, and educational material, that help software testers focus their energy, insite, knowledge and time where it is most valuable while Hexawise software automates some tasks and does other tasks - that are essentially impossible for people to do manually (such as creating the most effective test plan, with pairwise, or better, coverage, given the parameters and values determined by the tester).

Our help, training and webcast resources have been found to be very useful by Hexawise users. We hope the software testing glossary will also prove to be of value to Hexawise users.

 

Related: Maximizing Software Tester Value by Letting Them Spend More Time Thinking - Hexawise Tip: Using Value Expansions and Value Pairs to Handle Dependent Values - Maximize Test Coverage Efficiency And Minimize the Number of Tests Needed

By: John Hunter on Jun 24, 2013

Categories: Hexawise test case generating tool, Hexawise tips, Software Testing

Many teams are trying to generate unusually powerful and varied sets of software tests by using Design of Experiments-based methods to generate many or most of their tests. The two most popular software test design methods are orthogonal array testing and pairwise testing. This article describes how these two approaches are similar but different and suggests that in most cases, pairwise testing is preferable.

Before advancing, it may be worth pointing out that Orthogonal Array Testing is also known as OA or OATS. Similarly, pairwise testing is sometimes referred to as all pairs testing, allpairs testing, pair testing, pair-wise testing, or simply 2-way testing. The difference between these two very similar approaches of pairwise vs. orthogonal array is that orthogonal array-based solutions require the same coverage goal that pairwise solutions do (e.g., that every pair of inputs is tested at least once) plus an additional hurdle/characteristic, that there be a uniform distribution throughout the domain.

I have studied the question of how can software testing inputs be combined most efficiently and effectively pretty steadily for the last 7 years. I started by searching the web for "Design of Experiments" and "software testing" and found references to Dr. Madhav Phadke (who, by coincidence, turns out was a former student of my father).

  • I discovered that Dr. Phadke had designed RDExpert which, although it had been primarily created to help with Research & Design projects in manufacturing settings, could also be used to select small sets of powerful test sets in software testing projects, using the Orthogonal Array-based test selection criteria.

  • I used RDExpert to create test sets (and compared those test sets against sets of tests that had been selected manually by software testers)

  • I gathered results by asking one tester to execute the manually selected tests and another tester to execute the the Orthogonal Array-based tests; the OA-based tests dramatically outperformed the manually-selected ones in terms of defects found per tester hour and defexts found overall.

So, in short, I had confirmed to my satisfaction that an OA-based test data combination strategy was far more effective than manually selecting combinations for the kinds of projects I was working on, but I was curious if other techniques worked better.

 

After more study I have concluded that:

  • Pairwise is more efficient and effective than orthogonal arrays for software testing.

  • Orthogonal Arrays are more efficient and effective for manufacturing, and agriculture, and advertising, and many other settings.

 

And we have built Hexawise as a software tool to help software producers test their software, based on what I have learned from my experience. We take full advantage of the greatly increased efficiency and effectiveness of letting testers to determine what needs to be tested and software algorithms to quickly create comprehensive test plans that provide more coverage with dramatically fewer tests.

But we also go well beyond this to create a software as a service solution that aids the software testing team with many huge advantages such as: automatically generating Expected Results in test scripts, automated importing of data from Excel or mind maps, exporting tests into other tools, preventing impossible to test for values from appearing together, and much more.

 

Why is a pairwise testing strategy better than an orthogonal array strategy?

  • Pairwise testing almost always requires fewer tests than orthogonal array-based solutions (it is possible, in some situations, for them to have an equal number of tests).

  • Remember, the reason that orthogonal array-based solutions require more tests than a pairwise solution to reach the coverage goal of testing all pairs of test conditions together in at least one test is the additional hurdle/characteristic that orthogonal array testing has, e.g., that there be a uniform distribution throughout the domain.

  • The "cost" of the extra tests (AKA experiments) is worth paying in many settings outside of the software testing industry because the results are non-binary in those tests. Someone seeking a desired darkness and gloss and luminosity and luster for a particular shade of green in the processing of film, for example, would benefit from with the information obtained from the added information gathered from orthogonal arrays.

  • In software testing, however, the added costs imposed by the the extra tests are not worth it. You're generally not seeking some ideal point in a continuum; you're looking to see what two specific pieces of data will trigger a defect when they appear in the same transaction. To identify that binary approach most efficiently and effectively, what you want is a pairwise solution (with fewer tests), not a longer list of orthogonal array-based tests.

 

Let me also add these points.

  • First, unlike some of my other views on combinatorial test design, my opinion on this narrow subject is not based on multiple empirical studies; it is based on (a) the reasoning I laid out above, and (b) a dozen or so conversations I've had with PhD's who specialize in the intersection of "Design of Experiments" and software test design, and (c) anecdotal evidence from using both methods.

  • Secondly, to my knowledge,very few, if any, studies have gathered empirical data showing benefits of pairwise solutions vs. orthogonal array-based solutions in software testing scenarios.

  • Thirdly, I strongly suspect that if you asked Dr. Phadke, he would give you his reasons for why orthogonal array-based solutions are appropriate (and even preferable) to pairwise test case selection methods for certain kinds of software projects. I have a huge amount of respect for both him and his son.

 

Time doesn't allow me to get into this last point much now, but "mixed strength" tests are another even more powerful test design approach for you to be aware of as well. With mixed strength testing solutions, the test designer is able to select a default coverage strength for the entire plan (e.g., pairwise / AKA 2-way coverage) and, in the same set of tests, select certain high priority values to receive higher coverage strength (e.g., 4-way coverage strength selected for each "Credit Rating" and "Income" and "Loan Amount" and "Loan to Value Ratio" would give you a palm that achieved pairwise coverage for everything in the plan plus comprehensive coverage for every imaginable combination of values from those four high priority parameters. This approach allows you to focus on risk-based testing considerations.

 

Sorry if I got a bit long-winded. It's a topic I'm passionate about.

Originally posted on Stack Exchange, Additional note added after the first 3 comments were submitted:

@Hannibal, @Peter K., and @MichaelF, Thanks for your comments! If you'd like to read more about this stuff, I recommend the multiple links available through this "bundle of links" about pairwise testing and combinatorial testing. In particular, Michael Bolton's article on pairwise testing is directly relevant and very clearly written. It is one of the few introductory articles around that accurately describes the difference between orthogonal array-based solutions and pairwise solutions. If I remember correctly though, the example Michael uses is a rare exception to the rule; the OA solution has the same number of tests as an optimal pairwise solution does.

Related: The Empirical Evidence for Using Pairwise and Combinatorial Software Testing - 3 Strategies to Maximize Effectiveness of Your Tests - Hexawise TV

More than 100 Fortune 500 firms use Hexawise to design their software tests. While large companies pay six figures per year for enterprise licenses, Hexawise is available for free to schools, open source projects, other non-profits, and teams of up to 5 users from any kind of company. Sign up for your Hexawise account.

By: John Hunter and Justin Hunter on Jun 11, 2013

Categories: Combinatorial Testing, Design of Experiments, Efficiency, Multi-variate Testing, Pairwise Software Testing, Software Testing, Testing Strategies, Experimenting

The Expected Result feature in Hexawise provides 3 benefits in the software testing process. Specifically, it...

  • Saves you time documenting test scripts.

  • Makes it less likely for you to make a mistake when documenting Expected Results for your tests.

  • Makes maintaining sets of tests easy from release to release and/or in the face of requirements changes.

There are two different places you can insert auto-generated Expected Results into your tests: the "Auto-script" screen, and the "Requirements" screen. So which place should you use?

It may depend upon how many values are required to trigger the Expected Result you're documenting.

If you want to create an Expected Result that can be triggered by 0, 1, or 2 values, you can safely add your Expected Result using the Auto-script screen. If you're creating an Expected Result that can only be triggered when 3 or more specific values appear in the same test case, then - at least if you're creating a set of 2-way tests - you will probably want to add that particular Expected Result to the Requirements screen, not on the Auto-script screen. Why is that?

It's because if you use the Expected Result feature on the Auto-script screen, it is like telling the test generation engine "if you happen to see a test that matches the setting I provided then show this Expected Result text to the tester." This automates the process so the tester will be provided clear instructions on what to look for in cases that might otherwise be less clear to the tester. If the only way

Using the requirements feature is similar but a bit different. Using the requirements feature, in contrast, is like telling the test generation engine "make sure that the specific combination of values I'm specifying here definitely appear together in the set of tests at least one time (and, if I tell you that there's an expected result associated with that scenario, please be sure to include it)." The requirements feature is used when you not only want to provide an expected value for a specific test case situation but to require that the specific test case scenario be included in the generated test plan.

See our help page for more details: How to save test documentation time by automatically generating Expected Results in test scripts.

 

Related: How to Model and Test CRUD Functionality - 3 Strategies to Maximize Effectiveness of Your Tests

By: John Hunter on Jun 4, 2013

Categories: Hexawise test case generating tool, Hexawise tips, Software Testing

The test cases in a given test plan should be sufficiently similar and should not have wildly divergent paths depending on the value of parameters in a test case.

When you do find your test case flow diverges too much you often want to either break your test plan down into a few different test plans, so that you have a plan for each different kind of pass through the system.

Another similar approach is that you may want to decrease the scope of your test plan a bit so that you end up with test cases that are all similar in the plan.

Lastly, let's say the flows aren't wildly divergent, but only slightly so. As a silly example let's say you were testing a recipe that varied based on the fruit selected.

Fruit: Apple, Grape, Orange, Banana

And then you wanted a step for how the peeling was done.

Peeling: By hand, By manual peeling tool, By automated peeler Peeler type: Hand crank, Battery powered, AC powered

Now... our testing flow here has some divergence. Grapes and Apples don't get peeled in this recipe, so they never enter that flow. And Bananas are always peeled by hand so they only get a part of that flow. If this was just the tip of the iceberg of the divergence, we should create a test plan for Grapes and Apples and a different one for Oranges and Bananas.

But if this is the entire extent of the divergent flow, then we want to take advantage of N/A values and married and invalid pairs.

We update our parameter values for the peeling flow to include N/A.

Peeling: By hand, By manual peeling tool, By automated peeler, N/A Peeler type: Hand crank, Battery powered, AC powered, N/A

We marry Grape and Orange (uni-directionally) to the two N/A's so they don't participate in the peeling flow. We marry Banana (unidirectionally) to "By hand" and the 2nd N/A so it has a partial and circumscribed pass through the peeling flow.

Lastly we don't allow Orange to be paired with either N/A with an invalid pair.

That's how a slight flow variation can be accommodated. Please comment with any questions about any of these approaches to your problem.

 

Related: 3 Strategies to Maximize Effectiveness of Your Tests - How Not to Design Pairwise Software Tests - How to Model and Test CRUD Functionality

By: Sean Johnson on May 21, 2013

Categories: Hexawise tips, Scripted Software Testing, Software Testing, Testing Strategies

A client informed us that they had created (and used) approximately 3,500 test cases to test the search functionality of their application. They had a strong suspicions that (a) they should be able to test the search functionality of their application with fewer tests, (b) the tests they had accidentally omitted tests of many hundreds of plausible combinations of values that would be useful to test for (but did not know how to precisely identify were those gaps were without a huge amount of work), and that (c) many of these tests were quite inefficient in that they repeated many steps that had already been tested in other tests in the plan (even if they were not 100% duplicative of any other single test in the plan).

This client should have spoken to Lanette Creamer before they got into that situation. Lanette is a testing expert and blogger with ideas worth paying attention to. For example, her paper, Reducing Test Case Bloat, is well worth reading as is her blog.

There are times when what you cut may not be bloat. There are some situations where the decisions are the equivalent of “Do we cut off the arm or the head?” Well, a person can live without an arm. If you are in a situation where you are so time constrained that critical areas will be untested, you can still communicate the risk, be transparent and use a strategy to test the most important areas first. It is possible to plan for and do testing for a very time constrained project.

 

Of course avoiding this situation is best. Improving testing processes to use the best thoughts and tools is a better option. Cutting the bloat can allow resources to be applied to those areas they are really needed. Often though, people are scared of trying new ideas and cling to old methods, even if that results in the organization having to take increased risks by failing to test critical areas sufficiently. They are just more scared of trying new ideas than of getting away with saying we need more funding if you want more testing.

Lanette's article provides 8 specific suggestions for process improvements to reduce bloat. The first suggestion is to use combinatorial testing tools to greatly improve coverage while reducing workload. Another suggestion is to run the bloat reduction ideas by the stakeholders.

As part of your plan to reduce bloat, it can be helpful to state your assumptions about who is important and where you are placing testing priority and why. When reducing test case bloat you are taking a calculated risk. You are weighing the risk of being unable to test new features by insisting on testing every legacy case against the risk of purposefully not running some tests. When you share your starting assumptions with your stakeholders you offer them the chance to counter with their own assumptions and often you can clarify the boundaries of your testing this way to avoid gaps in testing or duplication.

 

See the full article for more good ideas on how to get better results for the existing testing resources available to your organization.

 

Related: Design of Experiments is about Learning ASAP and, in Software Testing, Finding Bugs ASAP - Pairwise and Combinatorial Software Testing in Agile Projects - Cem Kaner: Testing Checklists = Good / Testing Scripts = Bad?

By: John Hunter and Justin Hunter on Apr 24, 2013

Categories: Combinatorial Software Testing, Efficiency, Multi-variate Testing, Software Testing, Testing Strategies

The Hexawise Software Testing carnival focuses on sharing interesting and useful blog posts related to software testing.

 

  • Testing and Checking Refined by James Bach and Michael Bolton - "a robust role for tools in testing must be embraced. As we work toward a future of skilled, powerful, and efficient testing, this requires a careful attention to both the human side and the mechanical side of the testing equation. Tools can help us in many ways far beyond the automation of checks. But in this, they necessarily play a supporting role to skilled humans; and the unskilled use of tools may have terrible consequences."

  • Bugs Spread Disease by Elisabeth Hendrickson - "Cancel all the bug triage meetings; spend the time instead preventing and fixing defects. Test early and often to find the bugs earlier. Fix bugs as soon as they’re identified."

 

haija-sophia-istanbul

Haija Sophia in Istanbul, Turkey by Justin Hunter

 

  • Becoming a World-Class Tester by Ilari Henrik Aegerter - "World-class testing means following the urge to get to the bottom of things, not giving up until you have experienced enough value for yourself, thinking more expansively about the role of a tester, and thinking non-traditionally about what skills are required to thrive in the role."

  • Test Coverage Triage by Parimala Hariprasad - "My experience shows that a mind map based test design works great at this stage. Business teams will be thrilled to have a Visual Walkthrough of tests and provide inputs quickly. As a participant observer on several dozen IT projects, I have found out that testers’ personally walking them through the tests works really well."

  • What Does it Take to Change the Software Testing Industry? Courage! by Keith Klain - "According to Mark Twain, courage is not the absence of fear – but the mastery of it. There are people working in software testing all over the globe who are questioning long standing ways of working - some for the first time. Get yourself energized and get involved."

  • Explaining Exploratory Testing Relies On Good Notes. by Rob Lambert - "Being able to do good exploratory testing is one thing, being able to explain this testing (and the insights it brings) to the people that matter is a different set of skills. I believe many testers are ignoring and neglecting the communication side of their skills, and it’s a shame because it may be directly affecting the opportunities they have to do exploratory testing in the first place."

  • Human-Computer Cooperation by John Hunter - "people are better at figuring out interesting ideas to test. Once those are identified, those test conditions and other test ideas need to be combined together and put into tests. Generating a highly efficient, maximally varied, minimally repetitive set of tests based on a given set of test inputs is something computer algorithms are more effective at than a person."

  • Software testing can be sexy, too by Ole Lensmar - "What Klain, who serves on the board of the AST, and the others would like to do is not only bring those skills back home but increase the availability and accessibility of this kind of training and job opportunity. Ideally, colleges and universities would start offering majors in Software Testing so we can set young people on a path toward testing as a career."

  • Yet another future of testing post (YAFOTP) by Alan Page - "I think we’ll always need people to look at big end-to-end scenarios and determine how non-functional attributes (e.g. performance, privacy, usability, reliability, etc.) contribute to the user experience. Some of this evaluation will come from manually walking through scenarios), but there will be plenty of need for programmatic measurement and analysis as well..."

By: John Hunter on Apr 18, 2013

Categories: Software Testing

Attempting to assess the relative benefits of more than 200 software development practices is not for the faint of heart. Context-specific considerations run the risk of confounding the conclusions at every turn. Even so, Capers Jones, a software development expert with dozens of years of experience and nearly twenty books related to software development to his credit, recently attempted the task. He's literally devoted decades of his career to assessing such things for clients. We're quite pleased with how using Hexawise fared in the analysis.

Scoring and Evaluating Software Methods, Practices and Results by Capers Jones (Vice President and CTO, Namcook Analytics) provides some great idea on software project management. The article is based on the Software Engineering Best Practices with some new data is taken from The Economics of Software Quality (two of the books Capers Jones has authored).

Software development, maintenance, and software management have dozens of methodologies and hundreds of tools available that are beneficial. In addition, there are quite a few methods and practices that have been shown to be harmful, based on depositions and court documents in litigation for software project failures.

In order to evaluate the effectiveness or harm of these numerous and disparate factors, a simple scoring method has been developed. The scoring method runs from +10 for maximum benefits to -10 for maximum harm.

The scoring method is based on quality and productivity improvements or losses compared to a mid-point. The mid point is traditional waterfall development carried out by projects at about level 1 on the Software Engineering Institute capability maturity model (CMMI) using low-level programming languages. Methods and practices that improve on this mid point are assigned positive scores, while methods and practices that show declines are assigned negative scores.

The data for the scoring comes from observations among about 150 Fortune 500 companies, some 50 smaller companies, and 30 government organizations. Negative scores also include data from 15 lawsuits.

 

The article provides guidance, based on the results achieved by many, and varied, organizations with respect to software projects.

finding and fixing bugs is overall the most expensive activity in software development. Quality leads and productivity follows. Attempts to improve productivity without improving quality first are not effective.

 

This is an extremely important point for business managers to understand. Those involved in software development professionally don't find this surprising. But business people often greatly underestimate the costs of maintaining and updating software. The costs of bugs introduced by fairly minor feature requests to a system that doesn't have good software test coverage or test plans often create far more trouble than business managers expect.

This is especially true because there is a high correlation between software applications that have poor software testing processes (including poor test coverage and poor or completely missing test plans) and those application that were designed without long term maintenance in mind. Both deficiencies result of decisions made to minimize initial development costs and time. They both show a lack of appreciation for wise software engineering practices and software application project management.

The article discusses a complicating factor for accessing the most effective software development practices: the extremely wide differences in software engineering scope. Projects range from simple applications one software developer can create in a short period of time to massive application requiring thousands of developer-years or effort.

In order to be considered a “best practice” a method or tool has to have some quantitative proof that it actually provides value in terms of quality improvement, productivity improvement, maintainability improvement, or some other tangible factors.

Looking at the situation from the other end, there are also methods, practices, and social issues have demonstrated that they are harmful and should always be avoided. ... Although the author’s book Software Engineering Best Practices dealt with methods and practices by size and by type, it might be of interest to show the complete range of factors ranked in descending order, with the ones having the widest and most convincing proof of usefulness at the top of the list. Table 2 lists a total of 220 methodologies, practices, and social issues that have an impact on software applications and projects.

The average scores shown in table 2 are actually based on the composite average of six separate evaluations:

  1. Small applications < 1000 function points

  2. Medium applications between 1000 and 10,000 function points

  3. Large applications > 10,000 function points

  4. Information technology and web applications

  5. Commercial, systems, and embedded applications

  6. Government and military applications

The data for the scoring comes from observations among about 150 Fortune 500 companies, some 50 smaller companies, and 30 government organizations and around 13,000 total projects. Negative scores also include data from 15 lawsuits.

The scoring method does not have high precision and the placement is somewhat subjective.

Top 10 tools and practices listed in the article:

Practice Score
1. Reusability (> 85% zero-defect materials) 9.65
2. Requirements patterns - InteGreat 9.50
3. Defect potentials < 3.00 per function point 9.35
4. Requirements modeling (T-VEC) 9.33
5. Defect removal efficiency > 95% 9.32
6. Personal Software Process (PSP) 9.25
7. Team Software Process (TSP) 9.18
8. Automated static analysis - code 9.17
8. Mathematical test case design (Hexawise) 9.17
10. Inspections (code) 9.15

 

We are obviously thrilled that Hexawise is listed. We have seen the value our customers have achieved using mathematical based combinatorial software test plans (see several Hexawise case studies). It is great to see that value recognized in comparison to other software development practices and judged to be of such high value to software development projects.

The article makes it clear the importance of the results is not "the precision of the rankings, which are somewhat subjective, but in the ability of the simple scoring method to show the overall sweep of many disparate topics using a single scale."

The methodology behind the results shown in the article can be used to evaluate your organization's software development practice and determine opportunities for improvement. But, as stated above, software projects cover a huge range of scopes. The specific software project needs will drive which practices are most critical to achieving success for a specific project. The list in the article, of what practices have provided huge value and what practices have resulted great harm, is a very helpful resource but project managers and software developers and testers need to apply their judgement to the information the article provides in order to achieve success.

A leading company will deploy methods that, when summed, total to more than 250 and average more than 5.5. Lagging organizations and lagging projects will sum to less than 100 and average below 4.0.

 

The use of Hexawise has been growing; that has helped increase the number of software projects using best practices (that score 9, or higher), however as the article states there is quite a need for improvement.

From data and observations on the usage patterns of software methods and practices, it is distressing to note that practices in the harmful or worst set are actually found on about 65% of U.S. Software projects as noted when doing assessments. Conversely, best practices that score 9 or higher have only been noted on about 14% of U.S. Software projects. It is no wonder that failures far outnumber successes for large software applications!

 

A score of 9 to 10 for a practice means that practice results 20-30% improvement in quality and productivity of software projects.

Conclusion: while your individual mileage may vary, this report provides further evidence that using Hexawise really does lead to large, measurable improvements in efficiency and effectiveness.

We are very proud of the success of Hexawise thus far; as a new year starts we see huge potential to help many organizations improve their software development efforts.

The article includes a list of references and suggested readings that is valuable. Included in that list are:

DeMarco, Tom; Controlling Software Projects, 1986, 296 pages.

Gilb, Tom and Graham, Dorothy; Software Inspections, 1994, 496 pages.

Jones, Capers; Applied Software Measurement, 3rd edition, 2008, 662 pages.

McConnell, Code Complete, (I'm linking to the 2nd edition the article references the 1st edition) 2004, 960 pages.

 

Related: Maximizing Software Tester Value by Letting Them Spend More Time Thinking - A Powerful Software Test Design Approach - 3 Strategies to Maximize Effectiveness of Your Tests

By: John Hunter on Mar 18, 2013

Categories: Software Testing, Software Testing Efficiency, Testing Case Studies, Testing Strategies

I strongly agree with Cem Kaner's statement (in Inefficiency and Ineffectiveness of Software Testing: A Key Problem in Software Engineering) that: sometimes tests uncover defects that don’t fit within any coverage model because they are side effects of the tests rather than explicitly planned foci of the tests."

My experience indicates that an effective way to increase the likelihood that you will trigger such defects (without explicitly looking for them) is to try to maximize the variation between each test case you execute.

A case in point: when I sat down to dinner with James Bach a year or so ago in Boston at a testing conference, he gave me a quick testing challenge (as he is fond of doing with testers he meets for the fist time to see how we think). He asked how I would test a very simple calendar entry application that allowed users to record the start and end times of diary events. Key inputs to use as test conditions for these tests included start times and end times.

I proposed a set of times to try that were designed to provide as much variety as possible from one test case to the next. As inputs into the start and end times, I used a small number of different times spread throughout the morning, afternoon, and evening as well. The strategy I used quickly identified the testing defect the puzzle was designed to uncover in a small handful of tests. What was most memorable about the experience from my perspective was not that I "succeeded" in triggering the bug but that the tests I created triggered a type of bug that was, in Kaner's words, a "side effect of the tests rather than explicitly planned foci of the tests."

The business logic in the calendar application that should have identified invalid beginning and end time combinations was coded incorrectly. Instead of using numbers in the business logic, the business logic was ordering the numbers alphabetically. I was not consciously looking to identify that kind of a flaw in the business logic, but by maximizing the variation from test case to test case, I maximized my odds of finding it.

Efficiently achieving structured variation is difficult because it is hard for a human brain to remember whether dozens of different test conditions have been tested together (or we're accidentally repeating ourselves). This is where pairwise and combinatorial test case generating tools like our Hexawise tool come in. They are designed to achieve as much variation from test case to test case as possible. One of the relatively unsung benefits of this approach is that doing so will help find bugs, like these, that you aren't even consciously looking for.

 

Related: 3 Strategies to Maximize Effectiveness of Your Tests - Getting Started with a Test Plan When Faced with a Combinatorial Explosion - Book Review of “Explore It!” Elisabeth Hendrickson’s Excellent New Book on Software Testing

By: Justin Hunter on Feb 26, 2013

Categories: Software Testing, Testing Strategies

Malcolm Gladwell's three best-selling books are filled with fascinating examples of all sorts of things to underscore the interesting points he makes about a wide variety of topics. I'm currently readying his book "Blink," about how people make process information and make decisions in the 'blink of an eye."

The example he shares about an overcrowded hospital Emergency Room's decision making process really resonated with me. Brendan Reilly, named the chairman of Medicine at Cook's County Hospital in Chicago in 1996, was in dire need of a way to improve the conditions of the hospital when he showed up there. The Emergency Room faced extreme amounts of resource pressure in particular; there were too many patients needing care and too few doctors and beds to serve them all promptly. One of the most resource-sapping processes was the treatment of patients who entered the ER complaining of chest pains.

"From the beginning, the question of how to deal with heart attacks was front and center." About 30 people a day came into the ER "were worried that they were having a heart attack. And those thirty used more than their share of beds and nurses and doctors and stayed around a lot longer than other patients."

"Reilly's first act was to turn to the work of a cardiologist named Lee Goldman. In the 1970's, Goldman...p. 133

He... announced that he was holding a bake-off. For the first few months, the staff would use their own judgment in evaluating chest pain, the way they always had. Then they would use Goldman's algorithm, and the diagnosis and outcome of every patient treated under the two systems would be compared.

For two years, data were collected, and in the end, the result wasn't even close. Goldman's rule won hands down in two directions; it was a whopping 70 percent better than the old method at recognizing patients who weren't actually having a heart attack. at the same time, it was safer. the whole point of chest pain prediction is to make sure that patients end up having major complications are assigned right away to the coronary and intermediate units. Left to their own devices, the doctors guessed right on the most serious patients somewhere between 75 and 89 percent of the time. The algorithm guessed right more than 95 percent of the time.... He went to the ED and changed the rules."

 

I suspect that if you asked 100 cardiologists whether they would perform more accurate assessments that the 3 question Goldman algorithm would, you would find that all 100 of those cardiologists would feel confident in their ability to outperform the 3 question algorithm. After all, the algorithm didn't allow for subject matter expertise.

 

How is this Relevant to Software Testing?

When many business leaders think of software testing what they want is something that catches bugs before the product is released to the customer. To some extent this can be automated. There are many previous bugs (and experience with bugs in other software) that allow us to create specific tests which will identify known bugs.

So if we have a form on the web application, we can test various scenarios and make sure the form is processed properly: submitting the form when it should or displaying the proper error message when it should do that. It is wise to have test plans to check for these bugs.

Creating simple easy to follow guidelines to address known issues is a wise action. This is true for creating a simple checklist to use in screening for heart attacks and for creating test plans that cover the known issues.

While those are wise they are by no means sufficient. Physicians bring a great deal of expertise to evaluating patients for a huge number of symptoms. And they can use their knowledge and experience to judge where they need more evidence (for example, a diagnostic x-ray or an analysis of a blood sample) to make a judgement. And to judge is software will work as users wish simply following a test plan robotically is not sufficient. The experience and expertise of a software tester allows them to probe the software applications and spot not only bugs but weaknesses that should be addressed so users of of the software have the best experience.

The appropriate methods depend upon the need. And sometimes the expert is not as effective as simply using tools or a check sheet that has been developed specifically for that purpose. It is important to design processes in your organization to find the most effective strategies given the local circumstances.

Experimenting, by using evidence based management practices, is needed to determine what is most effective. Assuming that the best solution is always relying on the experts is not accurate (as the example Gladwell used shows). A great tool to aid in determining what practices work best is the Plan-Do-Study-Act cycle made famous by Dr. W. Edwards Deming.

 

Related: Maximizing Software Tester Value by Letting Them Spend More Time Thinking - Mistake Proofing Strategies for Software Code - Checklists in Software Development

By: John Hunter and Justin Hunter on Feb 18, 2013

Categories: Software Testing, Testing Strategies

I am passionate about pairwise software testing techniques. I have helped dozens of teams, for example, carefully measure the benefits that can be created when teams of testers adopt pairwise and related combinatorial testing approaches to identify the test cases they will execute (as compared to manual test case identification methods). What usually happens is that tester productivity doubles. (See Combinatorial Software Testing - pdf download).

I believe these approaches will be much more widely adopted in a few years than they are now for the simple reason that they consistently deliver dramatic benefits to both the speed of software test design and the efficiency and thoroughness of software test execution. As more teams try these methods for themselves, and measures the benefits they achieve with them, broader adoption seems highly likely to me.*

I see three main barriers to broader adoption by the testing community at large:

  1. The first barrier is that testers will not make an attempt to apply this method to their testing projects so they will never find out how effective it is. The second is that ill-informed testers will try to apply the approach but do such a poor job at implementation that they do not generate benefits.

  2. The second barrier is that even testers who use the approach effectively a few times, will not realize how much more effective it is making them. A dismissive thought process guilty of this might sound something like this: "Those 11 bugs I just found? Yeah. I found them because I'm a good tester; the fact that I happened to use pairwise tests just now? That's largely irrelevant. I'm sure I would have found them regardless.")

  3. The third barrier is that testers unfamiliar with the basics of pairwise testing principles will design test cases without thinking about what they are doing, and achieve "garbage in / garbage out" results. The benefits that would have been so easily achieved in the testing project - like Lindsey Jacobellis' opportunity to win a gold medal for Snow Boarding - disappear in a groan-worthy moment of bone-headed stupidity.

 

 

This blog post addresses this third barrier. When testers sabotage their own test plans with a poor choice of inputs, they may well blame the test design strategy rather than themselves, which would be unfortunate. Here's one common problem I see (exaggerated a bit in this example to make my point).

 

Objective: create a set of tests that will check to see if the underwriting engine for a car insurance firm is calculating premium estimates correctly.

Our aspiring pairwise test designer enters stage left and identifies a set of parameters:

First Name, Last Name, Age of Primary Driver, Credit Score, Number of Cars, Number of Accidents, Number of Speeding Tickets, and Number of Additional Drivers

So far so good. We now have the initial ingredients for a thing of beauty; we have a set of parameters that could quickly result in a combinatorial explosion of possibilities and, ready to save the day, we have a test designer who has correctly identified this as an opportunity to achieve efficiency and thoroughness benefits through the application of pairwise testing methods. Our potential hero is a couple minutes away from creating a concise set of tests that will confirm not only confirm that each of the data points in the plan work as they should but that they work as they should in combination with each of the other data points in the test plan.

In other words, the plan will not only confirm that "Number of Accidents = 3" will impact premiums as it should on its own, but also that "Number of Accidents = 3" will work as it should when tested in combination with the other relevant inputs in the application, e.g.,: 3 accidents with every relevant input for "Age of Primary Driver," 3 accidents with every relevant input for "Credit Score," 3 accidents with every relevant input for "Number of Cars," 3 accidents with every relevant combination for "Number of Speeding Tickets," and 3 accidents with every relevant input for "Number of Drivers."

He's seen the Promised Land of improved efficiency and effectiveness and he's ready to enter. Unfortunately, with his next move, he demonstrates he's a doofus. Entry to Promised Land denied. Check out the values he chose to enter for each of his parameters.

 

hexawise-screenshot

Notice anything wrong here?

 

Just for fun, let's take a close up look at Lindey's disastrous Snow Boarding maneuver here.

 

... and let's break down our shame-faced test designer's bone-headed move here. Can you notice what is wrong in with his choices of values?

There are nine different parameters in the mix here. Of those, two ("First Name" and "Second Name"), are the least important to our current objective of looking for problems in the underwriting engine calculations. And yet...

He's added ten values to each of them. Oops! Whenever you are putting together a pairwise (or 2-way) test plan, the number of tests required will never be lower than the product of the number of parameter values from the two parameters that have the highest number of values. In plain English, that high-falutin' previous sentence means: when you have a plan with 7 parameters that have a maximum of 4 values each, "10 largely irrelevant values X 10 largely irrelevant values = you're a big fat idiot" because you'll create a test plan that has 100 test cases (as compared to a test plan that could have covered the System Under Test more effectively with fewer than a quarter of the tests you've just created).

 

For more information on pairwise and combinatorial testing, I would recommend the following sources:

 

If you are attempting to use pairwise and/or combinatorial testing methods and running into questions, I'd sincerely like to help. Please consider one or more of the following:

 

Thank you,

Justin Hunter

 

*The manufacturing industry followed a similar pattern of adoption to similar methods that consistently delivered dramatic efficiency and effectiveness benefits. It took decades before multi-variate Design of Experiments methods were widely adopted by manufacturers even long after the benefits were proven to be dramatic and repeatable to anyone who would look at the clear, unambiguous, objectively-measurable evidence. Today, it is impossible to find a Fortune 500 manufacturing firm that does not regularly use multi-variate Design of Experiments in their manufacturing processes. One day it will be the same for Fortune 500 firms with respect to their adoption of multi-variate Design of Experiments methods of software testing.

By: Justin Hunter on Jan 29, 2013

Categories: Combinatorial Testing, Pairwise Software Testing, Software Testing, Software Testing Efficiency, Testing Strategies

This is the second edition of our carnival that focuses on finding interesting and useful blog posts related to software testing.

 

  • Testing != test execution by Jeff Fry - "We often talk about testing as if it’s only test execution, yet often the most interesting, challenging, skill-intensive aspects of testing are in creating a mental model that helps us understand the problem space, designing tests to quickly and effectively answer key questions, analyzing what specifically the problem is, and communicating it effectively."

  • Test Mercenaries by Mike Bland - "good testing practice goes a long way towards finding and killing a lot of bugs before they can grow more expensive, and possibly multiply. The bugs that manage to pass through a healthy testing regimen are usually only the really interesting ones. Few things are less worthy of our intellectual prowess than debugging a production crash to find a head-slappingly stupid bug that a straightforward unit or integration test could’ve caught."

 

yellow cameleon

Yellow Cameleon in South Africa by Justin Hunter

 

  • The Oracle Problem and the Teaching of Software Testing by Cem Kaner - "I’ve been emphasizing the oracle problem in my testing courses for about a dozen years. I see this as one of the defining problems of software testing: one of the reasons that skilled testing is a complex cognitive activity rather than a routine activity. Most of the time, I start my courses with a survey of the fundamental challenges of software testing, including an extended discussion of the oracle problem."

  • The new V-Model by Kim Ming Leung - "We design by specifying the measurements before coding but not writing the test before coding. We write code and invite user feedback before writing automated testing for these measurements. Code quality is still guaranteed because first, measurement is the design and second, we code with testing in mind (i.e. write testable code)."

  • Maximizing Software Tester Value by Letting Them Spend More Time Thinking by John Hunter - "Hexawise also lets the software tester easily tune the test coverage based on what is most important. Certain factors can be emphasized, others can be de-emphasised. Knowledge is needed to decide what factors are most important, but after that designing a test plan based on that knowledge shouldn’t take up staff time, good software can take care of that time consuming and difficult task."

  • Improving the State of your Testing Team: Part One – Values by Keith Klain - "Typically, the first thing out of my test teams mouths when asked “how can we improve the state of testing here”, usually relates to something that other people should do. Very few people or teams take an introspective based approach to improvement, or state their management values, but the ones that do, typically have great success.

  • Interview and Book Review: How Google Tests Software by Craig Smith - "stop treating development and test as separate disciplines. Testing and development go hand in hand. Code a little and test what you built. Then code some more and test some more. Test isn’t a separate practice; it’s part and parcel of the development process itself. Quality is not equal to test. Quality is achieved by putting development and testing into a blender and mixing them until one is indistinguishable from the other."

  • Introduction to test strategy (review of Rikard Edgren's presentation) by Mauri Edo - "Rikard referred to a test strategy as the outcome of the gathered and shared information plus our own thinking processes, encouraging us to find strategies for our context, learning to understand what is important for us, in our situation."

  • Experience report EuroSTAR Testlab 2012 by Martin Jansson - "The testlab is very much alike our every day life as testers. We prepare and plan for many things, but when reality hits us our preparations might be in vain. Therefore it is important in being prepared for the unknown and the unexpected. Working with the testlab is a great way of practising that."

 

Related: Software Testing Carnival #1 - Hexawise TV blog

By: John Hunter on Jan 21, 2013

Categories: Software Testing

Screen-Shot-2013-01-15-at-7.28.42-PM-250x300

 

Elisabeth Hendrickson's new book, Explore It!, will begin shipping from Amazon in a week. If you're interested in software testing, I highly recommend it without reservation. It's outstanding. It is currently available for sale on Prag Prog and for pre-order on Amazon. The paper version will be published on January 22nd. Since Amazon apparently doesn't allow people to review books until they officially go on sale, I can't yet post my review on Amazon, but here, one week early, is my glowing review:

Explore It! is one of the very best software testing books ever written. It is packed with great ideas and Elisabeth Hendrickson's writing style makes it very enjoyable to read. Elisabeth Hendrickson has a well-deserved global reputation in the software testing community as someone who has the enviable ability to clearly communicate highly-practical, well-thought-out ideas. Tens of thousands of software testers who have already read her "Test Heuristics Cheat Sheet" no doubt already appreciate her uncanny ability to clearly convey an impressive number of actionable ideas with a minimal use of ink and paper. A pdf download of the cheat sheet is available here. If you're impressed by how much useful stuff Hendrickson can pack into one double-sided sheet of paper, you should see what she can do with 160 pages.

Testers at all levels of experience will benefit from this book. Like the best TED talks, Explore It! contains advanced ideas, yet those ideas are presented in way that is both interesting and accessible to a broad audience. Beginning testers will benefit from learning about the fundamentals of Exploratory Testing (an important and incredibly useful approach to software testing that is increasingly getting the respect it deserves). Experienced testers will benefit from practical insights, frameworks for thinking about challenges that bedevil all of us, and Hendrickson's unmatched ability to clearly explain important aspects of testing (including her superb explanations of test design principles).

Chapter 4 "Find Interesting Variations" in itself is worth far more than the price of the book. It is my favorite chapter in any software testing book I have ever read. A large part of the reason I have so much appreciation for this chapter is that I have personally been teaching software testers how to create interesting variations in their testing efforts for the last six years and know from experience that it can be a challenging topic to explain. I was excited to see how thoroughly Hendrickson covered this important topic because relatively few software testing books address it. I was humbled by how effortlessly Hendrickson seemed to make this complex topic easy to understand.

Buy it. You won't regret it. I'm buying multiple copies to give to developers and testers at my company as well as multiple copies to give to our clients.

  • Justin Hunter

By: Justin Hunter on Jan 15, 2013

Categories: Book Review, Exploratory Testing, Software Testing, Testing Checklists, Testing Strategies, Training

A combinatorial explosion is when the configuration settings and user actions and data entered etc. makes it impossible to test everything. The number of tests required to individually test every single possibility is many thousands of times greater than could realistically be tested.

When faced with taking over an existing software applications without a good test suite (or any test plan) often is daunting. And the problems of creating an unfathomable number of tests face you due to combinatorial explosion. Hexawise is a software as a service that aids in dealing with this dilemma for software testers. Software test plans are created that provide far better coverage than is seen in practice with a tiny fraction of the test required for complete combinatorial coverage (that is testing every possible combination [pairwise or 3, 4, 5... way] individually).

The Google Maps test plan provides a good example of combinatorial explosion faced by the testers (in this case, those who tested Google Maps). Take a look at the Google Maps test plan by login to your Hexawise account (creating a demo account is free and simple). The Google Maps test plan is one of 9 samples currently provided in Hexawise.

For creating your own test plan, while you are exploring the software application and testing it out to find "where the weak points are," you will probably find it useful to vary things as much as possible, repeat your actions as little as possible. Those points are true whether you're doing relatively informal lightly documented exploratory testing or more heavily documented test scripts. It addition, since a large percentage of defects can be triggered by the interaction of just two test inputs, it would be nice, if you had time, to test every single possible combination involving two test inputs; that's the rationale behind allpairs, pairwise and orthogonal array-based test case prioritization methods.

To recreate a similar - very early draft - plan for yourself, I'd suggest going through the following steps to put together a relatively small number of highly informative end-to-end-ish tests:

Ask what can change as users go through the system? Think about configuration settings, user actions, data formats, data ranges, etc. even throw in more "creative" ideas like user personas. Let your creativity and common sense guide you. Enter those in as parameters.

Ask how those parameters can change? (for the parameter "Browser" enter IE7, IE8, FF, etc.) Put those in as values under each parameter (entering constraints as required)

Ask does that variation matter? When possible (when it doesn't matter as much) use equivalence classes and be biased towards fewer values - at least for your early draft tests.

Ask what special paths thorough the system do you want to be sure to include? (Most common happy path, paths to trigger certain business rules, etc.)

Click the Create Tests button in Hexawise and you'll instantly get a very nice draft starter set of highly varied tests. If they look like they're relatively interesting and don't miss hugely important things, start informally executing them and you'll be sure to learn some more things as you do about the system's weak points that would result in you going back to those draft tests and iterating them to make them stronger and cover more.

To get a bit more on using this approach see our case studies. Hexawise TV provides narrated videos online showing how to make your life easier as a software tester.

 

Related: Hexawise Tip: Using Value Expansions and Value Pairs to Handle Dependent Values - 3 Strategies to Maximize Effectiveness of Your Tests - Pairwise and Combinatorial Software Testing in Agile Projects

By: John Hunter and Justin Hunter on Jan 2, 2013

Categories: Combinatorial Software Testing, Exploratory Testing, Multi-variate Testing, Pairwise Software Testing, Software Testing, Testing Strategies

Creating test plans for create, read, update and delete (CRUD) functionality is a very common requirement. There are a few different ways to model it. Let's review the simplest solution first then move to a few optional extra ideas that you could use in addition.

Option 1: Only one C, R, U, D action tested in each of your tests

Have one Parameter called "CRUD action" - have 4 different values: Create, Read, Update, Delete

Option 2: Include two CRUD actions in (most of) your tests

Create 2 Parameters, each with the following 4 and 3 Values:

First CRUD action: Create, Read, Update, Delete
Second CRUD action: Read, Update, Delete

You may want to add an invalid pair with the first CRUD action = Delete and each of the three values in the second CRUD action (in which case you'd need to add a 4th Value called N/A and leave it as the only available option for scenarios that deleted a record in the first test.

Option 3: Up to 4 CRUD actions in each of your tests

  1. Create a new record: create, don't create
  2. Read a record: read, don't read
  3. Update a record: update, don't updat
  4. Delete a record: delete, don't delete

That's the most basic modeling decision. (e.g., one CRUD action / test, two CRUD actions / test, or potentially 4 CRUD actions / test).

What other things might be interesting to vary?

You might want to think about "newspaper reporter questions" - e.g., (who, what, when, where, why? how? how much?)

 

Who?

Which user type (or which system) is performing the CRUD ACTION? - Are certain types of users or systems not allowed to perform certain actions? Include test inputs to make sure they can't. Are others allowed to? Include test inputs in your model to make sure they can.

 

What?

What kind of CRUD actions are being made? Valid ones? Invalid ones? How many invalid types of actions can you think of? What is supposed to happen when special characters are entered? What is supposed to happen when blanks are entered? What is supposed to happen when extremely long names are entered (e.g., are there concatenation rules?)

 

When?

When are the updates being made? Are there any system downtimes? What happens to CRUD actions that are attempted then? What happens if the time stamp on a CRUD action is from the future? From the past? Originates in a different time zone?

 

When? ("version 2")

When are the CRUD actions tested in relation to one another? e.g., Update a record and then try to read it or read it and then try to update it? If you wanted to "mix up the order" of the Option 3 actions, for example, you could add a parameter called perform CRUD actions in what order? with values such as normal create then read then update then delete order, the reverse of the normal order as much as possible, start with read or update. This would not give you a "perfect solution" to cover every timing combination but it would lead to additional variability within your tests if you wanted them.

 

Why?

Why is a CRUD action being conducted? Do any interesting testing ideas come to mind when you think about this angle? How about bad or malicious motivations?

 

How?

How are the actions being undertaken? By a keyboard? By a mouse? By a batch processing system?

Lastly, if you have an account you created on Hexawise.com, check out sample test plan "08. Modifications to a Database" for some additional ideas.

Bottom line: when test designers create tests (whether they use Hexawise to generate their tests or whether they select and document their tests by hand), they will have a lot of discretion over what kinds ideas to include / exclude from their tests. Using Hexawise offers these advantages over selecting tests by hand:

  1. You will maximize variation in your tests
  2. You will minimize wasteful repetition in your tests
  3. You will achieve 100% coverage of the combinations you tell the tool to include in your solution (e.g., pairwise testing coverage, three-way coverage, risk based testing coverage, etc.)
  4. You will be able to select those combinations and documented tester instructions faster and with fewer errors in the documented tests.

This post is based on a user question. See the Hexawise user forum page for some additional details related to the users follow up questions.

 

Related: Efficient and Effective Test Design - 3 Strategies to Maximize Effectiveness of Your Tests - Maximize Test Coverage Efficiency And Minimize the Number of Tests Needed

By: John Hunter and Justin Hunter on Dec 18, 2012

Categories: Pairwise Software Testing, Software Testing, Testing Strategies

I have started focusing on the Hexawise blog recently. For a good part of this year we have not had much activity on the blog, but we plan to make this blog more active going forward. As part of that effort we are starting a Software Testing Blog Carnival to highlight some post on software testing that I find interesting and think others will too. Enjoy,

 

 

water buffalo south africa

Water buffalo, Timbavati Game Preserve, South Africa by Justin Hunter

 

  • Bugs Spread Disease by Elisabeth Hendrickson - "It wasn’t the bugs that killed us directly. Rather, the bugs became a pervasive infection. They hampered us, slowing down our productivity. Eventually we were paralyzed, unable to make even tiny changes safely or quickly."

  • A Sticky Situation by Michael Bolton - on using agile, kanban style, system for managing the software testing workload: "The whiteboard was the center of an active discussion between programmers and project managers about the project status. After the meeting, the whiteboard and the notes on it remained as a kind of information radiator. 'I suddenly realized that if they could do that, I could too,' Paul said. He began by dividing the whiteboard into three columns: To Be Done, Work in Progress, and Done."

  • A Quick Testing Challenge by Alan Page and a response, Angry Weasel Challenge, a presentation by to Figure out why TheApp.exe won't load and cause it to load by solving the problem.

  • 3 Strategies to Maximize Effectiveness of Your Tests by John Hunter - "Use the MECE principle. The MECE principles states you should define your values in a way that makes each of them “Mutually Exclusive” from the others in the list (no subsets should represent any other subsets, no overlaps) and “Collectively Exhaustive” as a group (the set of all subsets, taken together, should fully encompass all items, no gaps)."

  • Musings on Test Design by Alan Page - "The point is, and this is worth repeating so often, is that thinking of test automation and “human” testing as separate activities is one of the biggest sins I see in software testing today. It’s not only ineffective, but I think this sort of thinking is a detriment to the advancement of software testing."

  • Creating a Shared Understanding of Testing Culture on a Social Coding Site by Leif Singer - "those learning testing in this environment write tests to make sure that a program behaves a certain way, and forget that they might also need to test how it should not behave (e.g. on invalid input)."

  • Testing in Scrum with a Waterfall Interaction by Davide Noaro - "Testing each user story separately is, for me, the basis of the Agile process, even in an interaction with a Waterfall-at-end scenario like the one described. Integrating testing into the process itself is something we should do for any software development process, not only in Agile or Scrum."

By: John Hunter on Oct 31, 2012

Categories: Software Testing

Hexawise includes an array of sample plans when a new user account is created. These provide concrete examples of how to categorize items when creating a combinatorial test plans (also called pairwise test plans, orthogonal array test plans, etc.). Once you [sign in to your Hexawise account](http://hexawise.com/ (or setup a new, free, account) looking at this [sample test plan](https://app.hexawise.com/share/HT3UG7M8 (which is similar to the situation raised in the question that follows), might be useful.

Within your Hexawise account you can copy the sample test plans that you are provided with and then make adjustments to them. This lets you quickly see what effects changes you make have on real test plans. And it also lets you see how easy it is to adjust as changes in priorities are made, or gaps are found in the existing test plan.

 

A Hexawise user sent us the following question.

What is the recommended approach to configuring parameter with one or more values?

I have two parameters which are related.

If Parameter 1 = Yes, Parameter 2 allows the user to select one or more values out of a list of 25 - most of which are not equivalent.

For Parameter 2, is the recommended approach to handle this to create separate parameters each with a yes/no value? i.e. create one parameter for each non-equivalent value, and one parameter for the equivalent values. Then link each of these as a married pair to Parameter 1.

I'm open to suggestions as to alternatives.

Here's the screen in question. Parameter 1 = "Pilot", Parameter 2 = checkboxes for types of plans.

aviation question inline

Great question.

I would recommend that you use different parameters for each option (e.g., "Scheduled Commercial" as a parameter with "Selected, Not Selected" as your Values associated with it).

Also, I'd recommend following these 3 strategies to maximize the effectiveness of your tests.

First, consider using adjusted weightings. You may find it useful to weight certain values multiple times, e.g., have 4 values such as "Select, Do Not Select, Do Not Select, Do Not Select" to create 3 times as many tests with "Do Not Select" as "Select."

Second, use the MECE principle. The MECE principles states you should define your Values in a way that makes each of them "Mutually Exclusive" from the others in the list (no subsets should represent any other subsets, no overlaps) and "Collectively Exhaustive" as a group (the set of all subsets, taken together, should fully encompass all items, no gaps)

Third, avoid "ands" in your value names. As a general rule it is unwise to define values like "Old and Male" or "Young and Female", etc. A better strategy is to break those ideas into two separate Parameters, like so:

First Parameter = "Age" --- Values for "Age" = Old / Young

Second Parameter = "Gender" --- Values for "Gender" = "Male / Female"

 

Related: Efficient and Effective Test Design - Context-Driven Usability Considerations, and Wireframing - Why isn't Software Testing Performed as Efficiently and Effecively as it could be?

By: John Hunter on Oct 25, 2012

Categories: Efficiency, Hexawise tips, Pairwise Software Testing, Software Testing, Software Testing Efficiency, Testing Strategies

When we met with Hexawise users in India, we noticed that the page load response times for Hexawise users there were sometimes significantly slower than in the United States due to sporadic internet connectivity issues. One area that troubled us in particular was the extra few seconds it could take to enter test inputs into plans.

We are constantly looking for ways to make our software test case generating tool better and came up with a solution.

Hexawise Bulk Add Instructions 1 inline

Hexawise Bulk Add Instructions 2 inline

 

Even with a great internet connection this feature lets you be more productive, so if you have not tried out the bulk add feature, give it a try today.

Hexawise: More Coverage. Fewer Tests

 

Related: Hexawise Tip: Using Value Expansions and Value Pairs to Handle Dependent Values - Maximize Test Coverage Efficiency And Minimize the Number of Tests Needed - 13 Great Questions To Ask Software Testing Tool Vendors

By: John Hunter on Oct 23, 2012

Categories: Hexawise test case generating tool, Hexawise tips, Software Testing

We have thousands of users and we're growing by hundreds of new users a month. We don't have a way to accurately breakdown our users into Agile vs Waterfall users but I know from conversations with Hexawise users that many users are using Hexawise with great results on Agile projects.

Agile projects involve short sprints in which new features are added to existing functionality. These sprints are often a couple weeks long. In such situations, it is important to be able to create tests quickly. Hexawise is extremely good at accomplishing exactly that.

Let me explain by way of an example. A Hexawise user named Sandeep asked how Hexawise could be used to create tests for an insurance ratings engine. I was able to create a sample set of 37 2-way tests within 20 minutes.

Now lets imagine that a sprint creates 2 new features that we'd like to test in combination with the existing test inputs in the existing plan. Let's assume "Feature 1" is a new ability of the application to handle input coming in from three sources: (1) web, (2) agent, and (3) call center. Let's assume "Feature 2" is a new ability of the application to handle Group Discounts. The Group Discount parameter will have 3 categories of values as well: (N/A) for people who aren't able to claim them, "Private company," and "Government / Public."

Added the new features, and created another, completely different set of tests that achieve 100% 2-way coverage. Each value in the new feature is tested in at least one test with every other value in the plan. It took less than 4 minutes to create this new set of tests incorporating the new features. It is an extremely attractive benefit of using Hexawise that is excellent for both Waterfall projects (which inevitably have late-breaking requirements changes) as well as Agile projects (where late-breaking feature additions are expected to happen as part of the process).

You want to keep testing documentation light, you want to minimize the amount of time you spend in selecting and documenting tests, and it is helpful to achieve as much coverage in as little time as possible and minimize the risk that you'll accidentally forget to test things that you meant to test (which can be easier to do if you have light test cases with enough detail in them to maximize your odds of testing the important stuff without accidentally omitting tests). Hexawise helps you achieve all of thise objectives. The example shows, as clearly and concisely as I'm able to explain it, how you can use Hexawise to create a small set of powerful tests within four minutes (from 1.7 trillion possible tests) that are well suited to test new features developed in an Agile project iteration.

 

Related: What is Agile? What is not Agile? - Why isn't Software Testing Performed as Efficiently and Effecively as it could be? - Cem Kaner: Testing Checklists = Good / Testing Scripts = Bad?

By: Justin Hunter on Oct 19, 2012

Categories: Agile, Hexawise tips, Pairwise Software Testing, Software Testing

I came across Elisabeth Hendrickson's "Test Heuristics Cheat Sheet" yesterday and developed some pairwise testing (AKA 2-way combinatorial) test cases using many of the good ideas contained in it. I would highly recommend it, I'd recommend you send it (or email a link to this blog post) to everyone on your QA team.

As an indication that the Hendrikson's Test Heuristics Cheat Sheet works well to uncover defects,

  • I wanted to create a set of pairwise tests that could be broadly applicable to test thousands of different applications, I incorporated many ideas from the Test Heuristics Cheat Sheet.

  • I intend to use those inputs to test our test design tool, Hexawise.

  • The way Hexawise works is that users enter "things they want to test" into Hexawise on the first of three screens, the "Define Inputs" screen, then click on "Create Tests." Hexawise then uses a scientific approach to maximizing coverage of the combinations of all the "stuff to be tested" in the fewest possible number of tests. This scientific approach is based on the >40 years of Design of Experiments lessons and includes both pairwise / AllPairs methods as well as more thorough 3-way, 4-way, 5-way and 6-way tests (as well

  • Ironically, even before starting to execute the test conditions suggested by Hexawise, I discovered that the special characters that I had input into the "Define Inputs" screen (which I took from the Test Heuristics Cheat Sheet) triggered a previously unidentified defect in Hexawise itself.

  • The fact that it was triggered so quickly in an application that has been live for a year and used thousands of times is a strong indication that using checklists and cheat sheets can be a great way to efficiently find defects.

Why is using checklists to guide your testing often such an efficient and effective way of finding defects? Here's my top ten list:

  1. The "bad ideas" have already been weeded out.
    1. The ideas on the list have found enough defects to make the author of the checklist think there is value in testing the particular idea.
    2. If you've got a checklist or "cheat sheet" put together by someone as thoughtful and experienced as the Bachs, Bolton, and Hendrickson, you're getting a highly-condensed executive summary version of many of their valuable insights.
    3. All testers go through many, many, "I wonder what would happen if we did this or considered that?" scenarios.
    4. The checklists referenced above represent expertise culled from thousands of testing projects.

  2. Checklists are directly actionable. You can apply them in almost no time at all.
  3. They work well. See Cem Kaner's slides on the Value of Checklists (11 Mb pdf file).
  4. They can easily evolve into some of your most powerful test artifacts.
    • Start with the lists above. See if each of the ideas for tests trigger defects in your Systems Under Test.
    • Find a lot of defects from certain test ideas? Create your own checklist of ideas that worked and iterate them over time.. Consider expanding upon the checklist items and concepts that do bear fruit.
    • Don't ever find defects from certain of the test ideas? Consider dropping those items from the checklists if they don't bear fruit for you (or put tests for those ideas at the back of your lists and only include tests for them if you have extra time).

  5. Checklists include useful, defect-triggering ideas that you may not have thought of on your own.
  6. They're free.
    • No software or books to buy.
    • No courses or conferences to attend.

  7. Using checklists mitigate the risk that you will forget to test for things that you know you should be testing for (but could well forget to test for in any specific instance).
    • As humans, we're naturally forgetful as a species despite our best efforts.
    • Checklists are widely used with good results by doctors, lawyers, pilots, software testers, and people going to grocery stores to minimize the effects of these shortcomings.

  8. Software testing checklists are an efficient way to communicate actionable information.
  9. Software testing checklists are widely applicable to all kinds of software testing.
    • Checklists can be used in creating Unit Tests, Assembly Tests, Product Tests, System Tests, Functional Tests, Load Tests, Performance Tests, User Acceptance Tests, etc.
    • Checklists can be used by Exploratory Testers and "script-everything-in-advance" test-case-centric testers.
    • Checklists can be used in Agile projects as well as Waterfall projects.

  10. Software testing checklists can be easily used in pairwise and combinatorial testing.
    • Using elements from the checklists in a pairwise test will have the added benefit that not only will you test for every one of the testing ideas on the checklist (e.g., XXX) but also, you can easily test for every idea on the checklist **in combination with** every other test idea on the checklist in at least one test case.

    By: Justin Hunter on May 3, 2012

    Categories: Checklists, Software Testing, Testing Checklists, Uncategorized

It's common to have a test plan where the possible values of one parameter depend on the value of another parameter. There are many options for how you can represent this scenario in Hexawise, some options that involve using value expansions (when there is equivalence) and other options that do not use value expansions (when there is not equivalence).

Using Value Expansions in Hexawise

The general rule of thumb for value expansions is that they are for setting up equivalence classes. The key there being the equivalence. The expectations of the system should be the same for every value listed in that particular value expansion.

Let's consider a real world example involving a classification parameter with a value that is dependent on the value of a role parameter:

Inputs
Role: Student, Staff
Classification: Freshman, Sophomore, Junior, Senior, Adjunct, Assistant, Professor, Administrator

So if the Role parameter has a value of Student, then the Classification parameter must have a value of Freshman, Sophomore, Junior or Senior, but if the Role parameter has a value of Staff, then the Classification parameter must have a value of Adjunct, Assistant, Professor or Administrator.

Using value expansions in this case might be a good option. You could setup your inputs, value expansions and value pairs this way:

Inputs
Role: Student, Staff
Classification: Student Classification, Staff Classification

Value Expansion
Student Classification: Freshman, Sophomore, Junior, Senior
Staff Classification: Adjunct, Assistant, Professor, Administrator

Value Pairs
When Role=Student Always Classification=Student Classification
When Role=Staff Always Classification=Staff Classification

You would use this approach if there were no important differences in the business logic or expected behavior of the system when the different expansions of the value were used. If Freshman versus Sophomore is an important label for the users to be able to enter and see, but the system under test doesn't change its behavior based on which value is selected, then those expansions of the value are equivalent and don't need to be tested individually for how they might interact with other parts of the system and create bugs. If this equivalence scenario is true, then you will greatly simplify things for yourself and create fewer tests that are just as powerful by using value expansions.

In the scenario that would support using value expansions, the system might have different behavior for a Junior versus an Adjunct Professor, but not for a Freshman versus a Senior. A Freshman and a Senior are always equivalent in the system, so they can be combined in a value expansion.

However, if the expectations are not the same, then a value expansion should not be used. For example, let's suppose this hypothetical system has business logic giving priority class scheduling to Seniors and only last available scheduling priority to Administrators. In this case, using value expansions as described above would probably be a mistake. Why? Because a Sophomore and a Senior aren't treated the same way by the system, yet Hexawise considers all the expansions of the Student Classification value as equivalent. As long as you've got a test that has paired a value expansion of the Student Classification value with the Overbooked value of the Class Status parameter, then Hexawise won't insist on pairing all the other value expansions for the Student Classification value with Class Status = Overbooked in other tests. You could therefore miss a bug that only occurs when a Senior signs up for an overbooked class.

"One to many" or "multi-valued" married pair model

If the system under test does not consider the values to be equivalent and has requirements and business logic to behave differently, then by using value expansions to signal equivalency to Hexawise when there isn't equivalency is probably a mistake.

So what would you do in that case?

We've decided that it might be nice to be able to set up your inputs and value pairs like this:

Inputs
Role: Student, Staff
Classification: Freshman, Sophomore, Junior, Senior, Adjunct, Assistant, Professor, Administrator

Value Pairs
When Role=Student Always Classification=Freshman, Sophomore, Junior, or Senior When Role=Staff Always Classification=Adjunct, Assistant, Professor, or Administrator

Unfortunately, this kind of a "one to many" or "multi-valued" value pair is something we've only recently realized would be very helpful, and is something we have on the drawing board for Hexawise in the intermediate future, but is not a feature of Hexawise today. In the meantime, you could model it with three parameters:

Inputs
Role: Student, Staff
Student Classification: Freshman, Sophomore, Junior, Senior, N/A Staff Classification: Adjunct, Assistant, Professor, Administrator, N/A

Value Pairs
When Role=Student Always Staff Classification=N/A
When Role=Staff Always Student Classification=N/A

Another modeling option to consider, if there is only special logic for Administrator and for Seniors, but the rest of the values we've been discussing are equivalent, is to use value expansions for just the equivalent values:

Inputs
Role: Student, Staff
Classification: Underclassman, Senior, Professor, Administrator

Value Expansions
Underclassman: Freshman, Sophomore, Junior Professor: Adjunct, Assistant, Full

Value Pairs
When Role=Student Never Classification=Professor When Role=Student Never Classification=Administrator When Role=Staff Never Classification=Underclassman When Role=Staff Never Classification=Senior

I hope this helps you understand the role of value expansions in Hexawise, when to use them (in cases of equivalency), and when to avoid them, and how value pairs and value expansions can be used together to handle cases of dependent parameter values. Value Expansions are a powerful tool to help you decrease the number of tests you need to execute, so take advantage of them, and if you have any questions, just let us know!

By: John Hunter on Apr 26, 2012

Categories: Combinatorial Software Testing, Combinatorial Testing, Hexawise tips, Pairwise Testing, Software Testing, Testing Strategies

84 percent coverage in 20 tests

Hexawise test coverage graph showing 83.9% coverage in just 20 tests

 

Among the many benefits Hexawise provides is creating a test plan that maximizes test coverage with each new scenario tested. The graph above shows that after just 20 test 83.9% of the test combinations have been tested. Read more about this in our case study of a mortgage application software test plan. Just 48 test combinations are needed to test for every valid pair (3.7 million possible tests combinations exist in this case). If you are lost now, this video may help.

The coverage achieved by the first few tests in the plan will be quite high (and the graph line will point up sharply) then the slope will decrease in the middle of the plan (because each new test will tend to test fewer net new pairs of values for the first time) and then at the end of the plan the line will flatten out quite a lot (because by the end, relatively few pairs of values will be tested together for the first time).

One of the benefits Hexawise provides is making that slope as steep as possible. The steeper the slope the more efficient your test plan is. If you repeat the same tests of pairs and triples and... while not taking advantage of the chance to test, untested pairs and triples you will have to create and run far more test than if you intelligently create a test plan. With many interactions to test it is far too complex to manually derive an intelligent test plan. A combinatorial testing tool, like Hexawise, that maximizes test plan efficiency is needed.

For any set of test inputs, there is a finite number of pairs of values that could be tested together (that can be quite a large number). The coverage chart answers, after each tests, what percentage of the total number of pairs (or triples, etc.) that could be tested together have been tested together so far?

The Hexawise algorithms achieve the following objectives that help testers find as many defects as possible in as few tests as possible. In each and every step of each and every test case, the algorithm chooses a test condition that will maximize the number of pairs that can be covered for the first time in the test case. (Or, the maximum number of triplets or quadruplets, etc. based on the thoroughness setting defined by the user). Allpairs (AKA pairwise) is a well known and easy to understand test design strategy. Hexawise lets users create pairwise sets of tests that will test not only every pair but it also allows test designers to generate far more thorough sets of tests (3-way to 6-way coverage). This allows users to "turn up the coverage dial" and generate tests that cover every single possible triplet of test inputs together at least once (or every 4-way combination or 5-way combination or 6-way combination).

Note that the coverage ratio Hexawise shows is based on the factors entered as items to be tested: not a code coverage percentage. Hexawise sorts the test plan to front load the coverage of the tuple pairs, not the coverage of the code paths. Coverage of code paths ultimately depends on how good a job the test designer did at extracting the relevant parameters and values of the system under test. You would expect there to be some loose correlation between coverage of identified tuple pairs and coverage of code paths in most typical systems.

If you want to learn more about these concepts, I would recommend Scott's Scott Sehlhorst articles on pairwise and combinatorial test design. They are some of the clearest introductory articles about pairwise and combinatorial testing that I have seen. They also contain some interesting data points related to the correlation between 2-way / allpairs / pairwise / n-way coverage (in Hexawise) and the white box metrics of branch coverage, block coverage and code coverage (not measurable by Hexawise).

In Software testing series: Pairwise testing, for example, Scott includes these data points:

 

  • We measured the coverage of combinatorial design test sets for 10 Unix commands: basename, cb, comm, crypt, sleep, sort, touch, tty, uniq, and wc... The pairwise tests gave over 90 percent block coverage.

 

  • Our initial trial of this was on a subset Nortel’s internal e-mail system where we able cover 97% of branches with less than 100 valid and invalid testcases, as opposed to 27 trillion exhaustive testcases.

 

  • A set of 29 pair-wise... tests gave 90% block coverage for the UNIX sort command. We also compared pair-wise testing with random input testing and found that pair-wise testing gave better coverage.

 

Related: Why isn't Software Testing Performed as Efficiently and Effecively as it could be? - Video Highlight Reel of Hexawise – a pairwise testing tool and combinatorial testing tool - Combinatorial Testing, The Quadrant of Massive Efficiency Gains

Specific guidance on how to view the percentage of coverage graph for the test plan in Hexawise:

 

When working on your test plan in Hexawise, to get the checklist to be visible, click on the two downward arrow keys located shown in the image:

How-To Progress Checklists-2 inline

Then you'll want to open up the "Advanced" list. So you might need to click here:

Advanced How-To Progress Checklist inline

Then the detailed explanation will begin when you click on "Analyze Tests"

Decreasing Marginal Returns inline

 

This post is adapted (and some new content added) from comments posted by Justin Hunter and Sean Johnson.

By: John Hunter on Feb 3, 2012

Categories: Combinatorial Software Testing, Combinatorial Testing, Efficiency, Multi-variate Testing, Pairwise Software Testing, Pairwise Testing, Scripted Software Testing, Software Testing, Software Testing Efficiency

jobgraph

 

This chart shows that the number of job listings posted for testers has remained essentially flat for testers over the last six years. During this same period, the number of job listings for developers has increased almost 50%. If your company's hiring has mirrored this chart's trends, and your software testers are increasingly feeling that there's not enough time in the day to test everything that's being thrown at them, your testers might have a point.

By: Justin Hunter on Feb 8, 2011

Categories: Software Testing

I can be too verbose with some of my posts. This will be quick.

I recommend that you read this. It is the Exploratory Testing Dynamics document, a tightly condensed list of useful testing heuristics authored by three of the most thoughtful and experienced software testers alive today: James Bach, Michael Bolton, and Jon Bach. Using it will help you improve your software testing capabilities. http://www.satisfice.com/blog/wp-content/uploads/2009/10/et-dynamics22.pdf

Told you it'd be brief. Go. Now. Read.

By: Justin Hunter on Jul 13, 2010

Categories: Exploratory Testing, Software Testing

A friend passed me this set of recent tweets from Wil Shipley, a Mac developer with 11,743 followers on Twitter as of today. Wil recently encountered the familiar problem of what to do when you've got more software tests to run than you can realistically execute.

 

20100623-nixnbwu9urxaufu6hjt2143j3h

I love that. Who can't relate?

Now if only there were a good, quick way to reduce the number of tests from over a billion to a smaller, much more manageable set of tests that were "Altoid-like" in their curious strength. :) I rarely use this blog for shameless plugs of our test case generating tool, but I can't help myself here. The opening is just too inviting. So here goes:

 

"Wil,

There's an app for that... See www.hexawise.com for Hexawise, a "pairwise software test case generating tool on steroids." It eats problems like the one you encountered for breakfast. Hexawise winnows bazillions of possible test cases down in the blink of an eye to small, manageable sets of test cases that are carefully constructed to maximize coverage in the smallest amount of tests, with flexibility to adjust the solutions based upon the execution time you have available. In addition to generating pairwise testing solutions, Hexawise also generates more thorough applied statistics-based "combinatorial software testing" solutions that include tests for, say, all possible 6-way combinations of test inputs.

Where your Mac cops an attitude and tells you "Bitch, I ain't even allocating 1 billion integers to hold your results" and showers you with taunting derisive sneers, head-waggling and snaps all carefully choreographed to let you know where you stand, Hexawise, in contrast, would helpfully tell you: "Only 1 billion total possibilities to select tests from? Pfft! Child's play. Want to start testing the 100 or so most powerful tests? Want to execute an extremely thorough set of 10,000 tests? Want to select a thoroughness setting in the middle? Your wish is my command, sir. You tell me approximately how many tests you want to run and the test inputs you want to include, and I'll calculate the most powerful set of tests you can execute (based on proven applied statistics-based Design of Experiments methods) before you can say "I'm Wil Shipley and I like my TED Conference swag."

More info at:
http://hexawise.tv/intro/
or
https://hexawise.com/Hexawise_Introduction.pdf
free trials at:
http://hexawise.com/signup

By: Justin Hunter on Jun 23, 2010

Categories: Combinatorial Software Testing, Combinatorial Testing, Interesting People , Pairwise Software Testing, Pairwise Testing, Recommended Tool, Software Testing

There are good reasons James Bach is so well known among the testing community and constantly invited to give keynote presentations around the globe at software testing conferences. He's passionate about testing and educating testers; he's a gifted, energetic, and entertaining speaker with a great sense of humor; and he takes joy in rattling his saber and attacking well-established institutions and schools of thought that he disagrees with. He doesn't take kindly to people who make inflated claims of benefits that would materialize "if only you'd perform testing in XYZ way or with ABC tool" given that (a) he can always seem to find exceptions to such claims, (b) he doesn't shy away from confrontation, and (c) he (rightly, in my view) thinks that such benefits statements tend to discount the importance of critical thinking skills being used by testers and other important context-specific considerations.

Leave it up to James to create a list of 13 questions that would be great to ask the next software testing tool vendor who shows up to pitch his problem-solving product. In his blog post titled "The Essence of Heuristics," he posed this exact set of questions in a slightly different context, but as a software testing tool vendor myself, they really hit home. They are:

 

  1. Do they teach you how to tell if it’s working?
  2. Do they teach you how to tell if it’s going wrong?
  3. Do they teach you heuristics for stopping?
  4. Do they teach you heuristics for knowing when to apply it?
  5. Do they compare it to alternative heuristics?
  6. Do they show you why it works?
  7. Do they help you understand when it probably works best?
  8. Do they help you know how to re-design it, if needed?
  9. Do they let you own it?
  10. Do they ask you to practice it?
  11. Do they tell stories about how it has failed?
  12. Do they listen to you when you question or challenge it?
  13. Do they praise you for questioning and challenging it?

 

[Side note: Apparently I wasn't the only one who thought of Hexawise and pairwise / combinatorial test design approaches when they saw these 13 questions. I was amused that after I drafted this post, I saw Jared Quinert's / @xflibble's tweet just now:]

20100601-br4ud66pcc7f79q1ywgbat74jw

Where do I come down on each of James' 13 questions with respect to people I talk to about our test design tool, Hexawise, and the types of benefits and the size of benefits it typically delivers? Quite simply, "Yes" to all 13. I enjoy talking about exactly the kinds of questions that James raised in his list. In fact, when I sought out James to ask him questions at a conference in Boston earlier this year, it was because I wanted his perspective on many of the points above, particularly #11: (hearing stories about how James has seen pairwise and combinatorial approaches to test design fail), and #7 (hearing his views on where it works best and where it would be difficult to apply it). I'll save my specific answers to another post, but I am serious about wanting to share my thoughts on them; time constraints are holding me back today. I gave a speech at the ASQ World Conference on Quality Improvement in St. Louis last week though that addressed many, but not all, of James' questions.

I'm not your typical software tool vendor. Basically, my natural instincts are all wrong for sales. I agree with the premise that "a fool with a tool is still a fool"; when talking to target clients and/or potential partners, I'm inclined to point out deficiencies, limitations, and various things that could go wrong; I'm more of an introvert than an extrovert, etc. Not exactly the typical characteristics of a successful salesman... Having said that, I believe that we've built a very good tool that helps enable dramatic efficiency and thoroughness benefits in many testing situations but our tool, along with the pairwise and combinatorial test design approaches that Hexawise enables both have their limitations. It is primarily by talking to software testers about their positive and negative experiences that our company is able to improve our tool, enhance our training, and provide honest, pragmatic guidance to users about where and how to use our tool (and where and how not to).

Tool vendors who defend their tools (and/or the approaches by which their tools helps users solve problems) as magical, silver bullet solutions are being both foolish and dishonest. Tool vendors who choose not to engage in serious, honest and open discussions with users about the challenges that users have when applying their tools in different situations are being short-sighted. From my own experiences, I can say that talking about the 13 topics raised by James have been invaluable.

By: Justin Hunter on Jun 1, 2010

Categories: Combinatorial Testing, Design of Experiments, Hexawise test case generating tool, Pairwise Testing, Software Testing, Software Testing Efficiency, Uncategorized