Software Black Box Testing

8 min readJun 30, 2020

Introduction

Functional testing requires the analysis of the outputs generated by the system (or its components) in response to input (test cases) defined on the basis of knowledge of the requirements of the system (or its components). Often made in Black Box mode, i.e. without access in any way to the internal structure of the software.

Functional testing is also called black box testing. We are interested in the contour of the system and have no information about what it looks like inside. Based on knowledge of the software structure, and in particular the code, associated inputs and oracle, for the definition of test cases.
Necessarily realized by accessing the source code, then in white box mode.

Whitebox, we can also access the source code, I want the most detailed testing possible. Then there is grey box testing which wants to indicate that something I know and something I don’t. Functional and black box testing are not really the same thing, because there are black box testing of things that are not functionality, e.g. also performance testing is black box testing.
When I talk about functional testing I typically test the functionality that the application promised to do.

Testing Black Box

The common point of all “Black box” techniques is the fact that the software is accessed only through its interface, without direct access to the code of the component to be tested (at the limit, without access to the code at all).

There is no such thing as a black box technique:

Testing based on requirements;
Testing based on use case scenarios;
Testing with equivalence classes (minimum coverage of equivalence classes, coverage of adjacent equivalence classes, coverage of n-ple of equivalence classes, combinatorial coverage of equivalence classes);
Testing with equivalence classes and limit values;
Testing from decision tables;

Testing based on requirements

The principle of verifiability of requirements states that requirements should be testable, i.e. written in such a way that tests can be designed to demonstrate that the requirement has been met.

Requirement-based testing is a validation technique where various tests are designed for each requirement.

Note: the V model said: the day I do the requirements test I also do the requirements test, so I say what the system should do and what tests we should do to say it really did it.

Use Case Testing

I note the Use Case Diagram and the description of all use case scenarios, one or more test cases are designed for each scenario;
The designed test cases are performed manually or automatically;
The testing strategy aims at covering use cases and scenarios.

If the requirements are made as use cases and scenarios, the scenario by its very nature lends itself to verification, because the ideal scenario is precisely a sequence of steps to be followed to obtain the result.
In this case it becomes essential that when we design scenarios we do so in detail, with all the exception scenarios.
If we write scenarios with all the exceptions we have half a test already written, just follow the instructions of the scenarios.

Testing of Partitions (or Classes of Equivalence)

Input and output data can generally be divided into classes where all members of the same class are somehow related. Each of the classes constitutes an equivalence class (a partition) and the program will (likely) behave in the same way for each class member. Test cases should be chosen from within each partition.

Example: If we want to try the sum thoroughly, we should do 2³² for both addends, and we get to 2³²*2³²=2⁶⁴ tests, that’s too many tests. But note we must be careful that the oracle is overflow.
Let’s start thinning out the rehearsals, saying that if you can do 4+3 then I trust you can do 4+5 and I don’t try all the values.
So I take some values that I think are completely equivalent from the point of view of function and put them in a class of equivalence.
In the case of the sum I have only one class, if we have the doubt about positive and negative numbers, we would do the class of positive numbers and the class of negative numbers for the first and then the second addendo. Now our tests could be done by covering the equivalence classes in every possible way. So I take a number from the first equivalence class and a number from the second, and then the various combinations.
I do 4 tests because: the first input had 2 classes of equivalence, the second input had 2 classes of equivalence, I get the combinations with the Cartesian product and they are 4.

Partitions are identified using program specifications or other documentation. A possible subdivision is one where the equivalence class represents a set of valid or invalid states for a condition on input variables.

Search for equivalency classes

Basic Technique:

For each input you get: A valid equivalence class, corresponding to the set of values considered valid for that input; A set of invalid equivalence classes, one for each invalid condition. Each of these conditions corresponds to a set of values (invalid equivalence class);

More detailed technique:

Even for the valid classes more than one equivalence class is distinguished, depending on the different scenarios that can be exercised

If the input is a:

range of values: a valid class for values within the range, an invalid class for values below the minimum, and an invalid class for values above the maximum;
specific value: one class valid for the specified value, one class invalid for lower values, and one invalid for higher values;
element of a discrete set: one valid class corresponding to the set (classical technique) or one valid class for each element of the set (detailed technique), one invalid for an element not belonging to the set;
Boolean value: As in the previous case, but for a two-value discrete set (true, false);

In all cases, it is good to consider also an additional class of invalid equivalence, corresponding to the input not belonging to the expected type;

Coverage strategies

Minimum coverage of equivalence classes. Each class of equivalence is covered by at least one test case. Minimum number of test cases equal to the number of input classes with multiple classes of equivalence.

Coverage of adjacent equivalence classes. Each equivalence class is covered by at least one test case. There is at least one test case per test case that differs by only one equivalence class. The number of test cases is in the order of the total number of equivalence classes.

Coverage of n-ple of equivalence class values (k-way). All k-ple of different equivalence classes are exercised at least once.

Coverage of all combinations of equivalence classes. Coverage of any combination of equivalence classes. Number of test cases equal to the production of the cardinality of the quantities of equivalence classes of each class. Equivalent to the previous case with k=number of inputs.

Equivalency classes example

In a form you must enter your date of birth, composed of day (numerical), month (string that may be January … December), year (numerical, between 1583 and 2100). The software must correctly recognize between valid dates (corresponding to days that actually existed) and invalid dates and provide the corresponding day of the week for valid dates.
Select test cases by partitioning into equivalence classes.

The conditions on the ‘day’ input:

Entry conditions is that the day can be between 1 and 31, so equivalence classes are:

Valid: EC1 : 1 ≤ DAY ≤31;
Not valid: EC2 : DAY < 1, EC3 : DAY > 31, EC4 : DAY is not a whole number.

If our software takes a stream as input then an invalid class may be if the user writes in words. But e.g. in the unit test of a function then it will necessarily be an integer. Note that the equivalence classes depend on the programming language because we are solving a practical testing problem.

The conditions on the ‘month’ input:

Input conditions is that the month must be in the whole M=(January, February, March, April, May, June, July, August, September, October, November, December) so equivalence classes are:

Valid: EC5: MONTH ∈ M
EC51: MONTH = January, EC52: MONTH = February, EC53: MONTH = March, …. (Total 12 classes of equivalence);
Not valid: EC6: MONTH ∉ M;

The conditions on the ‘year’ input:

Input conditions must be between 1583 and 2100 Equivalence classes:

Valid: EC7: 1583 ≤ YEAR ≤ 2100;
Not valid EC8: YEAR< 1583, EC9: YEAR > 2100, EC10: YEAR is not a whole number.

Selection of test cases from equivalence classes

Minimum testing with coverage of equivalence classes. In this way we generate the minimum number of test cases able to cover each equivalence class at least once, thus maximizing efficiency;
Testing with coverage of adjacent equivalence classes. We generate test cases that differ by the minimum number of equivalence classes covered (ideally one class). Good compromise between effectiveness and efficiency with positive consequences also for debugging activities;
Combinatorial testing in which we generate all possible combinations of the defined classes in order to maximize effectiveness.

Minimum testing with coverage of equivalence classes

An efficient Test Suite could be the following:

All classes of equivalence are covered but it is very difficult to detect errors. For example in TC2 the system might respond with an exception because the day is less than 1, without evaluating the month and year.

Code example in Java 8 with JUnit 5:

For the TC4 test the compiler with a static analysis already makes me realize that I put a string instead of a number and so there is no need for some tests.

Testing with coverage of adjacent equivalence classes

Let’s evaluate how many test cases find problems.

All valid (classical technique) and invalid equivalence classes are covered. If a test is successful, we may immediately identify the invalid equivalence class that is not handled correctly. We do not test dates in February, for example.

In this case it is simple, but in complicated cases it may take an algorithm. We take the first test case and then we vary one, and then we vary the one in every possible way. We vary the year by taking a value in the respective equivalence classes. Now we have to take another input and make it vary. Per line the tests are adjacent. So if a test of a certain line is wrong, I know the input that made it wrong. Note that building adjacent tests will not always be the same, we have different solutions and they are not all able to find the problems.

Testing with minimum coverage of equivalence classes (valid classes listed exhaustively)

All valid equivalence classes (detailed technique) are covered. If a test is successful, we may immediately identify the invalid equivalence class that is not handled correctly. However, we do not test dates such as 30 February.

Realization with JUnit 5

The code for these tests can be found in this repository https://github.com/lonardogio/black_box_testing on github. We will perform in JUnit the tests corresponding to these three
possible test suites:

TestBlackBoxAdjacentDetailedCoverage.java

You may notice that some tests are not feasible (the JUnit code would not compile), e.g. those with text input where numbers are required (e.g. “two thousand”).