# Testing and Debugging

## Chapter Goals

• To learn how to carry out unit tests
• To understand the principles of test case selection and evaluation
• To learn how to use logging
• To become familiar with using a debugger
• To learn strategies for effective debugging

## Unit Tests

• The single most important testing tool
• Checks a single method or a set of cooperating methods
• You don't test the complete program that you are developing; you test the classes in isolation
• For each test, you provide a simple class called a test harness
• Test harness feeds parameters to the methods being tested

## Example: Setting Up Test Harnesses

• To compute the square root of a use a common algorithm:
1. Guess a value x that might be somewhat close to the desired square root (x = a is ok)
2. Actual square root lies between x and a/x
3. Take midpoint (x + a/x) / 2 as a better guess
4. Repeat the procedure. Stop when two successive approximations are very close to each other
• Method converges rapidly. Square root of 100:
`Guess #1: 50.5Guess #2: 26.24009900990099Guess #3: 15.025530119986813Guess #4: 10.840434673026925Guess #5: 10.032578510960604Guess #6: 10.000052895642693Guess #7: 10.000000000139897Guess #8: 10.0`

## Testing the Program

• Output
```Enter a number: `100`
Guess #1: 50.5
Guess #2: 26.24009900990099
Guess #3: 15.025530119986813
Guess #4: 10.840434673026925
Guess #5: 10.032578510960604
Guess #6: 10.000052895642693
Guess #7: 10.000000000139897
Guess #8: 10.0
Guess #9: 10.0
Guess #10: 10.0
Square root: 10.0```
• Does the RootApproximator class work correctly for all inputs?
It needs to be tested with more values
• Re-testing with other values repetitively is not a good idea; the tests are not repeatable
• If a problem is fixed and re-testing is needed, you would need to remember your inputs
• Solution: Write test harnesses that make it easy to repeat unit tests

## Self Check

1. What is the advantage of unit testing?
2. Why should a test harness be repeatable?

1. It is easier to test methods and classes in isolation than it is to understand failures in a complex program.
2. It should be easy and painless to repeat a test after fixing a bug.

## Providing Test Input

• There are various mechanisms for providing test cases
• One mechanism is to hardwire test inputs into the test harness
• Simply execute the test harness whenever you fix a bug in the class that is being tested
• Alternative: place inputs on a file instead

## File RootApproximatorHarness1.java

Output
`   square root of 100.0 = 10.0   square root of 4.0 = 2.0   square root of 2.0 = 1.414213562373095   square root of 1.0 = 1.0   square root of 0.25 = 0.5   square root of 0.01 = 0.1`

## Providing Test Input

• You can also generate test cases automatically
• For few possible inputs, feasible to run through (representative) number of them with a loop

## File RootApproximatorHarness2.java

Output
`   square root of 1.0 = 1.0   square root of 1.5 = 1.224744871391589   square root of 2.0 = 1.414213562373095   . . .   square root of 9.0 = 3.0   square root of 9.5 = 3.0822070014844885   square root of 10.0 = 3.162277660168379`

## Providing Test Input

• Previous test restricted to small subset of values
• Alternative: random generation of test cases

## File RootApproximatorHarness3.java

Output
`   square root of 810.4079626570873 = 28.467665212607223   square root of 480.50291114306344 = 21.9203766195534   square root of 643.5463246844379 = 25.36821485017103   square root of 506.5708496713842 = 22.507128863348704   square root of 539.6401504334708 = 23.230156057019308   square root of 795.0220214851004 = 28.196134867834285   . . .`

## Providing Test Input

• Selecting good test cases is an important skill for debugging programs
• Test all features of the methods that you are testing
• Test typical test cases
100, 1/4, 0.01, 2, 10E12, for the SquareRootApproximator
• Test boundary test cases: test cases that are at the boundary of acceptable inputs
0, for the SquareRootApproximator
• Programmers often make mistakes dealing with boundary conditions
Division by zero, extracting characters from empty strings, and accessing null pointers
• Gather negative test cases: inputs that you expect program to reject
Example: square root of -2. Test passes if harness terminates with assertion failure (if assertion checking is enabled)

## Reading Test Inputs from a File

• More elegant to place test values in a file
• Input redirection:
`java Program < data.txt`
• Some IDEs do not support input redirection. Then, use command window (shell).
• Output redirection:
`java Program > output.txt`

## Reading Test Inputs from a File

• File test.in:
``1` 100`2` 4`3` 2`4` 1`5` 0.25`6` 0.01`
• Run the program:
`java RootApproximatorHarness4 < test.in > test.out`
• File test.out:
``1` square root of 100.0 = 10.0`2` square root of 4.0 = 2.0`3` square root of 2.0 = 1.414213562373095`4` square root of 1.0 = 1.0`5` square root of 0.25 = 0.5`6` square root of 0.01 = 0.1`

## Self Check

1. How can you repeat a unit test without having to retype input values?
2. Why is it important to test boundary cases?

1. By putting the values in a file, or by generating them programmatically.
2. Programmers commonly make mistakes when dealing with boundary conditions.

## Test Case Evaluation

• How do you know whether the output is correct?
• Calculate correct values by hand
E.g., for a payroll program, compute taxes manually
• Supply test inputs for which you know the answer
E.g., square root of 4 is 2 and square root of 100 is 10
• Verify that the output values fulfill certain properties
E.g., square root squared = original value
• Use an Oracle: a slow but reliable method to compute a result for testing purposes
E.g., use Math.pow to slower calculate x1/2 (equivalent to the square root of x)

## Output

`Test passed: x = 913.6505141736327, root squared = 913.6505141736328Test passed: x = 810.4959723987972, root squared = 810.4959723987972Test passed: x = 503.84630929985883, root squared = 503.8463092998589Test passed: x = 115.4885096006315, root squared = 115.48850960063153Test passed: x = 384.973238438713, root squared = 384.973238438713. . .Pass: 100Fail: 0`

## Output

`Test passed: square root = 718.3849112194539, oracle = 718.3849112194538Test passed: square root = 641.2739466673618, oracle = 641.2739466673619Test passed: square root = 896.3559528159169, oracle = 896.3559528159169Test passed: square root = 591.4264541724909, oracle = 591.4264541724909Test passed: square root = 721.029957736384, oracle = 721.029957736384. . .Pass: 100Fail: 0`

## Self Check

1. Your task is to test a class that computes sales taxes for an Internet shopping site. Can you use an oracle?
2. Your task is to test a method that computes the area of an arbitrary polygon. Which polygons with known areas can you use as test inputs?

1. Probably not–there is no easily accessible but slow mechanism to compute sales taxes. You will probably need to verify the calculations by hand.
2. There are well-known formulas for the areas of triangles, rectangles, and regular n-gons.

## Regression Testing

• Save test cases
• Use saved test cases in subsequent versions
• A test suite is a set of tests for repeated testing
• Cycling = bug that is fixed but reappears in later versions
• Regression testing: repeating previous tests to ensure that known failures of prior versions do not appear in new versions

## Test Coverage

• Black-box testing: test functionality without consideration of internal structure of implementation
• White-box testing: take internal structure into account when designing tests
• Test coverage: measure of how many parts of a program have been tested
• Make sure that each part of your program is exercised at least once by one test case
E.g., make sure to execute each branch in at least one test case
• Tip: write first test cases before program is written completely → gives insight into what program should do
• Modern programs can be challenging to test
• Graphical user interfaces (use of mouse)
• Network connections (delay and failures)
• There are tools to automate testing in this scenarios
• Basic principles of regression testing and complete coverage still hold

## Self Check

1. Suppose you modified the code for a method. Why do you want to repeat tests that already passed with the previous version of the code?
2. Suppose a customer of your program finds an error. What action should you take beyond fixing the error?

1. It is possible to introduce errors when modifying code.
2. Add a test case to the test suite that verifies that the error is fixed.

## Unit Testing with JUnit

• http://junit.org
• Built into some IDEs like BlueJ and Eclipse
• Philosophy: whenever you implement a class, also make a companion test class

## Program Trace

• Messages that show the path of execution
`if (status == SINGLE){   System.out.println("status is SINGLE");   . . . }. . .`
• Drawback: Need to remove them when testing is complete, stick them back in when another error is found
• Solution: use the Logger class to turn off the trace messages without removing them from the program

## Logging

• Logging messages can be deactivated when testing is complete
• Use global object Logger.global
• Log a message
`Logger.global.info("status is SINGLE");`
• By default, logged messages are printed. Turn them off with
`Logger.global.setLevel(Level.OFF);`
• Logging can be a hassle (should not log too much nor too little)
• Some programmers prefer debugging (next section) to logging

## Logging

• When tracing execution flow, the most important events are entering and exiting a method
• At the beginning of a method, print out the parameters:
`public TaxReturn(double anIncome, int aStatus){   Logger.global.info("Parameters: anIncome = " + anIncome      + " aStatus = " + aStatus);   . . .}`
• At the end of a method, print out the return value:
`public double getTax(){   . . .   Logger.global.info("Return value = " + tax);   return tax;}`

## Self Check

1. Should logging be activated during testing or when a program is used by its customers?
2. Why is it better to send trace messages to Logger.global than to System.out?

1. Logging messages report on the internal workings of your program–your customers would not want to see them. They are intended for testing only.
2. It is easy to deactivate Logger.global when you no longer want to see the trace messages, and to reactivate it when you need to see them again.

## Using a Debugger

• Debugger = program to run your program and analyze its run-time behavior
• A debugger lets you stop and restart your program, see contents of variables, and step through it
• The larger your programs, the harder to debug them simply by logging
• Debuggers can be part of your IDE (Eclipse, BlueJ) or separate programs (JSwat)
• Three key concepts:
• Breakpoints
• Single-stepping
• Inspecting variables

## Debugging

• Execution is suspended whenever a breakpoint is reached
• In a debugger, a program runs at full speed until it reaches a breakpoint
• When execution stops you can:
• Inspect variables
• Step through the program a line at a time
• Or, continue running the program at full speed until it reaches the next breakpoint
• When program terminates, debugger stops as well
• Breakpoints stay active until you remove them
• Two variations of single-step command:
• Step Over: skips method calls
• Step Into: steps inside method calls

## Single-step Example

• Current line:
```String input = in.next();Word w = new Word(input);`int syllables = w.countSyllables();`
System.out.println("Syllables in " + input + ": " + syllables);```
• When you step over method calls, you get to the next line:
`String input = in.next();Word w = new Word(input);int syllables = w.countSyllables();`System.out.println("Syllables in " + input + ": " + syllables);``
• However, if you step into method calls, you enter the first line of the countSyllables method
```public int countSyllables(){`   int count = 0;`
int end = text.length() - 1;
. . .
}```

## Self Check

1. In the debugger, you are reaching a call to System.out.println. Should you step into the method or step over it?
2. In the debugger, you are reaching the beginning of a long method with a couple of loops inside. You want to find out the return value that is computed at the end of the method. Should you set a breakpoint, or should you step through the method?

1. You should step over it because you are not interested in debugging the internals of the println method.
2. You should set a breakpoint. Stepping through loops can be tedious.

## Sample Debugging Session

• Word  class counts syllables in a word
• Each group of adjacent vowels (a, e, i, o, u, y) counts as one syllable
• However, an e at the end of a word doesn't count as a syllable
• If algorithm gives count of 0, increment to 1
• Constructor removes non-letters at beginning and end

## Debug the Program

• Buggy output (for input "hello yellow peach"):
`Syllables in hello: 1Syllables in yellow: 1Syllables in peach: 1`
• Set breakpoint in first line of countSyllables of Word class
• Start program, supply input. Program stops at breakpoint
• Method checks if final letter is 'e'
• Check if this works: step to line where check is made and inspect variable ch
• Should contain final letter but contains 'l'

## More Problems Found

• end is set to 3, not 4
• text contains "hell", not "hello"
• No wonder countSyllables returns 1
• Culprit is elsewhere
• Can't go back in time
• Restart and set breakpoint in Word constructor

## Debugging the Word Constructor

• Supply "hello" input again
• Break past the end of second loop in constructor
• Inspect i and j
• They are 0 and 4–makes sense since the input consists of letters
• Why is text set to "hell"?
• Off-by-one error: Second parameter of substring is the first position not to include
• text = substring(i, j);
should be
text = substring(i, j + 1);

## Another Error

• Fix the error
• Recompile
• Test again:
`Syllables in hello: 1Syllables in yellow: 1Syllables in peach: 1`
• Oh no, it's still not right
• Start debugger
• Erase all old breakpoints and set a breakpoint in countSyllables method
• Supply input "hello"

## Debugging countSyllables (again)

• Break in the beginning of countSyllables. Then, single-step through loop
`boolean insideVowelGroup = false;for (int i = 0; i <= end; i++){   ch = Character.toLowerCase(text.charAt(i));   if ("aeiouy".indexOf(ch) >= 0)   {      // ch is a vowel      if (!insideVowelGroup)      {         // Start of new vowel group         count++;         insideVowelGroup = true;      }   }}`
• First iteration ('h'): skips test for vowel
• Second iteration ('e'): passes test, increments count
• Third iteration ('l'): skips test
• Fifth iteration ('o'): passes test, but second if is skipped, and count is not incremented

## Fixing the Bug

• insideVowelGroup was never reset to false
• Fix
`if ("aeiouy".indexOf(ch) >= 0){   . . .}else insideVowelGroup = false;`
• Retest: All test cases pass
`Syllables in hello: 2Syllables in yellow: 2Syllables in peach.: 1`
• Is the program now bug-free? The debugger can't answer that.

## Self Check

1. What caused the first error that was found in this debugging session?
2. What caused the second error? How was it detected?