Testing and Code Coverage

Stockholm -- 25th May 2008

Paul Johnson

www.pjcj.net

Thanks

• Tack

What is code coverage?

• Tells you how much of your code you have tested
• There are various coverage criteria
• "Testing never proves the absence of faults, it only shows their presence." Edsger Dijkstra
• Trying to cover more of the testing problem space

Black Box Testing

• No knowledge of object internals
• Internals may change without affecting tests
• Interface should remain constant

White Box Testing

• Looks inside object
• Uses this knowledge to assist in testing

Where does it fit?

• Where does code coverage fit into the testing process?
• White box testing methodology
• Most important at module testing phase
• Regression tests are sometimes black box tests
• Most tests are the same in Perl, just run at different times

How do people normally write tests?

• Write what?
• Sanity check
• How's the project coming along?
• It's 99% done, I just need to test it.
• Write code coverage tests during development
• Bug reports

Code coverage metrics

• Show how well exercised your code is
• Many have variations and synonyms

Statement coverage

• A statement is covered if it is executed
• Statement != line of code
• Sequences of statements
• Weakest form of coverage

Statement coverage

• Still not easy to get 100% statement coverage
• Error conditions, rarely occurring events
• if ($param > 20) {
      die "This should never happen!";
  }
• Nice to mark code that shouldn't be executed
• Synonyms: statement execution, line, block, basic block or segment coverage

Branch Coverage

• A program should jump to all possible destinations
• if ($x)
  {
      print "a";
  }
  else
  {
      print "b";
  }
• $x must be true on one occasion and false on another

Branch Coverage

• Protects against errors in which some requirements are not met in one branch
• if ($x)
  {
      $h = { a => 1 };
  }
  else
  {
      $h = 0;
  }
  print $h->{a};
• This code will fail if $x is false (and you are using strict refs).

Branch Coverage

• Missing elses
• $h = 0;
  if ($x)
  {
      $h = { a => 1 };
  }
  print $h->{a};
• 100% branch coverage implies 100% statement coverage
• Synonyms: decision, arc or all edges coverage

Path Coverage

• There are classes of errors which branch coverage cannot detect:
• $h = 0;
  if ($x) { $h = { a => 1 } }
  if ($y) { print $h->{a} }
• 100% branch coverage with ($x, $y) set to (1, 1) and (0, 0)
• (0, 1) => BOOM

Path Coverage

• Goal is to ensure that all paths through program are taken
• Too many paths
• Restrict to paths in a subroutine
• or to two consecutive branches

Path Coverage

• $h = 0;
  if ($x) { $h = { a => 1 } }
  if ($y) { print $h->{a} }
• Four paths through this code
• $x  $y
   0   0
   0   1
   1   0
   1   1
• Take all for 100% path coverage
• Missing elses count as paths

Path Coverage

• 100% path coverage is not always possible:
• a if $x;
  b;
  c unless $x;
• 50% path coverage is the best you can get here
• Code coverage tools could recognise this to a limited extent
• Halting problem

Path Coverage

• Loops contribute to paths
• Simplest solution is to count multiple iterations as one branch
• 100% path coverage implies 100% branch coverage
• Synonyms: predicate, basis path and LCSAJ (Linear Code Sequence And Jump) coverage.

Condition Coverage

• Boolean expression
• Ensure all terms in the expression are exercised
• a if $x || $y;
• Four combinations of values for $x and $y
• $x  $y
   0   0
   0   1
   1   0
   1   1
• Take all for 100% condition coverage

Condition Coverage

• Short circuiting operators
• a if $x || $y;
• Three combinations of values for $x and $y
• $x  $y
   0   0
   0   1
   1   x

Condition Coverage

• Of course, && is different
• a if $x && $y;
• Three combinations of values for $x and $y
• $x  $y
   0   x
   1   0
   1   1

Condition Coverage

• Can get very complicated
• a if ($p && $q) || ($r && $s) ||
       ($u && ($w || $x) && ($y || $z))
• n variables => 2 ^ n combinations (ignoring shortcut operators)
• Various ways of reporting condition coverage
• Focus on important combinations

Condition Coverage

• Expressions which are not part of a branching construct
• $z = $x || $y;
• Synonyms: expression, condition-decision and multiple decision coverage

Time Coverage

• aka profiling
• Well, sort of
• But it might highlight algorithm problems

Documentation Coverage

• Not really code coverage either
• But docs are important, right?

How to use Code Coverage

• Assume you have managed to run your test suite with your coverage tool
• You get, for example, 86% statement coverage
• Not too bad, but room for improvement
• A more detailed report shows a couple of methods are never called

How to use Code Coverage

• Options to increase coverage:
• Write tests for the methods
• Decide they are not needed and delete them
• Bury your head in the sand
• You choose option 1
• Your code coverage increases
• Your test suite is strengthened
• World peace ensues

How to use Code Coverage

• But it's not always that easy
• Adding a test may show errors in the newly covered code
• It might be impossible to exercise some code
• This can show errors or omissions in implementation or specification
• $x = 0;
  print "a" if $x;

How to use Code Coverage

• If coverage can find errors even in design and specification,
• use it early, use it often

How to use Code Coverage

• Decide you need a new program / module / method / subroutine / hack
• Write tests and documentation
• Write the code
• Run the tests
• Check the coverage
• Add tests or refine the design and code until coverage is satisfactory
• This may be a little idealistic

Which coverage metrics to use

• Start with simple metrics:
• Subroutine
• Statement
• Branch
• Condition and Path

What percentages to aim for

• 100% for everything is nice, but probably unrealistic
• It depends
• goals of your project
• cost of failure
• what the software/hardware will be used for
• and by how many people
• when you start using coverage
• priorities of those you work with and for
• how the software was designed and written
• code coverage tool you use
• It just depends

What percentages to aim for

• But that's not very helpful
• Subroutine > 98%
• Statement > 95%
• Branch > 90%
• Path lower still, depending on how you define a path
• Condition also lower, depending on how you define a condition
• Be Pragmatic

Danger, Will Robinson! Danger!

• So, what's the downside?
• Why isn't everybody using code coverage all the time?
• Why doesn't everybody use other good ideas, tools and methodologies?
• Why isn't everybody nice to everybody else?

Why don't people use coverage

• They don't know about it
• They don't understand it
• They don't see any benefit

Problems of using coverage

• Need to find a decent coverage tool
• Then learn how to use it
• Then run your test suite with the coverage tool

Getting your code through the tool

• Coverage is white box
• Coverage tool must understand your code
• Maybe instrument the code
• May not give the same results
• Coverage tool may have bugs or deficiencies

Overheads

• Primarily time
• Also memory, disk space etc.
• Overhead varies
• x1.2 - x20+

When the tests have run

• Only the beginning
• Any serious drive for quality takes effort and resources
• Quality is not guaranteed, merely improved
• Don't rely on code coverage

What code coverage won't do

• Hopefully you'll at least get consistent Statement and Branch coverage
• Condition and Path if available probably have incomplete information
• Other criteria

Data Coverage

• my $input = int shift;
  my @squares = (0, 1, 5, 9);
  if ($input < 4 && $input > -4) {
      print $squares[abs $input];
  }
• Full coverage with input of -4, 0 and 4
• What's two squared?
• Check each value is read
• Lookup table and tests get values from same source

Regular Expressions

• Their own little language
• With statements, branches, paths and conditions
• And embedded Perl

Devel::Cover

• Code coverage tool for Perl
• Mostly designed cycling up the hill from Triemli to Uitikon
• Subroutine Coverage
• Statement Coverage
• Branch Coverage
• Condition Coverage
• Time Coverage
• Documentation Coverage

runops

• Perl compiles code to an optree
• runops function walks the optree and calls appropriate functions
• Devel::Cover replaces the runops function and does evil things
• Experimental code messes with the vtable to call different functions

Back to Reality

• At the end, Devel::Cover walks the optree again, mapping the ops back to reality and associating the coverage data
• Databases containing coverage data from different runs are merged
• Finally a report outputs the data in a pretty web page
• Or in any other format

Other Languages

• Translates from other coverage tools into Devel::Cover's database format
• Currently only gcov from gcc
• Works nicely for combined Perl and C projects

C and XS

• Only for gcc
• Uses gcov
• cover -test

Devel::Cover

• cover -test
• Or read the docs for other options
• Requires 5.6.1
• > 5.8.2 recommended

Future

• Implement Path Coverage
• Mutation Coverage
• Regular Expression Coverage
• Fully buzzword compliant AJAXy GUI

Questions

• ?