Practical Code Coverage

Bern -- 22nd March 2013

Paul Johnson

www.pjcj.net

Testing

• Test.pm
• Test::More.pm
• Test::*
• "Testing never proves the absence of faults, it only shows their presence." Edsger Dijkstra

What is code coverage?

• Tells you how much of your code you have tested
• There are various coverage criteria
• Trying to cover more of the testing problem space

Black Box Testing

• No knowledge of object internals
• Internals may change without affecting tests
• Interface should remain constant

White Box Testing

• Looks inside object
• Uses this knowledge to assist in testing

Code coverage metrics

• Show how well exercised your code is
• Many have variations and synonyms

Statement coverage

• A statement is covered if it is executed
• Statement != line of code
• Sequences of statements
• Weakest form of coverage

Statement coverage

• Still not easy to get 100% statement coverage
• Error conditions, rarely occurring events
• if ($param > 20) {
      die "This should never happen!";
  }
• Nice to mark code that shouldn't be executed
• Synonyms: statement execution, line, block, basic block or segment coverage

Branch Coverage

• A program should jump to all possible destinations
• if ($x) {
      print "a";
  } else {
      print "b";
  }
• $x must be true on one occasion and false on another

Branch Coverage

• Protects against errors in which some requirements are not met in one branch
• if ($x) {
      $h = { a => 1 };
  } else {
      $h = 0;
  }
  print $h->{a};
• This code will fail if $x is false (and you are using strict refs).

Branch Coverage

• Missing elses
• $h = 0;
  if ($x) {
      $h = { a => 1 };
  }
  print $h->{a};
• 100% branch coverage implies 100% statement coverage
• Synonyms: decision, arc or all edges coverage

Condition Coverage

• Boolean expression
• Ensure all terms in the expression are exercised
• a if $x || $y;
• Four combinations of values for $x and $y
• $x  $y
   0   0
   0   1
   1   0
   1   1
• Take all for 100% condition coverage

Condition Coverage

• Short circuiting operators
• a if $x || $y;
• Three combinations of values for $x and $y
• $x  $y
   0   0
   0   1
   1   x

Condition Coverage

• Of course, && is different
• a if $x && $y;
• Three combinations of values for $x and $y
• $x  $y
   0   x
   1   0
   1   1

Condition Coverage

• Can get very complicated
• a if ($p && $q) || ($r && $s) ||
       ($u && ($w || $x) && ($y || $z))
• n variables => 2 ^ n combinations (ignoring shortcut operators)
• Various ways of reporting condition coverage
• Focus on important combinations

Condition Coverage

• Expressions which are not part of a branching construct
• $z = $x || $y;
• Synonyms: expression, condition-decision and multiple decision coverage

Time Coverage

• aka profiling
• Well, sort of
• But it might highlight algorithm problems
• Use NYTProf to do it properly

Documentation Coverage

• Not really code coverage either
• But docs are important, right?
• Pod::Coverage

Devel::Cover

• Code coverage tool for Perl
• Mostly designed cycling up the hill from Triemli to Uitikon
• Subroutine Coverage
• Statement Coverage
• Branch Coverage
• Condition Coverage
• Time Coverage
• Documentation Coverage

runops

• Perl compiles code to an optree
• runops function walks the optree and calls appropriate functions
• Devel::Cover replaces the runops function and does evil things
• New code messes with the vtable to call different functions and do evil things

Back to Reality

• At the end, Devel::Cover walks the optree again, mapping the ops back to reality and associating the coverage data
• Databases containing coverage data from different runs are merged
• Finally a report outputs the data in a pretty web page
• Or in any other format

Other Languages

• Translates from other coverage tools into Devel::Cover's database format
• Currently only gcov from gcc
• Works nicely for combined Perl and C projects

C and XS

• Only for gcc
• Uses gcov
• cover -test

Devel::Cover

• cover -test
• Or read the docs for other options
• Requires 5.6.1
• > 5.8.2 recommended

Future

• Working on TPF grant
• Implement Path Coverage
• Mutation Coverage
• Regular Expression Coverage
• Fully buzzword compliant AJAXy GUI

Questions

• ?