Effective Code Coverage

Copenhagen -- 13th August 2008

Paul Johnson

www.pjcj.net

Testing Survey

• Recent testing survey
• By Curtis "Ovid" Poe
• Results at
  http://www.perlmonks.org/?node_id=701817
• Question 11 concerned code coverage

Testing Survey

Q.11 What percentage of your code is covered (rounding down)?

Testing Survey

Q.11 What percentage of your code is covered (rounding down)?

Answer 						    Count
Don't know, but I do coverage testing
Don't do coverage testing
**other**
Less than 50%
50% to 74%
75% to 84%
85% to 94%
95% to 99%
100%

People who answered question:
People who skipped question:

Testing Survey

Q.11 What percentage of your code is covered (rounding down)?

Answer 						    Count
Don't know, but I do coverage testing 		  6 ( 4%)
Don't do coverage testing 			 47 (34%)
**other** 					  3 ( 2%)
Less than 50% 					 21 (15%)
50% to 74% 					 18 (13%)
75% to 84% 					 16 (11%)
85% to 94% 					 12 ( 9%)
95% to 99% 					 15 (11%)
100% 						  2 ( 1%)

People who answered question: 			140 (84%)
People who skipped question: 			 27 (16%)

Of the three people who answered "other":

   1. What tests?
   2. less than 1%
   3. A tiny fraction, so far

Testing Survey

Q.11 What percentage of your code is covered (rounding down)?

Answer 						    Count
Don't know, but I do coverage testing 		  6 ( 4%)
Don't do coverage testing 			 47 (34%)
**other** 					  3 ( 2%)
Less than 50% 					 21 (15%)
50% to 74% 					 18 (13%)
75% to 84% 					 16 (11%)
85% to 94% 					 12 ( 9%)
95% to 99% 					 15 (11%)
100% 						  2 ( 1%)

People who answered question: 			140 (84%)
People who skipped question: 			 27 (16%)

Testing Survey

Q.11 What percentage of your code is covered (rounding down)?

Answer 						    Count
Don't know, but I do coverage testing 		  6 ( 4%)
Don't do coverage testing 			 47 (34%)
**other** 					  3 ( 2%)
Less than 50% 					 21 (15%)
50% to 74% 					 18 (13%)
75% to 84% 					 16 (11%)
85% to 94% 					 12 ( 9%)
95% to 99% 					 15 (11%)
100% 						  2 ( 1%)

People who answered question: 			140 (84%)
People who skipped question: 			 27 (16%)

Testing Survey

Q.11 What percentage of your code is covered (rounding down)?

Answer 						    Count
Don't know, but I do coverage testing 		  6 ( 4%)
Don't do coverage testing 			 47 (34%)
**other** 					  3 ( 2%)
Less than 50% 					 21 (15%)
50% to 74% 					 18 (13%)
75% to 84% 					 16 (11%)
85% to 94% 					 12 ( 9%)
95% to 99% 					 15 (11%)
100% 						  2 ( 1%)

People who answered question: 			140 (84%)
People who skipped question: 			 27 (16%)

Testing Survey

Q.11 What percentage of your code is covered (rounding down)?

Answer 						    Count
Don't know, but I do coverage testing 		  6 ( 4%)
Don't do coverage testing 			 47 (34%)
**other** 					  3 ( 2%)
Less than 50% 					 21 (15%)
50% to 74% 					 18 (13%)
75% to 84% 					 16 (11%)
85% to 94% 					 12 ( 9%)
95% to 99% 					 15 (11%)
100% 						  2 ( 1%)

People who answered question: 			140 (84%)
People who skipped question: 			 27 (16%)

Testing Survey

Q.11 What percentage of your code is covered (rounding down)?

Answer 						    Count
Don't know, but I do coverage testing 		  6 ( 4%)
Don't do coverage testing 			 47 (34%)
**other** 					  3 ( 2%)
Less than 50% 					 21 (15%)
50% to 74% 					 18 (13%)
75% to 84% 					 16 (11%)
85% to 94% 					 12 ( 9%)
95% to 99% 					 15 (11%)
100% 						  2 ( 1%)

People who answered question: 			140 (84%)
People who skipped question: 			 27 (16%)

Testing Survey

Q.11 What percentage of your code is covered (rounding down)?

Answer 						    Count
Don't know, but I do coverage testing 		  6 ( 4%)
Don't do coverage testing 			 47 (34%)
**other** 					  3 ( 2%)
Less than 50% 					 21 (15%)
50% to 74% 					 18 (13%)
75% to 84% 					 16 (11%)
85% to 94% 					 12 ( 9%)
95% to 99% 					 15 (11%)
100% 						  2 ( 1%)

People who answered question: 			140 (84%)
People who skipped question: 			 27 (16%)

Categories

• People who don't do coverage testing
• People whose coverage testing is less than stellar
• People whose coverage testing is pretty good
• People who are having problems covering certain code constructs
• People who are waiting for more coverage metrics

People who don't do code coverage

• Don't know what it is
• Don't see any value in it
• Don't know how to use it

What is code coverage?

• Tells you how much of your code you have tested
• There are various coverage criteria
• "Testing never proves the absence of faults, it only shows their presence." Edsger Dijkstra
• Trying to cover more of the testing problem space

Code coverage metrics

• Show how well exercised your code is
• Many have variations and synonyms
• Subroutine
• Statement
• Branch
• Condition
• Path

Subroutine coverage

• A subroutine is covered if it is called

Statement coverage

• A statement is covered if it is executed
• Still not easy to get 100% statement coverage
• Error conditions, rarely occurring events
• if ($param > 20) {
      die "This should never happen!";
  }
• 100% statement coverage implies 100% subroutine coverage

Branch Coverage

• A program should jump to all possible destinations
• if ($x) { $h = { a => 1 } }
  else { $h = 0 }
  print $h->{a};
• $x must be true on one occasion and false on another
• Protects against errors in which some requirements are not met in one branch
• This code will fail if $x is false (and you are using strict refs).

Branch Coverage

• Missing elses
• $h = 0;
  if ($x)
  {
      $h = { a => 1 };
  }
  print $h->{a};
• 100% branch coverage implies 100% statement coverage

Condition Coverage

• Boolean expression
• Ensure all terms in the expression are exercised
• a if $x || $y;
• Four combinations of values for $x and $y
• $x  $y
   0   0
   0   1
   1   0
   1   1
• Take all for 100% condition coverage

People who don't do code coverage

• Don't know what it is
• Don't see any value in it
• Don't know how to use it

How to use Devel::Cover

• cover -test
• Or read the docs for other options

Example

$ cover -test
Deleting database /home/pjcj/g/perl/shellsource/cover_db
cover: running make test CCFLAGS=-O0\ -fprofile-arcs\ -ftest-coverage
  OTHERLDFLAGS=-fprofile-arcs\ -ftest-coverage
PERL_DL_NONLAZY=1 /usr/local/pkg/base/sw/bin/perl "-MExtUtils::Command::MM"
  "-e" "test_harness(0, 'blib/lib', 'blib/arch')" t/*.t
t/bash......ok
t/csh.......skipped: csh unavailable
t/ksh.......ok
t/sh........ok
t/tcsh......skipped: tcsh unavailable
t/zsh.......ok
All tests successful.
Files=6, Tests=16, 13 wallclock secs ( 0.06 usr  0.00 sys + 12.21 cusr  0.55 csys = 12.82 CPU)
Result: PASS
Reading database from /home/pjcj/g/perl/shellsource/cover_db

---------------------------- ------ ------ ------ ------ ------ ------ ------
File                           stmt   bran   cond    sub    pod   time  total
---------------------------- ------ ------ ------ ------ ------ ------ ------
blib/lib/Shell/Source.pm       96.6   65.0   66.7   90.9    n/a  100.0   86.1
Total                          96.6   65.0   66.7   90.9    n/a  100.0   86.1
---------------------------- ------ ------ ------ ------ ------ ------ ------

Writing HTML output to /home/pjcj/g/perl/shellsource/cover_db/coverage.html ...
done.

Categories

• People who don't do coverage testing
• People whose coverage testing is less than stellar

How to improve your Code Coverage

• Look at the pretty reports
• Write tests to exercise the bits that are red

Categories

• People who don't do coverage testing
• People whose coverage testing is less than stellar
• People whose coverage testing is pretty good

How to improve your coverage even more

• Look at the pretty reports
• Write more tests to exercise the bits that are red
• Remove unused code

Error Conditions

• open F, ">", \$file or die "Cannot open $file: $!"
• close F or die "Cannot close $file: $!"

Error Conditions

• Is it worth checking those error conditions?
• BEGIN { *CORE::GLOBAL::close = sub { 0 } }
• { local *Mylib::meth = sub { 0 }; tst }
• Test::MockObject and similar modules

Platform dependent code

• if ($^O eq "Solaris")
  {
      do_something_clever;
  }
  else
  {
      do_something_less_clever;
  } 
      
   

• Merge the databases
• $ cover solaris_db other_db

Categories

• People who don't do coverage testing
• People whose coverage testing is less than stellar
• People whose coverage testing is pretty good
• People who are having problems covering certain code constructs

Shut up!

• Have you ever wanted to say to Devel::Cover "Yes, I know, please shut up about it"?
• $x ||= get_x; # get_x always returns a true value
• Well now you can!

Uncoverable summary

Uncoverable code

#!/usr/bin/perl
my $x = 1;
my $y = 1;
if ($x && !$y) {
    $x++;
    z();
}
if (!$x || !$y) {
    b(); b();
}
sub z {
    $y++;
}

    

    

Uncoverable detail

Details

• Uncoverable details in .uncoverable file
• People wanted comments
• # uncoverable criterion [type] [count:count] [class:class] [note:note]

Reports

• Various HTML outputs:
• html
• - default
• - link to (sub class of) html_minimal
• html_subtle
• html_basic
• text
• text2
• compilation
• sort
• Your Report Here!

Annotations

• Various Annotations:
• svk
• version
• author
• date
• random
• add a random number to each line!
• example for building your own annotations
• Your Annotation Here!

File Selection

• Collecting coverage data is expensive
• So don't collect more than necessary
• Pass options into Devel::Cover
• -select
• -ignore
• -inc

File Selection

• These are regular expressions
• Any file that is selected is selected
• otherwise any file that is ignored is ignored
• otherwise any file in one of the inc directories is ignored
• otherwise the file is selected

File Selection

• Add to the REs to select by using +select
• Reset the selections using -select
• The same principle applies to the REs to ignore
• inc directories are populated by @INC at the time Devel::Cover is built
• +inc adds to the list, -inc resets it
• defaults for ignore if blib exists: '^t/', '\.t', '^test\.pl'
• -MDevel::Cover=-select,Mylib,+ignore,fools

Categories

• People who don't do coverage testing
• People whose coverage testing is less than stellar
• People whose coverage testing is pretty good
• People who are having problems covering certain code constructs
• People who are waiting for more coverage metrics

Those waiting for more coverage metrics

• You don't need any help
• In fact, you should be helping me write the metrics
• Path coverage
• Data coverage
• Mutation coverage
• Reports and Annotations
• Projects page http://www.pjcj.net/devel-cover/projects.html

Path Coverage

• There are classes of errors which branch coverage cannot detect:
• $h = 0;
  if ($x) { $h = { a => 1 } }
  if ($y) { print $h->{a} }
• 100% branch coverage with ($x, $y) set to (1, 1) and (0, 0)
• (0, 1) => BOOM

Data Coverage

• my $input = int shift;
  my @squares = (0, 1, 5, 9);
  if ($input < 4 && $input > -4) {
      print $squares[abs $input];
  }
• Full coverage with input of -4, 0 and 4
• What's two squared?
• Check each value is read
• Lookup table and tests get values from same source

Regular Expressions

• Their own little language
• With statements, branches, paths and conditions
• And embedded Perl

Stuff you didn't want to know

• Mostly designed cycling up the hill from Triemli to Uitikon

runops

• Perl compiles code to an optree
• runops function walks the optree and calls appropriate functions
• Devel::Cover replaces the runops function and does evil things
• Experimental code messes with the vtable to call different functions
• Should be faster and safer

Back to Reality

• At the end, Devel::Cover walks the optree again, mapping the ops back to reality and associating the coverage data
• Databases containing coverage data from different runs are merged
• Finally a report outputs the data in a pretty web page
• Or in any other format

Other Languages

• Translates from other coverage tools into Devel::Cover's database format
• Currently only gcov from gcc
• Works nicely for combined Perl and C projects

C and XS

• Only for gcc
• Uses gcov
• cover -test

Questions

• ?