Effective Code Coverage

Copenhagen -- 13th August 2008

Paul Johnson

www.pjcj.net

Testing Survey

survey.jpg

Testing Survey

survey.gif
Q.11 What percentage of your code is covered (rounding down)?

Testing Survey

survey.gif
Q.11 What percentage of your code is covered (rounding down)?

Answer 						    Count
Don't know, but I do coverage testing
Don't do coverage testing
**other**
Less than 50%
50% to 74%
75% to 84%
85% to 94%
95% to 99%
100%

People who answered question:
People who skipped question:

Testing Survey

survey.gif
Q.11 What percentage of your code is covered (rounding down)?

Answer 						    Count
Don't know, but I do coverage testing 		  6 ( 4%)
Don't do coverage testing 			 47 (34%)
**other** 					  3 ( 2%)
Less than 50% 					 21 (15%)
50% to 74% 					 18 (13%)
75% to 84% 					 16 (11%)
85% to 94% 					 12 ( 9%)
95% to 99% 					 15 (11%)
100% 						  2 ( 1%)

People who answered question: 			140 (84%)
People who skipped question: 			 27 (16%)

Of the three people who answered "other":

   1. What tests?
   2. less than 1%
   3. A tiny fraction, so far

Testing Survey

survey.gif
Q.11 What percentage of your code is covered (rounding down)?

Answer 						    Count
Don't know, but I do coverage testing 		  6 ( 4%)
Don't do coverage testing 			 47 (34%)
**other** 					  3 ( 2%)
Less than 50% 					 21 (15%)
50% to 74% 					 18 (13%)
75% to 84% 					 16 (11%)
85% to 94% 					 12 ( 9%)
95% to 99% 					 15 (11%)
100% 						  2 ( 1%)

People who answered question: 			140 (84%)
People who skipped question: 			 27 (16%)


People who don't do coverage testing

Testing Survey

survey.gif
Q.11 What percentage of your code is covered (rounding down)?

Answer 						    Count
Don't know, but I do coverage testing 		  6 ( 4%)
Don't do coverage testing 			 47 (34%)
**other** 					  3 ( 2%)
Less than 50% 					 21 (15%)
50% to 74% 					 18 (13%)
75% to 84% 					 16 (11%)
85% to 94% 					 12 ( 9%)
95% to 99% 					 15 (11%)
100% 						  2 ( 1%)

People who answered question: 			140 (84%)
People who skipped question: 			 27 (16%)


People who don't do coverage testing

Testing Survey

survey.gif
Q.11 What percentage of your code is covered (rounding down)?

Answer 						    Count
Don't know, but I do coverage testing 		  6 ( 4%)
Don't do coverage testing 			 47 (34%)
**other** 					  3 ( 2%)
Less than 50% 					 21 (15%)
50% to 74% 					 18 (13%)
75% to 84% 					 16 (11%)
85% to 94% 					 12 ( 9%)
95% to 99% 					 15 (11%)
100% 						  2 ( 1%)

People who answered question: 			140 (84%)
People who skipped question: 			 27 (16%)


People whose coverage testing is less than stellar

Testing Survey

survey.gif
Q.11 What percentage of your code is covered (rounding down)?

Answer 						    Count
Don't know, but I do coverage testing 		  6 ( 4%)
Don't do coverage testing 			 47 (34%)
**other** 					  3 ( 2%)
Less than 50% 					 21 (15%)
50% to 74% 					 18 (13%)
75% to 84% 					 16 (11%)
85% to 94% 					 12 ( 9%)
95% to 99% 					 15 (11%)
100% 						  2 ( 1%)

People who answered question: 			140 (84%)
People who skipped question: 			 27 (16%)


People whose coverage testing is pretty good

Testing Survey

survey.gif
Q.11 What percentage of your code is covered (rounding down)?

Answer 						    Count
Don't know, but I do coverage testing 		  6 ( 4%)
Don't do coverage testing 			 47 (34%)
**other** 					  3 ( 2%)
Less than 50% 					 21 (15%)
50% to 74% 					 18 (13%)
75% to 84% 					 16 (11%)
85% to 94% 					 12 ( 9%)
95% to 99% 					 15 (11%)
100% 						  2 ( 1%)

People who answered question: 			140 (84%)
People who skipped question: 			 27 (16%)


People who are having problems covering certain code constructs

Testing Survey

mary_poppins.jpg
Q.11 What percentage of your code is covered (rounding down)?

Answer 						    Count
Don't know, but I do coverage testing 		  6 ( 4%)
Don't do coverage testing 			 47 (34%)
**other** 					  3 ( 2%)
Less than 50% 					 21 (15%)
50% to 74% 					 18 (13%)
75% to 84% 					 16 (11%)
85% to 94% 					 12 ( 9%)
95% to 99% 					 15 (11%)
100% 						  2 ( 1%)

People who answered question: 			140 (84%)
People who skipped question: 			 27 (16%)


People who are waiting for more coverage metrics

Categories

clouds1.jpg

People who don't do code coverage

just_say_no.jpg

What is code coverage?

cover_me.jpg

Code coverage metrics

measure.jpg

Subroutine coverage

clare_grogan1.jpg

Statement coverage

clare_grogan2.jpg

Branch Coverage

branch_cover1.jpg

Branch Coverage

something_else.jpg

Condition Coverage

exercises.jpg

People who don't do code coverage

just_say_no.jpg

How to use Devel::Cover

dummy.jpg

Example

$ cover -test
Deleting database /home/pjcj/g/perl/shellsource/cover_db
cover: running make test CCFLAGS=-O0\ -fprofile-arcs\ -ftest-coverage
  OTHERLDFLAGS=-fprofile-arcs\ -ftest-coverage
PERL_DL_NONLAZY=1 /usr/local/pkg/base/sw/bin/perl "-MExtUtils::Command::MM"
  "-e" "test_harness(0, 'blib/lib', 'blib/arch')" t/*.t
t/bash......ok
t/csh.......skipped: csh unavailable
t/ksh.......ok
t/sh........ok
t/tcsh......skipped: tcsh unavailable
t/zsh.......ok
All tests successful.
Files=6, Tests=16, 13 wallclock secs ( 0.06 usr  0.00 sys + 12.21 cusr  0.55 csys = 12.82 CPU)
Result: PASS
Reading database from /home/pjcj/g/perl/shellsource/cover_db

---------------------------- ------ ------ ------ ------ ------ ------ ------
File                           stmt   bran   cond    sub    pod   time  total
---------------------------- ------ ------ ------ ------ ------ ------ ------
blib/lib/Shell/Source.pm       96.6   65.0   66.7   90.9    n/a  100.0   86.1
Total                          96.6   65.0   66.7   90.9    n/a  100.0   86.1
---------------------------- ------ ------ ------ ------ ------ ------ ------

Writing HTML output to /home/pjcj/g/perl/shellsource/cover_db/coverage.html ...
done.

Categories

clouds2.jpg

How to improve your Code Coverage

improvement.jpg

Categories

clouds3.jpg

How to improve your coverage even more

more_improvement.jpg

Error Conditions

error.jpg

Error Conditions

big_mistake.jpg

Platform dependent code

oil_rig.jpg

Categories

clouds4.jpg

Shut up!

dixie_chicks.jpg

Uncoverable summary

uc_summary.png

Uncoverable code

waterfall.jpg
#!/usr/bin/perl
my $x = 1;
my $y = 1;
if ($x && !$y) {
    $x++;
    z();
}
if (!$x || !$y) {
    b(); b();
}
sub z {
    $y++;
}

    

    

Uncoverable detail

uc_detail.png

Details

darling_buds.jpg

Reports

report.jpg

Annotations

little_mermaid.jpg

File Selection

selection1.jpg

File Selection

selection2.jpg

File Selection

files.jpg

Categories

clouds5.jpg

Those waiting for more coverage metrics

waiting_for_the_sun.jpg

Path Coverage

path1.jpg

Data Coverage

squares.jpg

Regular Expressions

regular_expressions.png

Stuff you didn't want to know

cycle_uphill.jpg

runops

op_tree.jpg

Back to Reality

guide.jpg

Other Languages

language.jpg

C and XS

xs.jpg

Questions

questions.gif