Building cpancover.com with Docker

Salzburg -- 11th October 2014

Paul Johnson

www.pjcj.net

Devel::Cover

Code coverage for Perl
Tells you how much of your code you have tested
There are various coverage criteria
A statement is covered if it is executed

cpancover

Needed a way to test Devel::Cover
Can't (properly) run it on itself
There's lots of code on CPAN
Might as well publish results

Early days

Image credit: https://web.archive.org/web/20021003151622/http://pjcj.sytes.net/cover/cpancover/AppConfig-1.52/cover_db/cover_db.html

Modules Covered

Some modules I use
Modules which showed bugs in Devel::Cover
Modules from top 100 lists
http://ali.as/top100/
Requests for modules to be added

Modules Not Covered

Everything else
Anything that caused problems
Anything but the latest stable version

Schedule

Ad hoc
Everything from scratch
Whenever I remembered to do it

MOAR!

But we want all of CPAN!
With different versions
And up to date
With metadata
And links from MetaCPAN

Problems

Need a proper server
Arbitrary code execution
Badly behaved modules
Devel::Cover bugs

Solutions

Bytemark - thanks!
Cover each module in sandbox

Initial Attempts

Image credit: http://www.ebay.com/itm/Wallace-Sterling-Silver-Fork-1941-Grande-Baroque-Pattern-7-1-2-/160768653655

fork, exec, sleep, kill
killpg

Solution

Image credit: http://www.clipartbest.com/cliparts/dc7/5dR/dc75dRdc9.jpeg

Docker

Image credit: https://c2.staticflickr.com/6/5221/5634923791_bcea67a931_z.jpg

Docker

Lightweight virtualisation
Containers
Sandbox
Docker is to packaging as git is to CVS
Docker index is like github

cpancover Container

Image credit: http://blog.insofast.com/wp-content/uploads/2014/02/Freight-Container-sizes.jpg

Each CPAN coverage run in a separate container
Can be run in parallel

Basic Local Run

$ cpancover -local_build Shell::Source

360 sub local_build {
361     my $self = shift;
362
363     $self->process_module_file;
364     $self->build_modules;
365     $self->add_build_dirs;
366     $self->run_all;
367     $self->generate_html;
368 }

Basic Run in Docker

Create docker image
Do basic run in docker
Extract results

Basic Docker Image for Perl

perl-5.20.0/Dockerfile

1 FROM ubuntu:latest
2
3 MAINTAINER Paul Johnson <paul@pjcj.net>
4
5 ENV TERM xterm
6
7 RUN apt-get update && apt-get -y install wget build-essential
8
9 WORKDIR /usr/local/src
10 RUN wget http://www.cpan.org/src/5.0/perl-5.20.0.tar.gz -O - | tar xzf -
11 RUN cd perl-5.20.0 && ./Configure -des && make install

Required Modules for Devel::Cover

devel-cover-base/Dockerfile

1 FROM pjcj/perl-5.20.0:latest
2
3 MAINTAINER Paul Johnson <paul@pjcj.net>
4
5 WORKDIR /usr/local/src
6 RUN apt-get -y install git libssl-dev
7
8 RUN cpan -iTf App::cpanminus
9 RUN cpanm -n Sereal Digest::MD5 Template Pod::Coverage::CountParents \
10              Capture::Tiny Parallel::Iterator Template Class::XSAccessor \
11              Moo namespace::clean CPAN::Releases::Latest CGI
12 RUN rm -rf ~/.cpan/build ~/.cpan/sources/authors ~/.cpanm \
13            ~/.local/share/.cpan/build ~/.local/share/.cpan/sources/authors

Build Devel::Cover

devel-cover-git/Dockerfile

1 FROM pjcj/devel-cover-base:latest
2
3 MAINTAINER Paul Johnson <paul@pjcj.net>
4
5 WORKDIR /usr/local/src
6 RUN git clone https://github.com/pjcj/Devel--Cover.git
7 WORKDIR /usr/local/src/Devel--Cover
8 RUN git checkout master
9 RUN perl -v
10 RUN perl Makefile.PL && make t
11 # RUN make install
12 ENV CPANCOVER_LOCAL 1
13 ENV PATH /usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:./utils

Environment for cpancover

cpancover/Dockerfile

1 FROM pjcj/devel-cover-git:latest
2
3 MAINTAINER Paul Johnson <paul@pjcj.net>
4
5 WORKDIR /usr/local/src/Devel--Cover
6 ENV PATH /usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:./utils

Building

1 #!/bin/sh
2
3 docker build            -t pjcj/perl-5.20.0      perl-5.20.0      && \
4 docker build            -t pjcj/devel-cover-base devel-cover-base && \
5 docker build --no-cache -t pjcj/devel-cover-git  devel-cover-git  && \
6 docker build            -t pjcj/cpancover        cpancover        && \
7 docker push pjcj/cpancover                                        && \
8 echo done

Preparation

Pull the images
$ docker pull pjcj/cpancover

Running

75         container=$($DOCKER run -d                                 \
76                     --volume=/home/pjcj/g/perl/Devel--Cover:/dc:ro \
77                     --volume=$CPANCOVER_STAGING:/remote_staging:ro \
78                     --workdir=/dc --rm=false --name="$name"        \
79                     --memory=1g                                    \
80                     $DOCKER_IMAGE dc cpancover-build-module $module)
82         $DOCKER wait "$name"

Copying Results

84             $DOCKER logs "$name" > "$staging/$name.out"
85             local_staging="$staging/$name"
86             sudo $DOCKER cp "$name:/staging" "$local_staging"
87             if [ -d "$local_staging" ]; then
88                 sudo chmod -R 755 "$local_staging"
89                 sudo find "$local_staging" -type f -exec chmod 644 {} \;
90                 sudo chown -R pjcj:pjcj "$local_staging"
91                 cd "$local_staging"/*
92                 for f in *; do
93                     if [ -d $f ]; then
94                         rm -rf "$staging"/$f
95                         mv $f "$staging"
96                     fi
97                 done
98                 rm -r "$local_staging"
99             fi

Cleaning Up

101 $DOCKER rm "$name"

Controlling

421     my @command = qw( utils/dc cpancover-docker-module );
423     my @res = iterate_as_array(
424         { workers => $self->workers },
425         sub {
426             my (undef, $module) = @_;

463         },
464         do { my %m; [sort grep !$m{$_}++, @{$self->modules}] }
465     );

Controlling Docker

440             my $name = sprintf("%s-%18.6f", $module, time)
441                          =~ tr/a-zA-Z0-9_./-/cr;
442             say "$dir -> $name";
443             eval {
444                 local $SIG{ALRM} = sub { die "alarm\n" };
445                 alarm $timeout;
446                 system @command, $module, $name;
447                 alarm 0;
448             };
449             if ($@) {
450                 die "propogate: $@" unless $@ eq "alarm\n";
451                 say "Timed out after $timeout seconds!";
452                 $self->sys($self->docker, "kill", $name);
453                 say "Killed docker container $name";
454             }

Modules to Cover

CPAN::Releases::Latest by Neil Bowers
Skip covered and failed modules

Resource Limits

Memory and swap accounting
Add command-line parameters to kernel
Edit /etc/default/grub
GRUB_CMDLINE_LINUX="cgroup_enable=memory swapaccount=1"

Tidying Up

Create top-level HTML
Create JSON
Compress HTML and JSON
Loop and sleep

Future

Much better front end
Something dynamic - lightweight framework
Accept coverage data from other users

Building cpancover.com with Docker

www.pjcj.net

Building cpancover.com with Docker

Salzburg -- 11th October 2014

Paul Johnson

www.pjcj.net

Devel::Cover

cpancover

Early days

Modules Covered

Modules Not Covered

Schedule

MOAR!

Problems

Solutions

Initial Attempts

Solution

Docker

Docker

Docker

Docker

Docker

cpancover Container

Basic Local Run

Basic Run in Docker

Basic Docker Image for Perl

Required Modules for Devel::Cover

Build Devel::Cover

Environment for cpancover

Building

Preparation

Running

Copying Results

Cleaning Up

Controlling

Controlling Docker

Modules to Cover

Resource Limits

Tidying Up

Future

Thanks

Questions