Building cpancover.com with Docker

Image credit: http://cpancover.com

Olten -- 5th September 2014

Paul Johnson


www.pjcj.net

Devel::Cover

Image credit: http://cpancover.com

cpancover

Image credit: https://web.archive.org/web/20021003051502/http://pjcj.sytes.net/cover/cpancover/cpancover.html

Early days

Image credit: https://web.archive.org/web/20021003151622/http://pjcj.sytes.net/cover/cpancover/AppConfig-1.52/cover_db/cover_db.html

Modules Covered

Image credit: http://ali.as/top100/

Modules Not Covered

Image credit: Mojolicious

Schedule

Schedule

MOAR!

MOAR!

Problems

Problems

Solutions

Solutions

Initial Attempts

Image credit: http://www.ebay.com/itm/Wallace-Sterling-Silver-Fork-1941-Grande-Baroque-Pattern-7-1-2-/160768653655

Solution

Image credit: http://www.clipartbest.com/cliparts/dc7/5dR/dc75dRdc9.jpeg

Docker

Image credit: https://upload.wikimedia.org/wikipedia/commons/2/28/Docker_on_forklift.jpg

Docker

Image credit: https://c2.staticflickr.com/6/5221/5634923791_bcea67a931_z.jpg

Docker

Image credit: http://freshpatrol.com/wp-content/uploads/2013/06/Docker-Mens-Shorts-2.jpg

Docker

Image credit: http://www.jugdo.de/wp-content/uploads/2014/06/docker.png

Docker

Image credit: http://www.adaptivecomputing.com/wp-content/uploads/2014/06/homepage-docker-logo.png

cpancover Container

Image credit: http://blog.insofast.com/wp-content/uploads/2014/02/Freight-Container-sizes.jpg

Basic Local Run 

$ cpancover -local_build Shell::Source

360 sub local_build {
361     my $self = shift;
362 
363     $self->process_module_file;
364     $self->build_modules;
365     $self->add_build_dirs;
366     $self->run_all;
367     $self->generate_html;
368 }

Basic Run in Docker

Basic Run in Docker

Basic Docker Image for Perl

perl-5.20.0/Dockerfile
 1 FROM ubuntu:latest
 2 
 3 MAINTAINER Paul Johnson <paul@pjcj.net>
 4 
 5 ENV TERM xterm
 6 
 7 RUN apt-get update && apt-get -y install wget build-essential
 8 
 9 WORKDIR /usr/local/src
10 RUN wget http://www.cpan.org/src/5.0/perl-5.20.0.tar.gz -O - | tar xzf -
11 RUN cd perl-5.20.0 && ./Configure -des && make install

Required Modules for Devel::Cover

devel-cover-base/Dockerfile
 1 FROM pjcj/perl-5.20.0:latest
 2 
 3 MAINTAINER Paul Johnson <paul@pjcj.net>
 4 
 5 WORKDIR /usr/local/src
 6 RUN apt-get -y install git libssl-dev
 7 
 8 RUN cpan -iTf App::cpanminus
 9 RUN cpanm -n Sereal Digest::MD5 Template Pod::Coverage::CountParents \
10              Capture::Tiny Parallel::Iterator Template Class::XSAccessor \
11              Moo namespace::clean CPAN::Releases::Latest CGI
12 RUN rm -rf ~/.cpan/build ~/.cpan/sources/authors ~/.cpanm \
13            ~/.local/share/.cpan/build ~/.local/share/.cpan/sources/authors

Build Devel::Cover

devel-cover-git/Dockerfile
 1 FROM pjcj/devel-cover-base:latest
 2 
 3 MAINTAINER Paul Johnson <paul@pjcj.net>
 4 
 5 WORKDIR /usr/local/src
 6 RUN git clone https://github.com/pjcj/Devel--Cover.git
 7 WORKDIR /usr/local/src/Devel--Cover
 8 RUN git checkout master
 9 RUN perl -v
10 RUN perl Makefile.PL && make t
11 # RUN make install
12 ENV CPANCOVER_LOCAL 1
13 ENV PATH /usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:./utils

Environment for cpancover

cpancover/Dockerfile
1 FROM pjcj/devel-cover-git:latest
2 
3 MAINTAINER Paul Johnson <paul@pjcj.net>
4 
5 WORKDIR /usr/local/src/Devel--Cover
6 ENV PATH /usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:./utils

Building

1 #!/bin/sh
2 
3 docker build            -t pjcj/perl-5.20.0      perl-5.20.0      && \
4 docker build            -t pjcj/devel-cover-base devel-cover-base && \
5 docker build --no-cache -t pjcj/devel-cover-git  devel-cover-git  && \
6 docker build            -t pjcj/cpancover        cpancover        && \
7 docker push pjcj/cpancover                                        && \
8 echo done

Preparation

Preparation

Running

75         container=$($DOCKER run -d                                 \
76                     --volume=/home/pjcj/g/perl/Devel--Cover:/dc:ro \
77                     --volume=$CPANCOVER_STAGING:/remote_staging:ro \
78                     --workdir=/dc --rm=false --name="$name"        \
79                     --memory=1g                                    \
80                     $DOCKER_IMAGE dc cpancover-build-module $module)
82         $DOCKER wait "$name"

Running

84             $DOCKER logs "$name" > "$staging/$name.out"
85             local_staging="$staging/$name"
86             sudo $DOCKER cp "$name:/staging" "$local_staging"
87             if [ -d "$local_staging" ]; then
88                 sudo chmod -R 755 "$local_staging"
89                 sudo find "$local_staging" -type f -exec chmod 644 {} \;
90                 sudo chown -R pjcj:pjcj "$local_staging"
91                 cd "$local_staging"/*
92                 for f in *; do
93                     if [ -d $f ]; then
94                         rm -rf "$staging"/$f
95                         mv $f "$staging"
96                     fi
97                 done
98                 rm -r "$local_staging"
99             fi

Cleaning Up

101         $DOCKER rm "$name"

Controlling

421     my @command = qw( utils/dc cpancover-docker-module );
423     my @res = iterate_as_array(
424         { workers => $self->workers },
425         sub {
426             my (undef, $module) = @_;

463         },
464         do { my %m; [sort grep !$m{$_}++, @{$self->modules}] }
465     );

Controlling Docker

440             my $name = sprintf("%s-%18.6f", $module, time)
441                          =~ tr/a-zA-Z0-9_./-/cr;
442             say "$dir -> $name";
443             eval {
444                 local $SIG{ALRM} = sub { die "alarm\n" };
445                 alarm $timeout;
446                 system @command, $module, $name;
447                 alarm 0;
448             };
449             if ($@) {
450                 die "propogate: $@" unless $@ eq "alarm\n";
451                 say "Timed out after $timeout seconds!";
452                 $self->sys($self->docker, "kill", $name);
453                 say "Killed docker container $name";
454             }

Modules to Cover

Modules to Cover

Resource Limits

Resource Limits

Tidying Up

Tidying Up

Future

Future

Thanks

Thanks

Questions

Questions