UP | HOME

How Do Parallel Builds Scale?

Most software packages can benefit from parallel builds, e.g., using GNU Make's -j option. A subset of Nixpkgs, part of the NixOS GNU/Linux distribution, was built on a 96-core machine of the Runtime Inria team-project, with -j1, -j2, and so on. The results are summarized below, and per-package details are available on a separate page.

Plots below show the number of packages with a speedup lower than or equal to x, for all the compiled packages (approximately 700), and then for specific subsets.

The scalability of individual build phases is also shown: the configure phase (which corresponds to a ./configure run for packages that use the GNU Build System), the build phase (make), and the check phase (make check or similar). Data for other phases is available but not shown here (for instance configure does not scale—never!—and install is usually short.)

Tools to collect and analyze this information are available as part of Hubble; raw data is available from Hubble's home page. Please email ludovic.courtes@inria.fr for more details.

Table of Contents

1 All Packages

./images/parallel-builds.all.png./images/parallel-builds.all.cumulative.png
./images/parallel-builds.all.build.png./images/parallel-builds.all.build.cumulative.png
./images/parallel-builds.all.check.png./images/parallel-builds.all.check.cumulative.png

2 25% Packages with the Longest Build Times

./images/parallel-builds.longest.png./images/parallel-builds.longest.cumulative.png
./images/parallel-builds.longest.build.png./images/parallel-builds.longest.build.cumulative.png
./images/parallel-builds.longest.check.png./images/parallel-builds.longest.check.cumulative.png

3 25% Packages with the Shortest Build Times

./images/parallel-builds.shortest.png./images/parallel-builds.shortest.cumulative.png
./images/parallel-builds.shortest.build.png./images/parallel-builds.shortest.build.cumulative.png
./images/parallel-builds.shortest.check.png./images/parallel-builds.shortest.check.cumulative.png

4 Packages on the Critical Path

Same for the packages on the critical path. As of Nixpkgs r31312, the critical path when building sequentially (-j1) is the following (sorted from longest to shortest; second column is the build duration in seconds):

/mnt/scratch3/lcourtes/nix/store/gb1n395lf3ld0gqs9sg7s66gkk9i2c7k-openoffice.org-3.2.1  12499.0
/mnt/scratch3/lcourtes/nix/store/3nxadn431ra1li1gz2hp1p6mrvw8yl6l-gcc-4.5.1     5479.0
/mnt/scratch3/lcourtes/nix/store/qgba796qsi2hppx422livlmlg28ncsq4-automake-1.11.1       2055.0
/mnt/scratch3/lcourtes/nix/store/8lli36amw6nmrmaxrf3fw7c0b86y7xlv-glibc-2.12.2  935.0
/mnt/scratch3/lcourtes/nix/store/izq1s6k2b0kn6vay4swym03nz8ijjd7g-autoconf-2.68 869.0
/mnt/scratch3/lcourtes/nix/store/fh21n1g7jsp7410xgl5h4bdk88kak8y4-samba-3.6.1   390.0
/mnt/scratch3/lcourtes/nix/store/1mv2hjj2lw3c0a5yi0bnbhkapdqdn04s-ppl-0.10.2    321.0
/mnt/scratch3/lcourtes/nix/store/kicfbwzig4kdjnjc57ndi4h58qmi36ia-perl-5.10.1   246.0
/mnt/scratch3/lcourtes/nix/store/yp8ac7gvghjrsnqjvr6glg75nfzh83vs-perl-5.10.1   226.0
/mnt/scratch3/lcourtes/nix/store/3k35h92ms17zaqyjz2437136c101k685-gmp-4.3.2     193.0
/mnt/scratch3/lcourtes/nix/store/p53bqrwwyfmg4jfswxvgg46dgd8nkfra-gmp-4.3.2     184.0
/mnt/scratch3/lcourtes/nix/store/28mj3m8l7q8xdys2kqchmqcnrczwjq07-openldap-2.4.13       179.0
/mnt/scratch3/lcourtes/nix/store/kxs0rhafp0gvlyyc3kpzhlhf52wwyh10-binutils-2.21 156.0
/mnt/scratch3/lcourtes/nix/store/kyxfzdir83dqw9zjyvagh25ghbfnaxyw-coreutils-8.7 133.0
/mnt/scratch3/lcourtes/nix/store/0f27m2z559ssf0njsxnp6yyh5fzj1sxk-gnome-vfs-2.24.4      113.0
/mnt/scratch3/lcourtes/nix/store/akh0g2065ca33ak8h6a3zcfgwpvyvyw0-gnum4-1.4.15  87.0
/mnt/scratch3/lcourtes/nix/store/sml8g2a8p67gvcjg8z6djs7qnj6b29fj-gnum4-1.4.15  87.0
/mnt/scratch3/lcourtes/nix/store/41nim5x5bsn8pxpdj56ai3245gv28gcl-libwpd-0.8.14 73.0
/mnt/scratch3/lcourtes/nix/store/ag27i46hsgk33h46hdjcxfcdf0bpicwf-findutils-4.4.2       56.0
/mnt/scratch3/lcourtes/nix/store/yaqvyk2vq0lml6pjslfd5bp62js1awfi-libgsf-1.14.16        44.0
/mnt/scratch3/lcourtes/nix/store/98b03yb1gjsqm349v524qnn552vgygaq-linux-headers-2.6.32.16       33.0
/mnt/scratch3/lcourtes/nix/store/rl72ajfqz4z0di3hd6hjq21jgxhjj3yf-cyrus-sasl-2.1.23     33.0
/mnt/scratch3/lcourtes/nix/store/pfra9a4b6gwcw6b2jp34fxd2fgm38h79-cloog-ppl-0.15.9      22.0
/mnt/scratch3/lcourtes/nix/store/pfac35yhz95y767qyivybwivjz5fx36p-zlib-1.2.5    8.0
/mnt/scratch3/lcourtes/nix/store/ljdnlhd686l3bg4s5lv5w7h7hy5wifhx-bootstrap-tools       6.0
/mnt/scratch3/lcourtes/nix/store/s4jqwkfsfndn8nvl191zzpvjdi5kd5z6-bootstrap-tools.cpio.bz2      0.0
/mnt/scratch3/lcourtes/nix/store/mgsxji51djxc3pydh0csigilx26716cc-bootstrap-gcc-wrapper 0.0
/mnt/scratch3/lcourtes/nix/store/4dkir17f5mff7zx216dqvgf7c8w1lxcf-stdenv-linux-boot     0.0
/mnt/scratch3/lcourtes/nix/store/8c5293ka5qz8xys3z2h09686xyj1jl0m-bootstrap-gcc-wrapper 0.0
/mnt/scratch3/lcourtes/nix/store/2vzy5w5vy6f7cqznnpksks9rnknqb242-stdenv-linux-boot     0.0
/mnt/scratch3/lcourtes/nix/store/a6m2jjdm62hamckqc1b82kxh61im2inr-gcc-wrapper-4.5.1     0.0
/mnt/scratch3/lcourtes/nix/store/0y0bwcmrrax0bzpbscypr9a8kfkqady9-stdenv-linux-boot     0.0
/mnt/scratch3/lcourtes/nix/store/a4kxmlfpzixhcz90g73px9wazx256ayd-stdenv-linux  0.0

total   24427.0
./images/parallel-builds.critical-path.png./images/parallel-builds.critical-path.cumulative.png
./images/parallel-builds.critical-path.build.png./images/parallel-builds.critical-path.build.cumulative.png
./images/parallel-builds.critical-path.check.png./images/parallel-builds.critical-path.check.cumulative.png

Date: 2012-02-21 17:30:00 CET

Author: Ludovic Courtès

Validate XHTML 1.0