[Solved] distcc: only localhost

Soukyuu · 2016-03-15 17:56:18

I followed the instructions on the wiki to set up my PCs as a distcc cluster.

On the master PC (2 cores, 192.168.178.2), I changed the makepkg.conf to this:

#
# /etc/makepkg.conf
#

#########################################################################
# SOURCE ACQUISITION
#########################################################################
#
#-- The download utilities that makepkg should use to acquire sources
#  Format: 'protocol::agent'
DLAGENTS=('ftp::/usr/bin/curl -fC - --ftp-pasv --retry 3 --retry-delay 3 -o %o %u'
          'http::/usr/bin/curl -fLC - --retry 3 --retry-delay 3 -o %o %u'
          'https::/usr/bin/curl -fLC - --retry 3 --retry-delay 3 -o %o %u'
          'rsync::/usr/bin/rsync --no-motd -z %u %o'
          'scp::/usr/bin/scp -C %u %o')

# Other common tools:
# /usr/bin/snarf
# /usr/bin/lftpget -c
# /usr/bin/wget

#-- The the package required by makepkg to download VCS sources
#  Format: 'protocol::package'
VCSCLIENTS=('bzr::bzr'
            'git::git'
            'hg::mercurial'
            'svn::subversion')

#########################################################################
# ARCHITECTURE, COMPILE FLAGS
#########################################################################
#
CARCH="x86_64"
CHOST="x86_64-unknown-linux-gnu"

#-- Compiler and Linker Flags
# -march (or -mcpu) builds exclusively for an architecture
# -mtune optimizes for an architecture, but builds for whole processor family
CPPFLAGS="-D_FORTIFY_SOURCE=2"
CFLAGS="-march=native -O2 -pipe -fstack-protector --param=ssp-buffer-size=4 -D_FORTIFY_SOURCE=2"
CXXFLAGS="${CFLAGS}"
LDFLAGS="-Wl,-O1,--sort-common,--as-needed,-z,relro"
#-- Make Flags: change this for DistCC/SMP systems
MAKEFLAGS="-j8"
#-- Debugging flags
DEBUG_CFLAGS="-g -fvar-tracking-assignments"
DEBUG_CXXFLAGS="-g -fvar-tracking-assignments"

#########################################################################
# BUILD ENVIRONMENT
#########################################################################
#
# Defaults: BUILDENV=(!distcc color !ccache check !sign)
#  A negated environment option will do the opposite of the comments below.
#
#-- distcc:   Use the Distributed C/C++/ObjC compiler
#-- color:    Colorize output messages
#-- ccache:   Use ccache to cache compilation
#-- check:    Run the check() function if present in the PKGBUILD
#-- sign:     Generate PGP signature file
#
BUILDENV=(distcc color !ccache check !sign)
#
#-- If using DistCC, your MAKEFLAGS will also need modification. In addition,
#-- specify a space-delimited list of hosts running in the DistCC cluster.
DISTCC_HOSTS="192.168.178.4"
#
#-- Specify a directory for package building.
BUILDDIR=/tmp/makepkg

#########################################################################
# GLOBAL PACKAGE OPTIONS
#   These are default values for the options=() settings
#########################################################################
#
# Default: OPTIONS=(strip docs !libtool !staticlibs emptydirs zipman purge !upx !debug)
#  A negated option will do the opposite of the comments below.
#
#-- strip:      Strip symbols from binaries/libraries
#-- docs:       Save doc directories specified by DOC_DIRS
#-- libtool:    Leave libtool (.la) files in packages
#-- staticlibs: Leave static library (.a) files in packages
#-- emptydirs:  Leave empty directories in packages
#-- zipman:     Compress manual (man and info) pages in MAN_DIRS with gzip
#-- purge:      Remove files specified by PURGE_TARGETS
#-- upx:        Compress binary executable files using UPX
#-- debug:      Add debugging flags as specified in DEBUG_* variables
#
OPTIONS=(strip docs !libtool !staticlibs emptydirs zipman purge !upx !debug)

#-- File integrity checks to use. Valid: md5, sha1, sha256, sha384, sha512
INTEGRITY_CHECK=(md5)
#-- Options to be used when stripping binaries. See `man strip' for details.
STRIP_BINARIES="--strip-all"
#-- Options to be used when stripping shared libraries. See `man strip' for details.
STRIP_SHARED="--strip-unneeded"
#-- Options to be used when stripping static libraries. See `man strip' for details.
STRIP_STATIC="--strip-debug"
#-- Manual (man and info) directories to compress (if zipman is specified)
MAN_DIRS=({usr{,/local}{,/share},opt/*}/{man,info})
#-- Doc directories to remove (if !docs is specified)
DOC_DIRS=(usr/{,local/}{,share/}{doc,gtk-doc} opt/*/{doc,gtk-doc})
#-- Files to be removed from all packages (if purge is specified)
PURGE_TARGETS=(usr/{,share}/info/dir .packlist *.pod)

#########################################################################
# PACKAGE OUTPUT
#########################################################################
#
# Default: put built package and cached source in build directory
#
#-- Destination: specify a fixed directory where all packages will be placed
PKGDEST=/home/azure/aur
#-- Source cache: specify a fixed directory where source files will be cached
#SRCDEST=/home/sources
#-- Source packages: specify a fixed directory where all src packages will be placed
#SRCPKGDEST=/home/srcpackages
#-- Log files: specify a fixed directory where all log files will be placed
#LOGDEST=/home/makepkglogs
#-- Packager: name/email of the person or organization building packages
#PACKAGER="John Doe <john@doe.com>"
#-- Specify a key to use for package signing
#GPGKEY=""

#########################################################################
# COMPRESSION DEFAULTS
#########################################################################
#
COMPRESSGZ=(gzip -c -f -n)
COMPRESSBZ2=(bzip2 -c -f)
COMPRESSXZ=(xz -c -z -)
COMPRESSLRZ=(lrzip -q)
COMPRESSLZO=(lzop -q)
COMPRESSZ=(compress -c -f)

#########################################################################
# EXTENSION DEFAULTS
#########################################################################
#
# WARNING: Do NOT modify these variables unless you know what you are
#          doing.
#
PKGEXT='.pkg.tar.xz'
SRCEXT='.src.tar.gz'

# vim: set ft=sh ts=2 sw=2 et:

On the slave PC (2 cores, 192.168.178.4), I edited the /etc/conf.d/distccd as follows:

DISTCC_ARGS="--allow 192.168.178.0/24"

When running "makepkg -sri" for, say, qbittorrent or wine-git, I only ever see 4 localhost entries in distccmon-text. The ~./distcc/state folder only ever contains stuff locally, and the ~./distcc/lock has a lockfile for my slave PC.
I've tried disabling iptables on both machines just in case, no difference.

Any idea what's wrong?

Last edited by Soukyuu (2016-05-14 16:29:57)

2ManyDogs · 2016-03-15 18:29:15

You did not tell distcc how many threads the slave should use (in DISTCC_HOSTS), but that appears to be optional. Are you sure you have the distccd service running on the slave?

Last edited by 2ManyDogs (2016-03-15 18:35:30)

Soukyuu · 2016-03-15 19:18:01

Yes, I can also see a couple of distcc processes running on the slave.
I've looked at journalctl and there seem to be one entry in there:

distccd[27876]: (dcc_job_summary) client: 192.168.178.2:53408 COMPILE_OK exit:0 sig:0 core:0 ret:0 time:15ms cc /tmp/cgAZXKrv/dummy.c

But it's the only entry for the whole compiling process. Did I expect too much out of distcc? 15ms "improvement" seem rather useless.

edit:

● distccd.service - A distributed C/C++ compiler
   Loaded: loaded (/usr/lib/systemd/system/distccd.service; disabled; vendor preset: disabled)
   Active: active (running) since Di 2016-03-15 18:01:55 CET; 2h 21min ago
     Docs: man:distccd(1)
 Main PID: 27836 (distccd)
    Tasks: 7 (limit: 512)
   CGroup: /system.slice/distccd.service
           ├─27836 /usr/bin/distccd --no-detach --daemon --allow 192.168.178.0/24
           ├─27838 /usr/bin/distccd --no-detach --daemon --allow 192.168.178.0/24
           ├─27842 /usr/bin/distccd --no-detach --daemon --allow 192.168.178.0/24
           ├─27844 /usr/bin/distccd --no-detach --daemon --allow 192.168.178.0/24
           ├─27846 /usr/bin/distccd --no-detach --daemon --allow 192.168.178.0/24
           ├─27874 /usr/bin/distccd --no-detach --daemon --allow 192.168.178.0/24
           └─27876 /usr/bin/distccd --no-detach --daemon --allow 192.168.178.0/24

Last edited by Soukyuu (2016-03-15 19:24:35)

Soukyuu · 2016-04-24 19:07:13

Well, it still not really working. I just compiled qbittorrent on my server and everything that I saw on my main machine with distccd running is

Apr 24 20:55:56 Takemikazuchi distccd[532]: (dcc_job_summary) client: 192.168.178.2:37434 COMPILE_OK exit:0 sig:0 core:0 ret:0 time:14ms cc /tmp/cgEeargC/dummy.c
Apr 24 20:58:03 Takemikazuchi distccd[545]: (dcc_job_summary) client: 192.168.178.2:37538 COMPILE_OK exit:0 sig:0 core:0 ret:0 time:22ms cc /tmp/cgxLqNJc/dummy.c
Apr 24 21:00:10 Takemikazuchi distccd[560]: (dcc_job_summary) client: 192.168.178.2:37638 COMPILE_OK exit:0 sig:0 core:0 ret:0 time:22ms cc /tmp/cg7GIM4Q/dummy.c
Apr 24 21:02:08 Takemikazuchi distccd[593]: (dcc_job_summary) client: 192.168.178.2:37724 COMPILE_OK exit:0 sig:0 core:0 ret:0 time:19ms cc /tmp/cgeOhVya/dummy.c
Apr 24 21:02:45 Takemikazuchi distccd[630]: (dcc_job_summary) client: 192.168.178.2:37762 COMPILE_OK exit:0 sig:0 core:0 ret:0 time:13ms cc /tmp/cgl1di9u/dummy.c

That doesn't look as if it's doing anything, except maybe checking connection by letting it compile a dummy.c file?

edit: also, I added the number of threads for each slave in makepkg.conf like this:

DISTCC_HOSTS="192.168.178.4/4 localhost/0"

It STILL uses only four localhost threads.

Last edited by Soukyuu (2016-04-24 19:29:16)

graysky · 2016-04-24 19:36:32

Yes, I wrote most of that article back in 2011... it works. A better test is something more intensive like the kernel; try compiling it with your setup. I recommend that you edit /etc/makepkg.conf to define the number of threads you want. Here is how I do it on mine:

% grep DIST/etc/makepkg.distcc 
DISTCC_HOSTS="localhost/9 --localslots_cpp=16 wasted/9 tripwire/5 relic/5"

% grep MAKEFLAG /etc/makepkg.distcc 
MAKEFLAGS="-j40"

Note that you can have two /etc/makepkg.conf files... I keep the DISTCC enabled one as /etc/makepkg.distcc and you can use it passing the --config switch it like this:

makepkg -src --config /etc/makepkg.distcc

When you're building, ssh into your slave box and run htop once the kernel gets going. Should be serving up jobs.

Last edited by graysky (2016-04-24 19:37:44)

Soukyuu · 2016-04-26 16:55:51

I will try compiling a kernel this weekend and see how it goes.

Though if it only kicks in on "heavier" stuff, then maybe I'm using the wrong tool. The Zacate based box is so ultra slow, compiling qbittorrent is already a pain, so I wanted to offload as much as possible onto my main machine. Maybe there is no way except actually cross-compiling in this case.

2ManyDogs · 2016-04-26 17:15:59

Have you tried using distcc's pump mode? That is supposed to push more work to the slave box.

graysky · 2016-04-26 18:56:37

@2MD - I have yet to be able to get pump mode to work with makepkg. If you can, please update the distcc wiki article.

Soukyuu · 2016-04-28 17:16:07

@2ManyDogs: If you mean running "pump makepkg -sri", yes. The result is the same as without it, though, only localhost entries. I still didn't have the time to tinker with it more, so will write back later.

Soukyuu · 2016-05-14 15:04:21

I think I know what I was missing - the "allow" part. The thing is, I took that paragraph as me having to enter the allowed clients in /etc/distcc/hosts. The same directory holds a "clients.allow" file, this is where I added my hosts. Now it works and I can see my slaves fully loaded. Oh, and the link to the distcc manual is a 404.

Bonus questions: the hosts are specified in the following order: "localhost/4 192.168.178.2/2" Why is distccmon listing the 192.168.178.2 slave first on distccmon-gnome? Does more "whitespace" on the tasks graph mean the CPU is resting more?

Bonus questions part2: I've already run into problems with mixed gcc versions across the slaves, so I know I should avoid that. Reading the gentoo wiki on distcc, it also warns against using -march=native. Is this also valid for makepkg on archlinux?

edit: it doesn't make sense. On my other PC I have "192.168.178.4/4 localhost/2" in makepkg, but distccmon-gnome is showing me that it's using TWO slots on the slave and FOUR on localhost?! Strangely, setting "192.168.178.4/4 localhost/0" works, no slots are used on the local machine.

Last edited by Soukyuu (2016-05-14 15:54:41)

graysky · 2016-05-14 15:58:14

Soukyuu wrote:

I think I know what I was missing - the "allow" part. The thing is, I took that paragraph as me having to enter the allowed clients in /etc/distcc/hosts. The same directory holds a "clients.allow" file, this is where I added my hosts. Now it works and I can see my slaves fully loaded. Oh, and the link to the distcc manual is a 404.

My /etc/distcc/hosts contains only 127.0.0.1 and distcc works as expected. I haven't tried pump mode though...

Soukyuu wrote:

Bonus questions: the hosts are specified in the following order: "localhost/4 192.168.178.2/2" Why is distccmon listing the 192.168.178.2 slave first on distccmon-gnome? Does more "whitespace" on the tasks graph mean the CPU is resting more?

Dunno for sure, but perhaps they are listed alpha/numerically? I don't use the gtk client so dunno about white space.

Soukyuu wrote:

On my other PC I have "192.168.178.4/4 localhost/2" in makepkg, but distccmon-gnome is showing me that it's using TWO slots on the slave and FOUR on localhost?! Strangely, setting "192.168.178.4/4 localhost/0" works, no slots are used on the local machine.

/etc/makepkg.conf doesn't matter on slave machines.

Soukyuu · 2016-05-14 16:25:48

graysky wrote:
Soukyuu wrote:
I think I know what I was missing - the "allow" part. The thing is, I took that paragraph as me having to enter the allowed clients in /etc/distcc/hosts. The same directory holds a "clients.allow" file, this is where I added my hosts. Now it works and I can see my slaves fully loaded. Oh, and the link to the distcc manual is a 404.
My /etc/distcc/hosts contains only 127.0.0.1 and distcc works as expected. I haven't tried pump mode though...
I think I know what was wrong. distcc doesn't seem to be working correctly if cmake is involved. Just tried compiling pcsx2 and it doesn't appear in distccmon at all - but the compilation fails with weird c++ abi errors even though both have the same compiler. Disabling distcc fixes the compilation errors. So editing clients.allow wasn't required after all.
Soukyuu wrote:
On my other PC I have "192.168.178.4/4 localhost/2" in makepkg, but distccmon-gnome is showing me that it's using TWO slots on the slave and FOUR on localhost?! Strangely, setting "192.168.178.4/4 localhost/0" works, no slots are used on the local machine.
/etc/makepkg.conf doesn't matter on slave machines.

I worded that badly. I meant that compiling from the other machine with those hosts set uses two slots on the slave (even though I told it to use four) and four on the local machine (even though I told it to use two). I'm using 8 jobs for make.

Arch Linux

#1 2016-03-15 17:56:18

[Solved] distcc: only localhost

#2 2016-03-15 18:29:15

Re: [Solved] distcc: only localhost

#3 2016-03-15 19:18:01

Re: [Solved] distcc: only localhost

#4 2016-04-24 19:07:13

Re: [Solved] distcc: only localhost

#5 2016-04-24 19:36:32

Re: [Solved] distcc: only localhost

#6 2016-04-26 16:55:51

Re: [Solved] distcc: only localhost

#7 2016-04-26 17:15:59

Re: [Solved] distcc: only localhost

#8 2016-04-26 18:56:37

Re: [Solved] distcc: only localhost

#9 2016-04-28 17:16:07

Re: [Solved] distcc: only localhost

#10 2016-05-14 15:04:21

Re: [Solved] distcc: only localhost

#11 2016-05-14 15:58:14

Re: [Solved] distcc: only localhost

#12 2016-05-14 16:25:48

Re: [Solved] distcc: only localhost

Board footer