You are not logged in.

#1 2020-12-19 12:24:09

graysky
Wiki Maintainer
From: :wq
Registered: 2008-12-01
Posts: 10,595
Website

ccache not working when compiling identical source

I tried ccache to compile the bzImage target of the linux 5.9.14 kernel but am finding that there is no functional benefit in terms of speed up.

Setup:

export CCACHE_DIR=/scratch/.ccache
export PATH="/usr/lib/ccache/bin/:$PATH"
ccache --set-config=sloppiness=file_macro,locale,time_macros
cd /path/to/perconfigured/kernel_source

The first run populates the cache which it did as I saw based on the output of `ccache -s`.

make clean
make -j9 bzImage

I repeated this several times but found that the time to compile did not shorten and that the number of cache hits was very low.  Based on what I read, I must be doing something incorrectly.  Thoughts are welcomed.

The time run initially vs the first repeat is more or less identical.  This is just above make clean and make bzImage all done in the same directory:

make -j9 bzImage  1501.90s user 155.42s system 737% cpu 3:44.81 total
make -j9 bzImage  1482.42s user 154.03s system 741% cpu 3:40.78 total

Before the last run, I zeroed the ccache stats (`ccache -z`) then, when the run finished, ran `ccache -s` which is here:

cache directory                     /scratch/.ccache
primary config                      /scratch/.ccache/ccache.conf
secondary config      (readonly)    /etc/ccache.conf
stats updated                       Sat Dec 19 07:21:55 2020
stats zeroed                        Sat Dec 19 07:16:55 2020
cache hit (direct)                   208
cache hit (preprocessed)               3
cache miss                          2489
cache hit rate                      7.81 %
called for link                       24
called for preprocessing              13
preprocessor error                    12
unsupported code directive             2
no input file                          7
cleanups performed                     0
files in cache                     30855
cache size                         621.4 MB
max cache size                       5.0 GB

CPU-optimized Linux-ck packages @ Repo-ck  • AUR packagesZsh and other configs

Offline

#2 2020-12-20 13:45:23

Lone_Wolf
Member
From: Netherlands, Europe
Registered: 2005-10-04
Posts: 11,868

Re: ccache not working when compiling identical source

I use ccache through makepkg.conf and do see apporx 60% cache hit rate and notice speedups but build  only a low number of pacakges frequently.
So I used asp to get the files for linux 5.10.1.arch1-1  (currently in Testing) .

Zeroed ccache stat, used makepkg --verifysource to get the sources (downloading  a git repo of 2+ Gib takes time).

I've started the first run, will post results later.

first run

==> Making package: linux 5.10.1.arch1-1 (zo 20 dec 2020 14:25:07 CET)
==> Finished making: linux 5.10.1.arch1-1 (zo 20 dec 2020 14:57:45 CET)

cache hit (direct)                   115
cache hit (preprocessed)               0
cache miss                         20460
cache hit rate                      0.56 %
called for link                       38
called for preprocessing              27
preprocessor error                    35
unsupported code directive             7
no input file                         19
cleanups performed                    29
files in cache                    166090
cache size                          12.7 GB
max cache size                      14.0 GB

time needed 32:38 and as you can see ccache did already help a little bit .

starting 2nd run.

==> Making package: linux 5.10.1.arch1-1 (zo 20 dec 2020 15:16:16 CET)
==> Finished making: linux 5.10.1.arch1-1 (zo 20 dec 2020 15:47:35 CET)

cache hit (direct)                   494
cache hit (preprocessed)               6
cache miss                         40650
cache hit rate                      1.22 %
called for link                       76
called for preprocessing              54
preprocessor error                    70
unsupported code directive            14
no input file                         38
cleanups performed                    68
files in cache                    165088
cache size                          13.2 GB
max cache size                      14.0 GB

time 31:19 so not much shorter.
the number of cleanups needed to be done by ccache to keep cache below max : 39

Going to do a 3rd run.

==> Making package: linux 5.10.1.arch1-1 (zo 20 dec 2020 15:53:46 CET)
==> Finished making: linux 5.10.1.arch1-1 (zo 20 dec 2020 16:25:53 CET)

cache hit (direct)                   873
cache hit (preprocessed)              12
cache miss                         60840
cache hit rate                      1.43 %
called for link                      114
called for preprocessing              81
preprocessor error                   105
unsupported code directive            21
no input file                         57
cleanups performed                   104
files in cache                    158415
cache size                          13.2 GB
max cache size                      14.0 GB

time 32:07 , number of cleanups increased by 36 .


4th run

==> Making package: linux 5.10.1.arch1-1 (zo 20 dec 2020 16:33:07 CET)
==> Finished making: linux 5.10.1.arch1-1 (zo 20 dec 2020 17:04:58 CET)

cache hit (direct)                  1252
cache hit (preprocessed)              18
cache miss                         81030
cache hit rate                      1.54 %
called for link                      152
called for preprocessing             108
preprocessor error                   140
unsupported code directive            28
no input file                         76
cleanups performed                   144
files in cache                    157039
cache size                          13.0 GB
max cache size                      14.0 GB

time 31:51 , cleanups 40

It does look like build system of linux kernel doesn't cooperate well with ccache.

Stopping for now.

Last edited by Lone_Wolf (2020-12-20 16:09:42)


Disliking systemd intensely, but not satisfied with alternatives so focusing on taming systemd.


(A works at time B)  && (time C > time B ) ≠  (A works at time C)

Offline

#3 2020-12-24 12:12:14

Lone_Wolf
Member
From: Netherlands, Europe
Registered: 2005-10-04
Posts: 11,868

Re: ccache not working when compiling identical source

I ran some more tests with 2 building runs for multiple packages.
When switching to another package I zeroed stats and cleared the cache .

Apart from cache size no changes were made to ccache settings.
The time gain in the second run varies, but is substantial.


llvm-minimal-git
first_run 28:31
2nd_run 03:51

mesa-minimal-git
06:35
05:34

openmw-git
04:27
00:29

0ad
13:32
01:37

libreoffice-still
45:36
24:36

llvm-minimal-git my aur package , built with --nocheck 

==> Making package: llvm-minimal-git 12.0.0_r375563.631501b1f90e-1 (wo 23 dec 2020 19:30:33 CET)
==> Finished making: llvm-minimal-git 12.0.0_r375564.7ad666798f12-1 (wo 23 dec 2020 19:59:04 CET)

==> Making package: llvm-minimal-git 12.0.0_r375564.7ad666798f12-1 (wo 23 dec 2020 20:09:51 CET)
==> Finished making: llvm-minimal-git 12.0.0_r375564.7ad666798f12-1 (wo 23 dec 2020 20:13:42 CET)

cache hit (direct)                  4137
cache hit (preprocessed)             442
cache miss                          4571
cache hit rate                     50.04 %
called for link                      578
compile failed                        14
preprocessor error                    80
unsupported code directive             4
no input file                         10
cleanups performed                     0
files in cache                     13259
cache size                         970.3 MB
max cache size                      14.0 GB

---------------------------------------------------------------------------------------
mesa-minimal-git (personal package, not yet uploaded)

==> Making package: mesa-minimal-git 21.0.0_devel.132396.b34bc3db674-1 (wo 23 dec 2020 20:24:31 CET)
==> Finished making: mesa-minimal-git 21.0.0_devel.132628.eb7e20b378b-1 (wo 23 dec 2020 20:31:06 CET)

==> Making package: mesa-minimal-git 21.0.0_devel.132628.eb7e20b378b-1 (wo 23 dec 2020 20:49:00 CET)
==> Finished making: mesa-minimal-git 21.0.0_devel.132628.eb7e20b378b-1 (wo 23 dec 2020 20:54:34 CET)

cache hit (direct)                  1062
cache hit (preprocessed)             472
cache miss                          1534
cache hit rate                     50.00 %
called for link                       42
called for preprocessing              12
no input file                         22
cleanups performed                     0
files in cache                      4941
cache size                         246.8 MB
max cache size                      14.0 GB

-------------------------------------
openmw-git my personal variant of aur openmw-git

==> Making package: openmw-git 1:0.47.0.r24486.a1065c837-1 (wo 23 dec 2020 21:00:35 CET)
==> Finished making: openmw-git 1:0.47.0.r24510.fd5429ae5-1 (wo 23 dec 2020 21:05:02 CET)

==> Making package: openmw-git 1:0.47.0.r24510.fd5429ae5-1 (wo 23 dec 2020 21:10:36 CET)
==> Finished making: openmw-git 1:0.47.0.r24510.fd5429ae5-1 (wo 23 dec 2020 21:11:05 CET)

cache hit (direct)                  1037
cache hit (preprocessed)               9
cache miss                          1054
cache hit rate                     49.81 %
called for link                       36
no input file                          4
cleanups performed                     0
files in cache                      3148
cache size                         132.5 MB
max cache size                      14.0 GB

----------------------------------------------------------------------
community/0ad

==> Making package: 0ad a23.1-14 (wo 23 dec 2020 22:29:12 CET)
==> Finished making: 0ad a23.1-14 (wo 23 dec 2020 22:42:44 CET)

==> Making package: 0ad a23.1-14 (wo 23 dec 2020 22:44:35 CET)
==> Finished making: 0ad a23.1-14 (wo 23 dec 2020 22:46:12 CET)

cache hit (direct)                  1319
cache hit (preprocessed)             226
cache miss                          1471
cache hit rate                     51.23 %
called for link                       68
called for preprocessing              72
compile failed                        16
preprocessor error                    20
bad compiler arguments                 4
autoconf compile/link                224
no input file                         26
cleanups performed                     0
files in cache                      4932
cache size                           1.4 GB
max cache size                      14.0 GB

----------------------------------------------
extra/libreoffice-still

==> Making package: libreoffice-still 6.4.7-8 (wo 23 dec 2020 23:05:42 CET)
==> Finished making: libreoffice-still 6.4.7-8 (wo 23 dec 2020 23:51:18 CET)

==> Making package: libreoffice-still 6.4.7-8 (do 24 dec 2020 00:10:58 CET)
==> Finished making: libreoffice-still 6.4.7-8 (do 24 dec 2020 00:25:34 CET)

cache hit (direct)                 10150
cache hit (preprocessed)              12
cache miss                         10150
cache hit rate                     50.03 %
called for link                      762
called for preprocessing             262
compile failed                        42
preprocessor error                   190
bad compiler arguments                62
autoconf compile/link                586
no input file                         72
cleanups performed                     0
files in cache                     21060
cache size                           1.3 GB
max cache size                      14.0 GB

Disliking systemd intensely, but not satisfied with alternatives so focusing on taming systemd.


(A works at time B)  && (time C > time B ) ≠  (A works at time C)

Offline

#4 2021-02-10 13:11:24

Lone_Wolf
Member
From: Netherlands, Europe
Registered: 2005-10-04
Posts: 11,868

Re: ccache not working when compiling identical source

ccache 4.2 is now in community , new test run.

preparation

$ ccache --zero-stats --clear
Statistics zeroed
Clearing... 100.0% [===================================================================================================]
$ ccache --show-stats 
cache directory                     /home/panoramix/.ccache
primary config                      /home/panoramix/.ccache/ccache.conf
secondary config (readonly)         /etc/ccache.conf
stats updated                       Wed Feb 10 12:44:57 2021
stats zeroed                        Wed Feb 10 12:44:53 2021
cache hit (direct)                     0
cache hit (preprocessed)               0
cache miss                             0
cache hit rate                      0.00 %
cleanups performed                    16
files in cache                         0
cache size                           0.0 kB
max cache size                      14.0 GB
$ 

My ~/.ccache/ccache.conf

max_size = 14
sloppiness = file_macro,locale,time_macros

first run results

==> Making package: linux 5.10.14.arch1-1 (wo 10 feb 2021 12:49:21 CET)
==> Finished making: linux 5.10.14.arch1-1 (wo 10 feb 2021 13:22:28 CET)

$ ccache --show-stats 
cache directory                     /home/panoramix/.ccache
primary config                      /home/panoramix/.ccache/ccache.conf
secondary config (readonly)         /etc/ccache.conf
stats updated                       Wed Feb 10 13:19:25 2021
stats zeroed                        Wed Feb 10 12:44:53 2021
cache hit (direct)                   115
cache hit (preprocessed)               0
cache miss                         20493
cache hit rate                      0.56 %
called for link                       38
called for preprocessing              27
preprocessor error                    35
unsupported code directive             7
no input file                         19
cleanups performed                    16
files in cache                     40965
cache size                           2.4 GB
max cache size                      14.0 GB
$ 

2nd run

==> Making package: linux 5.10.14.arch1-1 (wo 10 feb 2021 13:35:40 CET)
==> Finished making: linux 5.10.14.arch1-1 (wo 10 feb 2021 14:08:38 CET)

$ ccache --show-stats 
cache directory                     /home/panoramix/.ccache
primary config                      /home/panoramix/.ccache/ccache.conf
secondary config (readonly)         /etc/ccache.conf
stats updated                       Wed Feb 10 14:05:33 2021
stats zeroed                        Wed Feb 10 12:44:53 2021
cache hit (direct)                   494
cache hit (preprocessed)               6
cache miss                         40716
cache hit rate                      1.21 %
called for link                       76
called for preprocessing              54
preprocessor error                    70
unsupported code directive            14
no input file                         38
cleanups performed                    16
files in cache                     81398
cache size                           4.8 GB
max cache size                      14.0 GB
$ 

for kernel compiling ccache doesn't appear to be worth the effort.


Disliking systemd intensely, but not satisfied with alternatives so focusing on taming systemd.


(A works at time B)  && (time C > time B ) ≠  (A works at time C)

Offline

Board footer

Powered by FluxBB