Very poor performance of maven builds with / on SSD and /home [SOLVED]

diederick76 · 2018-05-07 13:32:32

I have a laptop and a desktop PC computer. The laptop has only an SSD, the PC has everything *but* /home on an SSD. /home Is on a HD.

When I time building a specific Java/Maven project on the laptop, it goes reasonably quick:

[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 7.616 s
[INFO] Finished at: 2018-05-07T14:40:31+02:00
[INFO] Final Memory: 47M/221M
[INFO] ------------------------------------------------------------------------
27.31user 0.99system 0:09.72elapsed 291%CPU (0avgtext+0avgdata 372968maxresident)k
189040inputs+161968outputs (267major+114468minor)pagefaults 0swaps

The same build on the PC goes unreasonably slow:

[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 23:30 min
[INFO] Finished at: 2018-05-07T15:10:08+02:00
[INFO] Final Memory: 44M/330M
[INFO] ------------------------------------------------------------------------
31.81user 4.59system 23:31.77elapsed 2%CPU (0avgtext+0avgdata 372648maxresident)k
48inputs+363120outputs (0major+117040minor)pagefaults 0swaps

The PC has a AMD FX-8370 Eight-Core Processor CPU and 16G RAM, much more than the laptop. The HD is also not particularly slow:

$ sudo hdparm -t /dev/sdc
/dev/sdc:
Timing buffered disk reads: 1540 MB in  3.00 seconds = 512.72 MB/sec

In fact, before I build the SSD in the PC, I had the entire system, including /home, on that HD. The only difference between then and now is that the HD only has /home and the systeem boots in UEFI instead of BIOS.

Oddly, other software that I use does not have this problem. Two Steam games that I tried, which are installed /home as well, run perfectly.

One thing I notice is the low CPU percentage on the PC with maven: only 2%, where the laptop reports 291%CPU. Is this a factor, and what causes this?

Thanks for any help!

Last edited by diederick76 (2018-05-10 07:47:38)

Lone_Wolf · 2018-05-08 12:23:52

291% cpu is very weird, do those numbers come from maven itself ?

laptop
189040inputs+161968outputs

pc 
48inputs+363120outputs

some kind of typo or are those the real numbers ?

diederick76 · 2018-05-08 13:47:17

Lone_Wolf wrote:

291% cpu is very weird, do those numbers come from maven itself ?
laptop
189040inputs+161968outputs

pc 
48inputs+363120outputs
some kind of typo or are those the real numbers ?

No typos. Those numbers are the output of time.

Yesterday evening I tried with

time mvn -T 8 clean install

The T flag with the 8 tells maven to use all 8 cores. I can see with

htop

that all 8 cores are used, but it still takes between 25 and 30 minutes.

radiomike · 2018-05-08 20:29:50

Probably the biggest impact on build times is disk access. Even with a quick HDD, the time taken to move the head to each location for reads and writes is huge, especially when it's lots of little files, as it often is when compiling or building.
This is also reflected in the cpu usage figures you're seeing. With the HDD figure, it means that the cpu is idle for so long waiting for data from the drive that it's equivalent to 2% constant load on a single core. 291% means that the ssd is able to keep the data throughput high enough to give the equivalent of 2 fully loded cores and 1 91% loaded.
There's a few options for you to improve this.

First, move your project to your SSD.

Second, assuming limited space, just move your build folder to the SSD.

These two should see a huge improvement, however you may not want to put so much wear on your SSD especially if you are doing lots of full builds.

So, Third configure maven so that your build folder is in /tmp or other tmpfs

Fourth, look at using anything-sync-daemon in aur which will put your project and/or build folder into tmpfs but sync back to HDD or SSD regularly. Also anything-sync-daemon makes backups, so would help to minimise data loss.

You have plenty of ram, so the latter option is probably a good one for you. You should see an improvement in build times, even over SSD build times.

jasonwryan · 2018-05-08 21:40:55

Not an Installation issue, moving to Apps & DEs

diederick76 · 2018-05-09 06:35:12

radiomike wrote:

You have plenty of ram, so the latter option is probably a good one for you. You should see an improvement in build times, even over SSD build times.

Agreed, and I will try this immediately when I get home today. And your suggestion that the many small files maven has to cope with are the problem is plausible. But it still leaves me wondering. Before I switched the PC from a BIOS/MBR configuration to an UEFI/GPT configuration, I used the very same HD to build the same project without any performance issue (this is why I put this question under installation BTW). The HD is a Western Digital Red 4TB and its capacity is the main reason I wanted GPT.

Lone_Wolf · 2018-05-09 10:54:50

Did you change filesystem / partition setup in the mbr > gpt transition ?

A possible cause might be a buggy efi implementation.
Is your bootloader setup flexible enough to try a BIOS + GPT boot ?

diederick76 · 2018-05-09 11:07:49

Lone_Wolf wrote:

Did you change filesystem / partition setup in the mbr > gpt transition ?

Before as well as after the switch I used ext4, on the HD as well as on the SSD now, so no changes there.

Lone_Wolf wrote:

A possible cause might be a buggy efi implementation.
Is your bootloader setup flexible enough to try a BIOS + GPT boot ?

My mother board allows me to choose between UEFI and UEFI + legacy. The second one is indeed what I used before I switched to UEFI+GPT. I could try that option once again to see if that makes any difference.

radiomike · 2018-05-09 11:58:26

You may also want to double check that your partitions are properly aligned as this can have an affect on access times.

diederick76 · 2018-05-09 13:17:48

radiomike wrote:

You may also want to double check that your partitions are properly aligned as this can have an affect on access times.

Is partition alignment a factor when there is only one partition on the disk that uses the entire available disk space?

epinephrine · 2018-05-09 13:48:48

I'd move the build, at least the place where temporary files are generated, to /tmp (which should be a tmpfs). Speeds up any build and doesn't wear the HD/SSD. I do all my makepkg builds in /tmp as well, using BUILDDIR=/tmp/makepkg.

radiomike · 2018-05-09 14:16:13

diederick76 wrote:

Is partition alignment a factor when there is only one partition on the disk that uses the entire available disk space?

Yes it can be. Have a look at Partiton Alignment on the wiki. Also have a look at advanced format from there, if it's appropriate.

diederick76 · 2018-05-09 17:37:23

radiomike wrote:

diederick76 wrote:
Is partition alignment a factor when there is only one partition on the disk that uses the entire available disk space?
Yes it can be. Have a look at Partiton Alignment on the wiki. Also have a look at advanced format from there, if it's appropriate.

$ sudo parted /dev/sdb
GNU Parted 3.2
Apparaat /dev/sdb wordt gebruikt.
Welkom bij GNU Parted!  Typ 'help' voor een opdrachtenoverzicht.
(parted) uitlijncontrole optimal 1                                        
1 is uitgelijnd

It appears to be aligned.

Side note: why are even commands translated into Dutch here? That makes documentation as well as finding answers really hard.

diederick76 · 2018-05-09 17:52:01

radiomike wrote:

So, Third configure maven so that your build folder is in /tmp or other tmpfs
(...)
You have plenty of ram, so the latter option is probably a good one for you. You should see an improvement in build times, even over SSD build times.

By putting this in my pom.xml

<build>
        <directory>/tmp/build/</directory>
        ...
</build>

the build itself became virtually instantaneous, but the install phase at the end, where it copies the resulting jar artefact to the local maven repository (in ~/.m2) still takes about 10 secs. That file is 40MB, so that's still too long, but at least its workable now. I guess I could further optimize this by symlinking, but for now this will suffice.

Booting in UEFI+legacy mode did not make it faster.

Last edited by diederick76 (2018-05-09 18:00:45)

radiomike · 2018-05-09 20:58:46

Some other things to look at.

Make sure AHCI is enabled. Should appear in dmesg or lsmod if it is. If not you'll need to turn it on in the bios. Could be specified as AHCI specifically, or SATA: PATA native mode or possibly disguised as an "optimise for windows" option - if you have "other OS" or equivalent selected it probably disables AHCI.

Also you could look at setting noatime in fstab, which will reduce number of writes to disk. atime or access time is written on every file access, not just when a file is written to. noatime stops this. You probably don't need this, unless you specifically know that you do need it, if you do need it you could try lazytime instead.

Last edited by radiomike (2018-05-09 20:59:43)

diederick76 · 2018-05-10 07:46:59

radiomike wrote:

Some other things to look at.
Also you could look at setting noatime in fstab, which will reduce number of writes to disk. atime or access time is written on every file access, not just when a file is written to. noatime stops this.
You probably don't need this, unless you specifically know that you do need it, if you do need it you could try lazytime instead.

After reading the man page for fstab, I decided to go for lazytime first and this brought total build and install time back to less than 6 seconds, even without building in /tmp. I now mount /home using

rw,lazytime,exec

Arch Linux

#1 2018-05-07 13:32:32

Very poor performance of maven builds with / on SSD and /home [SOLVED]

#2 2018-05-08 12:23:52

Re: Very poor performance of maven builds with / on SSD and /home [SOLVED]

#3 2018-05-08 13:47:17

Re: Very poor performance of maven builds with / on SSD and /home [SOLVED]

#4 2018-05-08 20:29:50

Re: Very poor performance of maven builds with / on SSD and /home [SOLVED]

#5 2018-05-08 21:40:55

Re: Very poor performance of maven builds with / on SSD and /home [SOLVED]

#6 2018-05-09 06:35:12

Re: Very poor performance of maven builds with / on SSD and /home [SOLVED]

#7 2018-05-09 10:54:50

Re: Very poor performance of maven builds with / on SSD and /home [SOLVED]

#8 2018-05-09 11:07:49

Re: Very poor performance of maven builds with / on SSD and /home [SOLVED]

#9 2018-05-09 11:58:26

Re: Very poor performance of maven builds with / on SSD and /home [SOLVED]

#10 2018-05-09 13:17:48

Re: Very poor performance of maven builds with / on SSD and /home [SOLVED]

#11 2018-05-09 13:48:48

Re: Very poor performance of maven builds with / on SSD and /home [SOLVED]

#12 2018-05-09 14:16:13

Re: Very poor performance of maven builds with / on SSD and /home [SOLVED]

#13 2018-05-09 17:37:23

Re: Very poor performance of maven builds with / on SSD and /home [SOLVED]

#14 2018-05-09 17:52:01

Re: Very poor performance of maven builds with / on SSD and /home [SOLVED]

#15 2018-05-09 20:58:46

Re: Very poor performance of maven builds with / on SSD and /home [SOLVED]

#16 2018-05-10 07:46:59

Re: Very poor performance of maven builds with / on SSD and /home [SOLVED]

Board footer