Switching to libjpeg-turbo?

ajk · 2010-06-15 07:32:58

Dear Archers,

I was wondering if we could use the somewhat new libjpeg-turbo package in Arch to replace the (much slower) libjpeg library. I came across an article on Distrowatch where they point out that Fedora is planning this replacement to for F14. Any comment on this?

Ps. There is an svn package in AUR.

Ajk

Last edited by ajk (2010-06-15 10:11:06)

Allan · 2010-06-15 07:52:11

The comparison is to libjpeg-6b which is a very old libjpeg release that Fedora was still using. In the recent releases upstream, libjpeg has gotten faster. So i am not jumping on the bandwagon that libjpeg-turbo is that much faster without seeing more comparisons to what we are actually using.

ajk · 2010-06-15 07:56:29

That I didn't know.. But the article is talking about a 100% increase in speed, and I didn't see that when we switched to libjpeg-7. Is there a way to compare the speed of both, and I mean hard numbers, not simply user experience...

ngoonee · 2010-06-15 07:59:29

Do jpegs load slow for you? I don't know if I could tell the difference.

ajk · 2010-06-15 08:02:21

High res pics do occasionally load visibly, and I'm using a I5 750... So I think I could tell the difference. And all little speed gains are good right?

Allan · 2010-06-15 08:31:20

Would be changing something that is designed to be compatible with a 12 year old release of libjpeg really be a good idea? Unless someone shows those speed differences are present compared to the current version then I do not think it is worth considering. (hint, i686 does not get the claimed 2-4x increase as it does not use all the SSE optimisations).

ajk · 2010-06-15 08:36:39

Here some more info from the Fedora mailing list. Even without SSE there's a performance increase of 25%.. I guess that would be noticeable. Check out the rest of the thread, it's quite interesting.

Edit: Fedora feature page

Last edited by ajk (2010-06-15 08:38:39)

Allan · 2010-06-15 08:55:20

The improvements that give the increase of 25% are mostly already in the newer libjpeg releases. The newer libjpeg releases also have server other new features and it does not look like libjpeg-turbo are going to include those due to potential patent issues. It only takes one project to start using these features and we have to supply both libraries. I'll wait a while and see if either project starts merging the others features...

ajk · 2010-06-15 09:17:52

They probably will be merging in the near future. Can you give some more info on your statements about the 25% performance increase in libjpeg7/8/8b?

kazuo · 2010-06-15 09:22:54

I did some quick/dirty tests on x86_64 using a Core2 Duo P7350 (2.0GHz). I used a 1080p image from Big Buck Bunny, and two random images generated with imagemagick. This is the result, the value is 'normal/turbo', where 'normal' is the time used by [cd]jpeg from libjpeg and 'turbo' the time used by libjpeg-turbo (time=elapsed real time, %e flag from /usr/bin/time). And sample size is 5 runs.

EDIT: using versions 8.0.2-1 and svn 214-1

encode

5.000000(0.000000): big_buck_bunny_01065.ppm
1.783187(0.007260): plasma.ppm
2.979617(0.015125): random.ppm

decode

2.500000(0.000000): big_buck_bunny_01065.jpg
2.660114(0.056892): plasma.jpg
1.889655(0.020690): random.jpg

samples

wget http://media.xiph.org/BBB/BBB-1080-png/big_buck_bunny_01065.png
convert -size 5000x5000 xc: +noise Random random.ppm
convert -size 5000x5000 plasma:blue plasma.ppm

I dont have access to a i686...

But, as Allan said speed is not the only concern, the library version is 62 (libjpeg.so.62) so if we use it we gonna need to do a version debump, ppl remember the previous libjpeg bump?....

ugly zsh code that do the bench

#!/bin/zsh
zmodload zsh/mathfunc

function mean()
{
  local -F sum
  sum=0.0
  for x in $*; do
    sum+=x
  done
  print $(( $sum/$# ))
}

function stddev()
{
  local m
  local -F sum
  m=$(mean $*)
  for x in $*; do
    sum+=$(((x-m)*(x-m)))
  done
  print $(( sqrt($sum/$#) ))
}

function do_enc_bench()
{
  local image
  local -F turbo normal
  local -a stats
  image=$1
  for i in {1..5}; do
    /usr/bin/time --format="%e" -o stats /opt/libjpeg-turbo/bin/cjpeg $image >/dev/null
    turbo=$(cat stats)
    /usr/bin/time --format="%e" -o stats cjpeg $image >/dev/null
    normal=$(cat stats)
    stats+=$((normal/turbo))
  done
  printf "%f(%f): %s\n" $(mean $stats) $(stddev $stats) $image
}

function do_dec_bench()
{
  local image
  local -F turbo normal
  local -a stats
  image=$1
  for i in {1..5}; do
    /usr/bin/time --format="%e" -o stats /opt/libjpeg-turbo/bin/djpeg $image >/dev/null
    turbo=$(cat stats)
    /usr/bin/time --format="%e" -o stats djpeg $image >/dev/null
    normal=$(cat stats)
    stats+=$((normal/turbo))
  done
  printf "%f(%f): %s\n" $(mean $stats) $(stddev $stats) $image
}

if [[ $1 == 'e' ]]; then
  shift
  for i in $*; do do_enc_bench $i; done
elif [[ $1 == 'd' ]]; then
  shift
  for i in $*; do do_dec_bench $i; done
fi

Last edited by kazuo (2010-06-15 09:30:01)

ajk · 2010-06-15 09:37:15

I'm not getting your numbers dude.. What did you actually calculate? Please clarify..

kazuo · 2010-06-15 09:42:11

Use the source! XD

Well, I timed the enc/dec of a ppm/jpeg file using djpeg and cjpeg with /usr/bin/time --format="%e", i.e. I measured the elapsed real time. and I divided the two values the one from libjpeg and other from libjpeg-turbo. The value is basically how much fast (in time) is libjpeg-turbo in comparison with libjpeg.

EDIT Now, I though... can you be confused be the 'value(std dev)'? I dont think so... but well, the value inside () is the standard dev of sample and the value before () is the mean.

Last edited by kazuo (2010-06-15 09:44:41)

ajk · 2010-06-15 09:49:24

So I guess what I'm seeing is not much off a difference between the two..

kazuo · 2010-06-15 09:53:59

Encoding big_buck_bunny_01065.ppm is five times fast (i.e. 400% faster)... this look like a difference to me...

Last edited by kazuo (2010-06-15 10:00:53)

ajk · 2010-06-15 10:00:12

Then I got it the first time.. This must be worth some consideration i guess.. It's a significant improvement over the standard libraries!

ajk · 2010-07-15 11:54:52

This package is now picked up by Fedora developers and version 1 is already released upstream. So this package seems to be out of 'beta'

Please look into this Arch-devs.. Seems promising, considering the Fedora-team is merging this in their next release (14).

Allan · 2010-07-15 11:58:08

Fedora were using a 12 year old libjpeg, which libjpeg-turbo replaces. We are using a few month old version which is incompatible with libjpeg-turbo. It would require a massive rebuild in Arch to use this.

ajk · 2010-07-15 14:10:17

Look at the pre's! I know who uses what.. I'm just looking at the speed improvements, they are huge!

wonder · 2010-07-15 14:25:01

ajk wrote:

Look at the pre's! I know who uses what.. I'm just looking at the speed improvements, they are huge!

allan concerns are valid. you should think about compatibility with the current libjpeg version 8.0.2. using the fork which is not compatible would require to recompile 200+ packages again.

Labello · 2010-07-15 18:01:08

are there any plans in merging the *-turbo-stuff back into "mainline"?

Allan · 2010-07-15 22:44:22

Labello wrote:

are there any plans in merging the *-turbo-stuff back into "mainline"?

No-one know as mainline development is not particularly open... Also there seems to be no plans to merge the mainline into the turbo version.

Labello · 2010-07-16 08:05:11

would it be possible to somehow slowly migrate one package after another to *-turbo? so one package makes a start and pulls in *-turbo as dependency while the old libjpeg remains on the machine side-by-side with the faster one?

ngoonee · 2010-07-16 10:05:29

Labello wrote:

would it be possible to somehow slowly migrate one package after another to *-turbo? so one package makes a start and pulls in *-turbo as dependency while the old libjpeg remains on the machine side-by-side with the faster one?

I don't think it works that way. Well yes, you could link against some new .so name, but having two side-by-side would just be confusing and a waste of space.

Janax · 2010-07-16 15:10:42

ngoonee wrote:

Labello wrote:
would it be possible to somehow slowly migrate one package after another to *-turbo? so one package makes a start and pulls in *-turbo as dependency while the old libjpeg remains on the machine side-by-side with the faster one?
I don't think it works that way. Well yes, you could link against some new .so name, but having two side-by-side would just be confusing and a waste of space.

Seems to me that space shouldn't be much of an issue:

$ pacman -Qi libjpeg
Name           : libjpeg
Version        : 8.0.2-1
URL            : http://www.ijg.org/
Licenses       : custom
Groups         : None
Provides       : None
Depends On     : glibc
Optional Deps  : None
Required By    : gegl  ghostscript  go-openoffice  gstreamer0.10-good-plugins  imlib2  jasper  libdjvu  libgphoto2  libmng  libtiff  libvncserver  libwmf  mjpegtools  perl-tk  poppler  sane
Conflicts With : None
Replaces       : None
Installed Size : 860.00 K
Packager       : Allan McRae <allan@archlinux.org>
Architecture   : x86_64
Build Date     : Wed 02 Jun 2010 06:40:15 PM CDT
Install Date   : Thu 03 Jun 2010 06:43:10 AM CDT
Install Reason : Installed as a dependency for another package
Install Script : No
Description    : Library of JPEG support functions

Granted, I don't have the size of libjpeg-turbo, as it's not installed, but unless it's significantly bigger, then <1MiB to keep this older version would only be a concern on the most frugal of setups.

Janax · 2010-07-16 15:18:00

On a related note, is the speedup because the IJG version of libjpeg uses x87 floating-point math? Since Arch uses i686+ for x86-style setups, then using SSE instructions would be much better, as argued here for NVIDIA's PhysX libraries (see 'x87= old and busted' section).

Without digging for specifics myself, it seems like that would be an area they might be optimizing for the "turbo" version...

Last edited by Janax (2010-07-16 15:19:12)

Arch Linux

#1 2010-06-15 07:32:58

Switching to libjpeg-turbo?

#2 2010-06-15 07:52:11

Re: Switching to libjpeg-turbo?

#3 2010-06-15 07:56:29

Re: Switching to libjpeg-turbo?

#4 2010-06-15 07:59:29

Re: Switching to libjpeg-turbo?

#5 2010-06-15 08:02:21

Re: Switching to libjpeg-turbo?

#6 2010-06-15 08:31:20

Re: Switching to libjpeg-turbo?

#7 2010-06-15 08:36:39

Re: Switching to libjpeg-turbo?

#8 2010-06-15 08:55:20

Re: Switching to libjpeg-turbo?

#9 2010-06-15 09:17:52

Re: Switching to libjpeg-turbo?

#10 2010-06-15 09:22:54

Re: Switching to libjpeg-turbo?

#11 2010-06-15 09:37:15

Re: Switching to libjpeg-turbo?

#12 2010-06-15 09:42:11

Re: Switching to libjpeg-turbo?

#13 2010-06-15 09:49:24

Re: Switching to libjpeg-turbo?

#14 2010-06-15 09:53:59

Re: Switching to libjpeg-turbo?

#15 2010-06-15 10:00:12

Re: Switching to libjpeg-turbo?

#16 2010-07-15 11:54:52

Re: Switching to libjpeg-turbo?

#17 2010-07-15 11:58:08

Re: Switching to libjpeg-turbo?

#18 2010-07-15 14:10:17

Re: Switching to libjpeg-turbo?

#19 2010-07-15 14:25:01

Re: Switching to libjpeg-turbo?

#20 2010-07-15 18:01:08

Re: Switching to libjpeg-turbo?

#21 2010-07-15 22:44:22

Re: Switching to libjpeg-turbo?

#22 2010-07-16 08:05:11

Re: Switching to libjpeg-turbo?

#23 2010-07-16 10:05:29

Re: Switching to libjpeg-turbo?

#24 2010-07-16 15:10:42

Re: Switching to libjpeg-turbo?

#25 2010-07-16 15:18:00

Re: Switching to libjpeg-turbo?

Board footer