You are not logged in.

#1 2013-01-12 14:19:56

Zucca
Member
From: KUUSANKOSKI, Finland
Registered: 2009-02-26
Posts: 135

Run optipng png optimizer on several cores - optipng_multicore.sh

So. It's a wrapper. I got bored when I have HEXA core on my server and I usually want to maximize (almost maximize) png optimization, but optipng as great tool it is - it lacks feature of running on multiple cores at the same time. So utilizing 1 of 6 of the cores in my case... led to this little implementation.
This script runs many optipng processes in unison/parallel to archieve threading-like support for optipng.
Secret lies within xargs.

Note that for now this script goes trough all the possible combinations/trials. I'm planning to change that to user defined.

It requires bash and xargs that are found from almost every Arch install. And, of course, optipng itself.

Here's the code (for you all to improve?):

#!/bin/bash
# optipng_multicore.sh
# optipng wrapper to use all the cores at maximum settings.
# Code: Zucca@dive.to Zucca@IRC (several networks)
# Licence: I guess MIT would be ok. Just mention my contact info if you want to fork/take over.

# Version 0.2.5

HELPMSG="
 = optipng spawner for multiple CPU cores =
 
-f|--force                   Force overwriting files
-c|--num-cpu|-t|--threads    Specify number of threads (default: automatic detection)
-h|--help                    Umm... It's a surprise?!?

Use '--' to stop searching switches."

# The dirty way to get CPU count (counts HT cores as two)
CPUCOUNT="$(grep -c ^processor /proc/cpuinfo)"

####################
## Parse cli args ##
####################

while [ "${1:0:1}" == "-" ]
do
    case $1 in
        -h|--help)
            echo "$HELPMSG"
            exit 0
        ;;
        -f|--force)
            FORCE=true
        ;;
        -c|--num-cpu)
            if egrep -q '^[0-9]+$' <<< "$2"
            then
                CPUCOUNT="$2"
            else
                echo '--num-cpu option must be followed by a number argument!' >&2
                exit 1
            fi
            shift
        ;;
        --) # Stop searching switches
            shift # break forces to exit the loop right here
                  # that's why we must shift here before
            break
        ;;
        *)
            echo -e "DUDE! I don't know what to do with that switch \"$1\" of yours." >&2
            exit 1
        ;;
    esac
    shift
done

for PNGIN in "$@"
do

    COUNTER=0
    TMPDIR="$(mktemp -d)"

    # Doing this "YO dawg" -loop is because one might want to change possible combinations
    # ... and for the future development. ;)
    for ZC in 1 2 3 8 9 # Compression levels
    do 
        for ZM in {1..9}
        do
            for ZS in {0..3}
            do
                for F in {0..5}
                do
                    for I in {0..1}
                    do
                        echo "-zc${ZC} -zm${ZM} -zs${ZS} -f${F} -i${I} -zw${ZW} -out \"${TMPDIR}/${COUNTER}.png\" \"$PNGIN\"" >> "$TMPDIR/list.txt"
                        let "COUNTER=$COUNTER+1" # It would be nice to have leding zeros but... meh.
                    done
                done
            done
        done
    done
    
    # Start the compression
    xargs -n 8 -P $CPUCOUNT -a "$TMPDIR/list.txt" optipng -quiet
    
    # FIXME: a dirty way to remove all but the smallest size png. Maybe use find + stat
    ls -Ss --block-size=1 "$TMPDIR"/*.png | head -n -1 | egrep -o "${TMPDIR}/[0-9]+\.png" | xargs rm
    
    # Then move the rest (the only one)
    let "WONBYTES=$(stat -c %s "$PNGIN")-$(stat -c %s "$TMPDIR"/*.png)"
    echo "$PNGIN size reduced by $WONBYTES bytes."
    test "$FORCE" == true && mv -f "$TMPDIR"/*.png "${PNGIN%.*}.png" || mv -i "$TMPDIR"/*.png "${PNGIN%.*}.png"
    
    # Delete temp directory
    rm -fr "$TMPDIR"
done

I could put this into AUR if I get enough ... um followers?

Last edited by Zucca (2013-04-13 12:08:35)


K.i.s.s. <3

Offline

#2 2013-01-12 17:50:49

brunogm0
Member
Registered: 2013-01-12
Posts: 4

Re: Run optipng png optimizer on several cores - optipng_multicore.sh

Nice wrapper, can you explain if xargs can auto detect  cpu cores or not ?

I have a patch to update the kde project optimizing script.
My main contribution besides parallel execution where tunings and test that show a nice compression improvement and speed/compression trade-offs.

https://svn.reviewboard.kde.org/r/7000/

You can improve yours with my tunings and i can use your CPUCOUNT.

Offline

#3 2013-01-13 14:07:57

Zucca
Member
From: KUUSANKOSKI, Finland
Registered: 2009-02-26
Posts: 135

Re: Run optipng png optimizer on several cores - optipng_multicore.sh

brunogm0 wrote:

Nice wrapper, can you explain if xargs can auto detect  cpu cores or not ?

Afaik if you specify -P 0 (zero) xargs spans as many processes as it can. So bertter to determine cpu count by ourselves. wink

brunogm0 wrote:

You can improve yours with my tunings and i can use your CPUCOUNT.

Thast's what the FOSS is for! wink
And thanks! I'll look if I can improve my script and I'll share everyting I discover/code here.

I really need to limit the number of trials on my script. -o5 level with few tweaks usually lead into desired size reduction.
I will also make my script more idiot-proof (like catching ^C and reporting possible errors).

Last edited by Zucca (2013-01-13 14:08:52)


K.i.s.s. <3

Offline

#4 2013-01-14 18:58:41

brunogm0
Member
Registered: 2013-01-12
Posts: 4

Re: Run optipng png optimizer on several cores - optipng_multicore.sh

Well, i spent many days doing tests and as show in my script report, the better speed/gain trade-off is -O4 in optipng. Then for each increment in -O level there is a 2% gain with 20% speed penalty.

Also good to test is a further pngout pass (can save a few more bytes).

advancecomp usually brings 1% gain on top of pngout and optipng but the order(optipng>advancecomp>pngout) has to respected.

obs.: it was accepted in KDE http://websvn.kde.org/trunk/KDE/kdesdk/ … iew=markup

Last edited by brunogm0 (2013-01-14 19:02:34)

Offline

#5 2013-02-20 21:55:29

Zucca
Member
From: KUUSANKOSKI, Finland
Registered: 2009-02-26
Posts: 135

Re: Run optipng png optimizer on several cores - optipng_multicore.sh

Fixed a bug in grepping part. That line should be replaced anyways.


K.i.s.s. <3

Offline

#6 2013-04-13 12:17:54

Zucca
Member
From: KUUSANKOSKI, Finland
Registered: 2009-02-26
Posts: 135

Re: Run optipng png optimizer on several cores - optipng_multicore.sh

Ok. Updated the script in first post.

I reduced the compression levels to 1, 2, 3, 8 and 9. It's HIGHLY unlikely that you'd get any better results by using all the compression levels. I think removing level three would be wise too. Anyways this is just temporary solution as I plan to add default levels and user defined levels too (somewhat like with optipng). I may even make it use the exact same switches as optipng so it could be potentially used in place of optipng command in scripts for example.

Note: if you give non-png image to this script it won't delete the original.

I upgraded my server hardware with Opteron 3380. So now I have eight cores to utilize. Even compressing my two monitor screenshots (1920x1200 + 1920x1080) is blazing fast compared to single core compression. wink

Last edited by Zucca (2013-06-23 11:07:23)


K.i.s.s. <3

Offline

Board footer

Powered by FluxBB