Official firefox-pgo thread

harryNID · 2010-02-05 23:17:15

Hi guys,

Thanks for the responses. So far it's been working great for me too.

davidm:

It compiled nicely for me but it had some crashing issues (which admittedly I never bothered to debug and just went back to the repo version of firefox because I was lazy). More than likely it has something to do with my local setup likely with flash because firefox-pgo-minefield does the same thing.

After going back and looking at your posting I see you seem to be using a P4. You might be having the same issues some of us were having with the CFLAGS that are set by default in the original PKGBUILD. I set those back to what they were to be more inline with the original. I was able to build successfully a few times with the original PKGBUILD but it would always cause seg faults for me. If you are using a P4 you could try my CFLAGS which are still in the PKBUILD. Using those I have yet to run into any more seg faults.

msg "Setting up build."
# Changing the user's optimization flags is justified, because this is
# a package specifically for an optimized software build, and because of
# the official branding, binaries can't be redistributed anyways.
# These flags just set guidelines for the build, they are overridden in
# most compile job pieces by Firefox's better judgement.
#export CFLAGS="-march=i686 -mtune=pentium4 -O2 -pipe -fomit-frame-pointer"
#export CXXFLAGS="-march=i686 -mtune=pentium4 -O2 -pipe -fomit-frame-pointer"
export CFLAGS="-march=native -O2 -pipe"
export CXXFLAGS="-march=native -O2 -pipe"
# The hash-style and as-needed flags are in Arch defaults anyways,
# and the other optimization flags are almost definitely safe.
export LDFLAGS="-Wl,-rpath,/usr/lib/firefox-$pkgver -Wl,-O1,--sort-common,--hash-style=gnu,--as-needed"

Just comment out -march=native lines and uncomment the two above. If that indeed is your problem then no matter which version of firefox-pgo you use you will have the same issue irregardless because something in mozilla's code is not picking up -march=native correctly for our processor type. If you decide to test this theory and compile again let us know if it worked for you.

Here's an easy (possible) test to tell if it might be a CFLAGS issue. Open your terminal and start firefox, go to Sunspider and do the test. If it crashes at the end and it looks something like this in the terminal:

/usr/lib/firefox-3.5.5/run-mozilla.sh: line 131: 11193 Segmentation fault "$prog" ${1+"$@"}

It's probably the same issue we are having. If everything runs smoothly with the Sunspider test and it's still crashing then it's probably something else. That's how I can tell if I got a good build or not but mileage may vary.

Hope the info helps!!:D
BTW, I just had to use the stock version of firefox here during the lib updates for a short while and the difference between these two packages is like night and day for me. PGO is is much faster on my computer so it might be worth a recompile just in case that is your problem.

davidm · 2010-02-06 02:05:53

harryNID,

I recompiled using what you suggested and all is fine now. It's a little curious though because the first time I compiled the standard firefox-pgo (3.5.7) it compiled fine with native and I used it for half a day without a single crash. I guess I just got lucky before... Kind of interesting. Thanks again!

Last edited by davidm (2010-02-06 02:06:49)

harryNID · 2010-02-06 15:17:47

harryNID,
I recompiled using what you suggested and all is fine now. It's a little curious though because the first time I compiled the standard firefox-pgo (3.5.7) it compiled fine with native and I used it for half a day without a single crash. I guess I just got lucky before... Kind of interesting. Thanks again!

Cool!!:D

Yeah it took a while (and about 20 builds if you can believe it) to isolate what was causing the issue. It is indeed strange. You might be on to something when you said 3.5.7 worked as I didn't start having issues with this until the 3.6 betas. I'm just wondering if it might not be a bugfix regression causing this or some new type of code mozilla is using for 3.6 and beyond. At least we figured out a way to make it work.

Anyway I'm glad you got it too work. Hope it's faster for you.

Ranguvar · 2010-02-07 20:31:00

Had a lot of trouble with my system, but I have it fixed now.
One thing to take note of is that I have deleted the 'autoconf-compat' package, since 'autoconf2.13' is in [extra] and fulfills the same purpose.

Thanks for helping people out with problems, harryNID

Arm-the-Homeless · 2010-02-07 21:59:22

I'm having problems building on my system.
My PKGBUILD is unchanged from the AUR.

Here's where it starts failing: http://pastebin.com/m1fde3cf9

harryNID · 2010-02-07 22:49:23

Thanks Ranguvar,
Props goes to you, blasse, and the rest of the posters here though.

Also thanks for the dependency update. I have updated the PKGBUILD and re-uploaded the whole caboodle. I fixed the previous post's links in case somebody else out there wants it. I don't want to add new links here as that might detract from your official release.

Seems I might need to recompile so I can get rid of autoconf-compat and get autoconf2.13 myself. Again thanks for the tip.:D

davidm · 2010-02-08 16:56:56

Ranguvar wrote:

If you're using x86_64, that difference is because of TraceMonkey being forcibly enabled in firefox-pgo
I can't see PGO providing THAT big of a boost, hehe.

Between the stock firefox 3.6 from the repos and your firefox-pgo-minefield-smp modified to grab the latest nightly sunspider is giving me a 23% difference and that is without SMP enabled (flash doesn't work currently for me with it enabled, so I wait even though I only have Hyperthreading). That's quite a difference even though it is between versions. Considering I'm on an old P4 3.0 Ghz (i686) with 2GB ram it feels like night and day in responsiveness. Thanks again to you and everyone else involved for maintaining this.

Ranguvar · 2010-02-09 01:53:01

Arm, I don't know your error.
There's a chance your ccache is screwing up (try with it disabled), or you could have a CFLAGS issue (read through the last few pages of this thread).

harry, I just updated the official packages, but thanks for your help

blasse · 2010-02-09 08:53:22

Arm-the-Homeless wrote:

I'm having problems building on my system.
My PKGBUILD is unchanged from the AUR.
Here's where it starts failing: http://pastebin.com/m1fde3cf9

It looks like ccache error. Eventually try with potential-pgo-fix patch (if I remember correctly, it changes compiler flags to deal with sum mismatch).

toxygen · 2010-02-09 16:52:47

quick update on status:
- why is pgo-beta marked out of date if no beta exists yet?
- pgo-beta and -pgo (both 3.6) build fine here, x86_64, with sunspider tests at ~980ms-1050ms and work well
- pgo-minefield has not build for me since 3.7a1, keep getting build errors in the nss modules (related to a bug referenced in mozilla-central)
- firefox-nightly runs (3.7a2), but flash player has major issues (slow, not playing on youtube, etc)
- firefox-kde-opensuse (3.6) works, sunspider tests ~1100 (sometimes lower)

I dont use firefox from extra. all my builds disable gnomevfs/gnomeui/wireless-tools. for the pgo builds, other than the 3.6, i disable all patches too (except the soundsystem = oss).

has anyone successfully built minefield (trunk) in the last few days?

Ranguvar · 2010-02-09 18:01:45

- No clue. There's not even a vaguely decent alpha out. Unmarked.
- Cool
- Builds for me, but I haven't tried running it yet. Unless you're using a later revision than the one I set, because later revisions right now are all broken.
- On 64-bit? Makes sense. They've had some problems with 64-bit plugins on 3.7.

davidm · 2010-02-09 18:46:57

toxygen wrote:

- firefox-nightly runs (3.7a2), but flash player has major issues (slow, not playing on youtube, etc)

Hello. For this one is "dom.ipc.plugins.enabled" set to true within about:config ? If so that caused this to happen for me. For a while they were running the nightly so that this preference is set to true but they recently changed it back to false by default. If it is still "true", changing it to "false" may fix your issue for the time being.

(It's part of the electrolysis project: https://wiki.mozilla.org/Content_Processes/Build and they are starting to integrate it into into 3.6 and 3.7a, not just in the electrolysis builds)

More info:

http://benjamin.smedbergs.us/blog/2010- … y-default/
https://bugzilla.mozilla.org/show_bug.cgi?id=544505

Based upon this from Jan 27 - Feb 05 I think it was on by default.

Oops. Just noticed they backed out on that again: http://hg.mozilla.org/mozilla-central/rev/609a51758b08
Latest info seems to be nightlies: ON (true) Alpha: OFF (off)

Does not overwrite existing profiles but if you do a new install...

Last edited by davidm (2010-02-09 20:17:57)

toxygen · 2010-02-09 20:37:04

wow, thanks. the ipc.plugins setting fixed it on the nightly release for 3.7a2. i had to set it to false on mine (i'm using last night's build though)

davidm · 2010-02-09 21:19:13

toxygen wrote:

wow, thanks. the ipc.plugins setting fixed it on the nightly release for 3.7a2. i had to set it to false on mine (i'm using last night's build though)

Nice. I'm searching for a bug report for this now. It's interesting that you are on x86_64 and I am on i686 (P4 3.0Ghz) and that we are both evidently having the same issue. They later plan on enabling this (OOPP) in a minor release soon for 3.6.x so it is important that they fix it before then.

Seems the closest bug I can find: https://bugzilla.mozilla.org/show_bug.cgi?id=543303

Last edited by davidm (2010-02-09 21:55:31)

toxygen · 2010-02-09 23:36:10

The difficult part for me is finding information on what's changed in both trunk and regular releases. mozilla-central has a steady stream of "fixed this" or "rolled back that", bugzilla can be daunting to use sometimes, and the forums, blogs, and mail-lists (there are a couple out there) are so disorganized, unsynchronized and lacking so much information i'm amazed mozilla even works as a project sometimes

Ranguvar, your revision does indeed build well, and works well. remind me, where do you find the working version (in tinderbox?)

Arm-the-Homeless · 2010-02-10 00:57:47

I've just compiled it correctly but when I go to google.com (probably others, but this is the first one I tried) I get this in console:

┌─[andy @ vithon]-[21:15:26]
└─[~]-[$]> firefox
database table: ubiquity_suggestion_memory already exists
wot_cache.add_target: caching archlinux.org
archlinux.org: 0: (r, c) = (94, 62)
archlinux.org: 0: (i) = (0)
archlinux.org: 0: (l) = (0)
archlinux.org: 1: (r, c) = (95, 61)
archlinux.org: 1: (i) = (0)
archlinux.org: 1: (l) = (0)
archlinux.org: 2: (r, c) = (95, 61)
archlinux.org: 2: (i) = (0)
archlinux.org: 2: (l) = (0)
archlinux.org: 4: (r, c) = (94, 61)
archlinux.org: 4: (i) = (0)
archlinux.org: 4: (l) = (0)
/usr/lib/firefox-3.6/run-mozilla.sh: line 131: 17076 Segmentation fault "$prog" ${1+"$@"}
┌─[andy @ vithon]-[21:15:38]
└─[~]-[$]>

It crashes at the Seg fault (obviously ).

Any ideas/fixes?

Last edited by Arm-the-Homeless (2010-02-10 02:13:57)

Ranguvar · 2010-02-10 12:15:55

Unfortunately, the Tinderbox unit tests haven't been running since December. So, I rely on info people give me in comments and such, and the Tinderbox build tests.

Build tests: http://tinderbox.mozilla.org/showbuilds … ee=Firefox
Unit tests (inactive): http://tinderbox.mozilla.org/showbuilds … x-Unittest

Actually, looking at it now, these "build tests" may be the unit tests... I dunno.

gav616 · 2010-02-10 22:55:32

successfully build and using.

For reference:

[PGKBUILD]
removed: 'libgnomeui' 'wireless_tools' 'startup-notification' 'desktop-file-utils' 'dbus-glib'
commented:

103: export CFLAGS="-march=native -O2 -pipe"
104: export CXXFLAGS="-march=native -O2 -pipe"
-----------
125: if [ $RC != 0 ]; then  # Build did not succeed, fail
126:         return 1
127: fi

pkgname=firefox-pgo
pkgver=3.6
_mozver=1.9.2
pkgrel=1
pkgdesc="Mozilla Firefox customizable web browser (XULRunner independent, PGO optimized, 64-bit TraceMonkey)"
url="http://www.mozilla.org/projects/firefox"
arch=('i686' 'x86_64')
license=('MPL' 'GPL' 'LGPL')

_soundsystem='alsa-lib'  # 'alsa-lib' for ALSA, 'oss' for OSS

makedepends=('autoconf2.13' 'gcc>=4.4' 'zip' 'pkgconfig' 'diffutils' 'python')
depends=('gtk2>=2.18.0' 'gcc-libs>=4.4' 'libidl2>=0.8.13' 'mozilla-common' 'nss>=3.12.4' 'libnotify' 'libxt' 'hunspell>=1.2.8' 'mime-types' 'cairo' 'libpng>=1.4.0' ${_soundsystem})
provides=("firefox=$pkgver" "firefox-pgo-beta=$pkgver" "firefox-beta=$pkgver")
conflicts=('firefox' 'firefox-pgo-beta' 'firefox-beta' 'firefox-pgo-minefield' 'firefox-pgo-minefield-smp')

#"ftp://ftp.mozilla.org/pub/mozilla.org/firefox/nightly/$pkgver-candidates/build3/source/firefox-$pkgver.source.tar.bz2"
source=("ftp://ftp.mozilla.org/pub/mozilla.org/firefox/releases/$pkgver/source/firefox-$pkgver.source.tar.bz2"
        'mozconfig'
        'firefox.desktop'
        'firefox-safe.desktop'
    'xulrunner-png14.patch'
        'mozilla-ps-pdf-simplify-operators.patch'
        '100-system-hunspell-corrections.patch'
        'fix-mozilla-launcher.patch'
        'mozilla-firefox-1.0-lang.patch'
        'ldflags-namespec.patch'
        'enable-x86_64-tracemonkey.patch'
        'jemalloc-enable-pgo.patch'
        'potential-pgo-fix.patch')


build() {
    msg "* Note: If the build fails, try again, try without jemalloc PGO, try without *"
    msg "* PGO at all, and if you're using x86_64, try without TraceMonkey. Also, try *"
    msg "* with 'potential-pgo-fix.patch'. PGO can be very temperamental -- it can    *"
    msg "* sometimes take two or three builds before you'll get a good one.           *"
    sleep 10
    cd "$srcdir"/mozilla-$_mozver

    cp "$srcdir"/mozconfig .mozconfig || return 1
    # Don't strip if the user doesn't want us to... ;p
    if [ "$(check_option strip)" = "n" ]; then
        sed -i 's/--enable-strip/--disable-strip/' \
            .mozconfig || return 1
        sed -i 's/--enable-install-strip/--disable-install-strip/' \
            .mozconfig || return 1
    fi

    if [ "$_soundsystem" = "oss" ]; then
        msg "Using OSS instead of ALSA."
        sed -i 's/sydney_audio_alsa/sydney_audio_oss/' \
               media/libsydneyaudio/src/Makefile.in || return 1
        # Get rid of ALSA config system stuff (requires autoconf rerun)
        sed -i '/alsa\//d' config/system-headers || return 1
        sed -i '/alsa\//d' js/src/config/system-headers || return 1
        sed -i '/LIB(asound/d' configure.in || return 1
    fi


    msg "Patching source."
    msg "Patches from main Arch xulrunner/firefox packages..."
    # Fix build with libpng 1.4.0
    patch -Np0 -i "$srcdir"/xulrunner-png14.patch || return 1
    # Upstream patch, only in 3.7.x
    patch -Np1 -i "$srcdir"/mozilla-ps-pdf-simplify-operators.patch \
        || return 1
    # Fix build with system Hunspell - Gentoo
    patch -Np0 -i "$srcdir"/100-system-hunspell-corrections.patch \
        || return 1
    # Fix stub launcher - Arch
    patch -Np0 -i "$srcdir"/fix-mozilla-launcher.patch || return 1
    # Use LANG environment variable to choose locale
    patch -Np1 -i "$srcdir"/mozilla-firefox-1.0-lang.patch || return 1


    msg "Patches from this package..."
    # PGO compilation LDFLAGS fix
    patch -Np1 -i "$srcdir"/ldflags-namespec.patch || return 1
    # Enable TraceMonkey engine on x86_64; normally only firefox>=3.7
    # has 64-bit TM.  See Mozilla bug #489146.  Requires autoconf run.
    # Does not affect 32-bit builds. This can be problematic, but it should
    # boost speed FAR more than PGO. So, if only PGO and only TM build,
    # but not both, go with TM.
    patch -Np0 -i "$srcdir"/enable-x86_64-tracemonkey.patch || return 1
    # Enable experimental PGO for jemalloc (speed).
    # See Mozilla bugs #418866 and #419470.
    patch -Np0 -i "$srcdir"/jemalloc-enable-pgo.patch || return 1
    # Sometimes necessary for PGO build to work, especially when PGO compile
    # aborts with an error about control sum mismatch (thx to methuselah!)
    #patch -Np1 -i "$srcdir/potential-pgo-fix.patch" || return 1


    msg "Setting up build."
    # Changing the user's optimization flags is justified, because this is
    # a package specifically for an optimized software build, and because of
    # the official branding, binaries can't be redistributed anyways.
    # These flags just set guidelines for the build, they are overridden in
    # most compile job pieces by Firefox's better judgement.
    #export CFLAGS="-march=native -O2 -pipe"
    #export CXXFLAGS="-march=native -O2 -pipe"
    # The hash-style and as-needed flags are in Arch defaults anyways,
    # and the other optimization flags are almost definitely safe.
    export LDFLAGS="-Wl,-rpath,/usr/lib/firefox-$pkgver -Wl,-O1,--sort-common,--hash-style=gnu,--as-needed"

    autoconf-2.13 || return 1


    msg "Actual build."
    # Yes, all this is SMP -- MOZ_MAKE_FLAGS takes care of it.
    # Compile a non-PGO build first to reduce chance of error in PGO build.
    make -j1 -f client.mk build MOZ_MAKE_FLAGS="$MAKEFLAGS" || return 1
    # Comment out remaining lines for a non-PGO build.
    msg "Profiled build now."
    # Don't let PGO disrupt the user, use a virtual framebuffer.
    LD_PRELOAD="" /usr/bin/Xvfb -nolisten tcp -extension GLX :99 &
    XPID=$!  # Store the PID of the framebuffer
    DISPLAY=:99 LD_PRELOAD="" make -j1 -f client.mk profiledbuild \
        MOZ_MAKE_FLAGS="$MAKEFLAGS"
    RC=$?  # Store the make return code
    kill -9 $XPID  # Kill the framebuffer
#    if [ $RC != 0 ]; then  # Build did not succeed, fail
#        return 1
#    fi
}
package() {
    cd "$srcdir/mozilla-$_mozver"

    msg "Installing to $pkgdir."
    make -j1 DESTDIR="$pkgdir" -C ff-pgo install || return 1

    install -Dm644 "$srcdir/mozilla-$_mozver/other-licenses/branding/firefox/mozicon128.png" \
        "$pkgdir/usr/share/pixmaps/firefox.png" || return 1
    install -Dm644 "$srcdir/firefox.desktop" \
        "$pkgdir/usr/share/applications/firefox.desktop" || return 1
    install -Dm644 "$srcdir/firefox-safe.desktop" \
        "$pkgdir/usr/share/applications/firefox-safe.desktop" \
        || return 1

    # Remove devel stuff.
    rm -rf "$pkgdir/usr/include/"
    rm -rf "$pkgdir/usr/lib/firefox-devel-$pkgver/"
    rm -rf "$pkgdir/usr/share/idl/"

[makepkg.conf]

CFLAGS="-march=athlon-xp -O2 -pipe -fomit-frame-pointer"
CXXFLAGS="${CFLAGS}

[mozconfig]
added:

ac_add_options --disable-dbus
ac_add_options --disable-gnomeui
ac_add_options --disable-gnomevfs
ac_add_options --disable-xprint

changed:

13: ac_add_options --disable-startup-notification
25: ac_add_options --disable-printing --disable-xinerama

Last edited by gav616 (2010-02-11 13:32:29)

gav616 · 2010-02-12 03:40:48

Firefox crashes when I drag files in 'Organize Bookmarks...'

I've tried different build with the same effect.

anyone getting this?

harryNID · 2010-02-12 23:12:18

Wee it's not just me.

Let me ask you a question. Did you compile GTK2 yourself? If so then there's your culprit. Load the stock version and everything works, build your own and Organize Bookmarks go wonky and crashes firefox. Every time I try to compile GTK2 myself everything works but I can't use any of the Bookmark features (Organize Bookmarks, Bookmarks Sidebar etc.). I have no clue as to why unless it has something to do with how GTK2 is compiled. I have no other issues or errors with my compiled version of GTK2 yet firefox apparently doesn't like it.

Marcel- · 2010-02-13 18:16:27

First of all a big thank you for providing this optimized package!

Now the bad news: Firefox-pgo 3.6 crashes every time a few seconds after startup (using Openbox WM):

$ strace -o output firefox
OpenOffice path is '/usr/lib/openoffice'
/usr/lib/firefox-3.6/run-mozilla.sh: line 131:  4126 Segmentation fault      "$prog" ${1+"$@"}
$ tail -20 output 
rt_sigprocmask(SIG_SETMASK, [INT CHLD], NULL, 8) = 0
_llseek(255, -35, [3896], SEEK_CUR)     = 0
clone(child_stack=0, flags=CLONE_CHILD_CLEARTID|CLONE_CHILD_SETTID|SIGCHLD, child_tidptr=0xb7692728) = 4122
rt_sigprocmask(SIG_SETMASK, [], NULL, 8) = 0
rt_sigprocmask(SIG_BLOCK, [CHLD], [], 8) = 0
rt_sigprocmask(SIG_SETMASK, [], NULL, 8) = 0
rt_sigprocmask(SIG_BLOCK, [CHLD], [], 8) = 0
rt_sigaction(SIGINT, {0x807bac0, [], 0}, {SIG_DFL, [], 0}, 8) = 0
waitpid(-1, [{WIFEXITED(s) && WEXITSTATUS(s) == 139}], 0) = 4122
rt_sigprocmask(SIG_SETMASK, [], NULL, 8) = 0
--- SIGCHLD (Child exited) @ 0 (0) ---
waitpid(-1, 0xbfe23ecc, WNOHANG)        = -1 ECHILD (No child processes)
sigreturn()                             = ? (mask now [])
rt_sigaction(SIGINT, {SIG_DFL, [], 0}, {0x807bac0, [], 0}, 8) = 0
rt_sigprocmask(SIG_BLOCK, NULL, [], 8)  = 0
read(255, "exitcode=$?\n\nexit $exitcode\n# EO"..., 3931) = 35
rt_sigprocmask(SIG_BLOCK, NULL, [], 8)  = 0
rt_sigprocmask(SIG_BLOCK, NULL, [], 8)  = 0
rt_sigprocmask(SIG_SETMASK, [], NULL, 8) = 0
exit_group(139)                         = ?

I don't know if this helps... I will try the things in gav616's post.

bobdob · 2010-02-14 15:20:48

Anyone tried the 3.7a1 dev preview yet?
I builds with little change to the 3.6 PKGBUILD, and it feels faster.

Whats going to happen to the firefox-pgo-beta pkg, since 3.7 hasn't branched yet and 3.6 is supposed to get new features with point releases.

davidm · 2010-02-14 16:14:19

Marcel-

What are you using for CFLAGS and what processor have you? Using native or the wrong flags has caused a segfault issue for many people.

toxygen · 2010-02-14 16:23:15

bobdob wrote:

Anyone tried the 3.7a1 dev preview yet?
I builds with little change to the 3.6 PKGBUILD, and it feels faster.
Whats going to happen to the firefox-pgo-beta pkg, since 3.7 hasn't branched yet and 3.6 is supposed to get new features with point releases.

firefox-pgo-minefield is at 3.7a2 already, or you can try firefox-nightly for the latest branch release. it's definitely faster for me, but still buggy and some extensions you'll have to force install, or not use at all.

Marcel- · 2010-02-14 17:06:09

davidm wrote:

What are you using for CFLAGS and what processor have you?

I was using the default ones:

# Changing the user's optimization flags is justified, because this is
# a package specifically for an optimized software build, and because of
# the official branding, binaries can't be redistributed anyways.
# These flags just set guidelines for the build, they are overridden in
# most compile job pieces by Firefox's better judgement.
export CFLAGS="-march=native -O2 -pipe"
export CXXFLAGS="-march=native -O2 -pipe"

on an Intel(R) Atom(TM) CPU N270 @ 1.60GHz stepping 02.

Arch Linux

#501 2010-02-05 23:17:15

Re: Official firefox-pgo thread

#502 2010-02-06 02:05:53

Re: Official firefox-pgo thread

#503 2010-02-06 15:17:47

Re: Official firefox-pgo thread

#504 2010-02-07 20:31:00

Re: Official firefox-pgo thread

#505 2010-02-07 21:59:22

Re: Official firefox-pgo thread

#506 2010-02-07 22:49:23

Re: Official firefox-pgo thread

#507 2010-02-08 16:56:56

Re: Official firefox-pgo thread

#508 2010-02-09 01:53:01

Re: Official firefox-pgo thread

#509 2010-02-09 08:53:22

Re: Official firefox-pgo thread

#510 2010-02-09 16:52:47

Re: Official firefox-pgo thread

#511 2010-02-09 18:01:45

Re: Official firefox-pgo thread

#512 2010-02-09 18:46:57

Re: Official firefox-pgo thread

#513 2010-02-09 20:37:04

Re: Official firefox-pgo thread

#514 2010-02-09 21:19:13

Re: Official firefox-pgo thread

#515 2010-02-09 23:36:10

Re: Official firefox-pgo thread

#516 2010-02-10 00:57:47

Re: Official firefox-pgo thread

#517 2010-02-10 12:15:55

Re: Official firefox-pgo thread

#518 2010-02-10 22:55:32

Re: Official firefox-pgo thread

#519 2010-02-12 03:40:48

Re: Official firefox-pgo thread

#520 2010-02-12 23:12:18

Re: Official firefox-pgo thread

#521 2010-02-13 18:16:27

Re: Official firefox-pgo thread

#522 2010-02-14 15:20:48

Re: Official firefox-pgo thread

#523 2010-02-14 16:14:19

Re: Official firefox-pgo thread

#524 2010-02-14 16:23:15

Re: Official firefox-pgo thread

#525 2010-02-14 17:06:09

Re: Official firefox-pgo thread

Board footer