LinuxQuestions.org

LinuxQuestions.org (/questions/)
-   Linux - Software (https://www.linuxquestions.org/questions/linux-software-2/)
-   -   Building Firefox 3.6 - A "Profile Guided Optimization" (PGO) Automated Build Script (https://www.linuxquestions.org/questions/linux-software-2/building-firefox-3-6-a-profile-guided-optimization-pgo-automated-build-script-784164/)

GrapefruiTgirl 01-23-2010 02:11 AM
Building Firefox 3.6 - A "Profile Guided Optimization" (PGO) Automated Build Script
 
So, Firefox 3.6 just came out a day or two ago, and is definitely improved over previous versions. And, just in time, I finally figured out what the heck I'm doing with this PGO build stuff, so here's a script that will build you a PGO Firefox 3.6 (or whatever version you like).

This script (as is much stuff I write) is very Slackware-centric, but can be fairly easily adapted to suit your Linux distro if it is not Slackware. This is based on the Slack 13.0/64 SlackBuild, and a bunch of other stuff I've gleaned from mozilla.org and elsewhere on the net.

Examine the script carefully to learn how it works. It can create a ready-to-install SlackPackage for Slackware 64, or it can make a generic Firefox tarball which you can manually extract & install somewhere.

Be careful with compiler optimization flags; the ones in the script work for me, but may not work for you (I'm using a E2160 Core2 Intel CPU and GCC 4.3.3) but too many flags will break the build.

The resulting build is FAST, both in page rendering type of areas, as well as chrome and GUI operations-- a LOT faster, speedier, and more responsive than a stock build.

An added bonus, for those who aren't aware: FF 3.6 has a new theme engine (if I can call it that) that allows lightweight UI themes called 'personas' to be instantly skinned onto the browser; no restart, no installing, just instant cool colorful themes!

Here goes:

Code:
#!/bin/sh
# This script is POSIX compliant and comes with no warranty.
# Tune it up or otherwise adjust as you see fit.
# By Sasha - Jan 22 2010
# My gcc used = 4.3.3
#
# PGO (Profile Guided Optimization) Firefox build
#  for 64bit Slackware 13.0
# Works for me with FF 3.6 (freshly released Jan 22/2010)
# Script based on Slackware 13/64 SlackBuild and other
# examples from here and there, including mozilla.org.
#
# This whole build is a multi-stage clobber-build
# which takes the better part of 1.5 hours, including
# time spent playing with the half-built browser half
# way through the build to generate profiling info.
#
# FF 3.6 build depends on libnotify from Galago and
# some junk from your Wireless Tools package (yes, wtf?!)
# BEGIN:
# First, see that we aren't root, because we _are_ going to
# fire up the new browser half-way through the build, to generate
# some profiling information. Don't wanna browse as root, and
# also, root is not allowed to connect to the X server by
# default, so the build will fail when the browser cannot be started.
echo "Checking UID and file ownership.."
if [ $(id -u) = 0 ]; then
 echo "You should not do this as root."
 echo "Please change to another UID before running this script."
 exit 1
else
# ensure everything is our UID:GID or quit
# (in case any of it is root owned)
 if [ "$(chown -R $(id -u):$(id -g) ./* 2>&1 >/dev/null)" ]; then
  echo "Stuff in here is not chownable to our UID:GID."
  echo "Please fix it and run this script again."
  exit 1
 fi;
fi;

# my favorite compiler optimizations:
CFLAGS="-fPIC -O3 -Wall -pipe -march=native -mtune=native -fomit-frame-pointer"
CXXFLAGS="${CFLAGS}"
CPPFLAGS="${CFLAGS} -I/usr/include"
MAKEFLAGS="-j3"

# FF apparently won't build properly with autoconf != 2.13 so
# in case this is still the case, I installed autoconf 2.13 in
# /usr/local/bin and this PATH makes it be found first, before
# my regular autoconf 2.6:
PATH=/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin:.
export PATH CPPFLAGS CXXFLAGS CFLAGS MAKEFLAGS

CWD=$(pwd)                      # where am I now (start base dir)
TMP=$CWD/temp                    # a slackpkg will be constructed in here.
LIBDIR=lib64                    # slack64 libdir.
SRCDIR=$CWD/SOURCE              # FF source dir.
OBJDIR=$CWD/OBJDIR              # build dest dir.
PKGDIR=$CWD/PGO_FINISHED_PROD    # slack package or tarball ends up in here.
PROFIL=$CWD/_PROFILEPROFILE      # PGO profile folder
                                #  (stick your places.sqlite in there if you want)
MOZAPP=browser                  # in mozconfig, which mozilla component to build
APP=firefox                      # self evident.
VERSION=3.6                      # self evident.
EXTENSION=en-US.linux            # FF tarball name includes this piece.
                                #  but may change depending on your locale.
ARCH=x86_64                      # architecture.
BUILD=1                          #


# Download all the parts here?

# make sure we have all the necessary pieces for
# creating a slackpackage, and the source archive too:
for i in firefox.png mimeTypes.rdf.gz firefox.moz_plugin_path.diff.gz \
        mozilla-firefox.desktop slack-desc $APP-$VERSION.source.tar.bz2; do
 if [ ! -f "$CWD/$i" ]; then
  echo "Error: $CWD/$i missing. Get it somewhere and try again."
  exit 1
 fi;
done

# clean away old build dirs and make fresh ones:
rm -Rf $SRCDIR $OBJDIR $TMP $PKGDIR
mkdir -p $SRCDIR $OBJDIR $PROFIL $TMP $PKGDIR

# run-script.sh ##################################################
# create the app startup script for profile generation:
cat << EOF > run-$APP.sh
#!/bin/sh
NO_EM_RESTART=1
DISPLAY=:0.0
export NO_EM_RESTART DISPLAY
$OBJDIR/dist/bin/$APP -no-remote -profile $PROFIL

EOF
chmod a+x run-$APP.sh
# /run-script.sh ##################################################

# extract the FF source tar.bz2 into $SRCDIR..
echo "Extracting source.."
cd $SRCDIR
tar xf $CWD/$APP-$VERSION.source.tar.bz2 || exit 1
sync
SRCDIR=$(pwd)/$(ls)
cd $SRCDIR || exit 1

#/usr/local/bin/autoreconf || exit 1

# Pat Volkerding says:
# Fix a long standing bug that's prevented staying current on GTK+.
# Thanks to the BLFS folks.  :-)
cat << EOF >> layout/build/Makefile.in

ifdef MOZ_ENABLE_CANVAS
EXTRA_DSO_LDOPTS += \$(XLDFLAGS) -lX11 -lXrender
endif

EOF

# mozconfig #######################################################
# and create our mozconfig in $SRCDIR:
cat << EOF > mozconfig

# FF PGO-build mozconfig file
# option to say use objdir (build dir):
mk_add_options MOZ_OBJDIR=$OBJDIR

# startup script, for generating profile info for optimize:
mk_add_options PROFILE_GEN_SCRIPT=$CWD/run-$APP.sh

# which mozilla app to build:
ac_add_options --enable-application=$MOZAPP

########################################: Stuff that always worked:
###################### (mostly) stuff from the -current slackbuild:
ac_add_options --enable-official-branding
ac_add_options --prefix=/usr --libdir=/usr/$LIBDIR
ac_add_options --host="x86_64-slackware-linux-gnu"
ac_add_options --with-default-mozilla-five-home=/usr/$LIBDIR/$APP-$VERSION
ac_add_options --with-system-zlib
ac_add_options --enable-default-toolkit=cairo-gtk2
ac_add_options --enable-crypto
ac_add_options --enable-svg
ac_add_options --enable-canvas
ac_add_options --enable-xft
ac_add_options --enable-xinerama
ac_add_options --enable-optimize="-fPIC -Os -Wall -pipe -march=native -mtune=native -fomit-frame-pointer"
ac_add_options --enable-reorder
ac_add_options --enable-strip
ac_add_options --enable-system-cairo           
ac_add_options --enable-cpp-rtti
ac_add_options --enable-single-profile
ac_add_options --disable-ldap
ac_add_options --disable-accessibility
ac_add_options --disable-debug
ac_add_options --disable-tests
ac_add_options --disable-logging
ac_add_options --disable-pedantic
ac_add_options --disable-installer
ac_add_options --disable-mailnews
ac_add_options --disable-composer
ac_add_options --disable-profilesharing
# End official slackbuild opts
########################################: stuff I'm 98% sure always works:
ac_add_options --with-gxx-include-dir=/usr/include/c++/4.3.3
ac_add_options --quiet
ac_add_options --with-system-bz2
ac_add_options --with-pthreads
ac_add_options --enable-threads=posix
ac_add_options --enable-system-lcms
ac_add_options --enable-system-sqlite
ac_add_options --enable-pango
ac_add_options --enable-install-strip
ac_add_options --disable-updater
ac_add_options --disable-gnomeui
ac_add_options --disable-gnomevfs
########################################: stuff I'm 95% sure works fine:
#ac_add_options --enable-libxul
#ac_add_options --with-system-jpeg             
#ac_add_options --with-system-nssac_add_options
ac_add_options --disable-crashreporter
ac_add_options --disable-help-viewer
ac_add_options --disable-debug-modules
ac_add_options --disable-debugger-info-modules
ac_add_options --disable-parental-controls
ac_add_options --disable-safe-browsing
ac_add_options --disable-startup-notification
########################################: Completely Experimental below..
########################################: Questionable/Optional stuff
########################################: that may, or just may not work:
#ac_add_options --enable-javaxpcom
#ac_add_options --enable-oji
#ac_add_options --enable-xpcom-fastload
#ac_add_options --enable-plugins
#ac_add_options --enable-jemalloc
#ac_add_options --enable-mathml
#ac_add_options --disable-xpcom-obsolete
#ac_add_options --enable-svg-foreignobject
# ac_add_options --enable-chrome-format=jar
# ac_add_options --enable-native-uconv # not needed/not recommended
# ac_add_options --enable-image-encoders=all # should not need
# ac_add_options --enable-update-channel=release # needed by moz.org people only?
# ac_add_options --enable-prebinding # OSX only?
# ac_add_options --enable-reflow-perf # try without.. may slow down FF for non-devel user?
# ac_add_options --enable-glitz # this thing won't install into lib64
# END of mozconfig file

EOF
# /mozconfig #######################################################

# configure the sources:
cd $OBJDIR
echo ; echo "Info: Running configure in 5" ; sleep 5
$SRCDIR/configure || exit 1

# make the sources:
cd $SRCDIR
echo ; echo "Info: Starting non-PGO build in 5" ; sleep 5
make -s -f client.mk build MOZ_MAKE_FLAGS="${MAKEFLAGS}" || exit 1

# make PGO build in same objdir, on top of what's already built:
echo ; echo "Info: Done non-PGO build... Continuing in 5" ; sleep 5
make -s -f client.mk profiledbuild MOZ_MAKE_FLAGS="${MAKEFLAGS}" || exit 1

# Here is when the profiling build should fire up. Use it for a while,
# do some browsing, give it a hard time; sometimes a bad build will
# run for 5, 10 or 15 minutes until it hits a snag, bug, segfault
# or whatever, so you want to make it happen NOW, not when you've
# got 25 tabs open tomorrow and the whole thing locks up on you.
# When you're done, and confident that the browser is stable,
# click File->Quit and the build will resume.

# OPTIONS HERE (functions):

make_tarball () {
# make tarball: This makes a generic tarball of the newly
# compiled Firefox and related components, and zips it up
# into a bzip2 archive. You can then do what you like with
# it, such as install it manually, or just unpack it somewhere
# and use it as-is.
# Package it into a tarball to fix symlinks & gather all pieces:
( cd $OBJDIR
  echo ; echo "Info: Packaging tarball in 5" ; sleep 5
  make package ) || exit 1
#move tarball to $CWD (start directory)
( cd $PKGDIR
  cp -fvt . $OBJDIR/dist/$APP-$VERSION.$EXTENSION-$ARCH.tar.bz2
) || exit 1

cd $CWD
# Now do what you want with the finished tarball in $PKGDIR.
}

# 2) make slackpkg:
make_slackpkg () {
# make a slackware package according to the general idea
# of what happens in the Slack13.0/64 SlackBuild script:
( cd $OBJDIR
  echo ; echo "Info: Creating slackpkg dir structure in 5" ; sleep 5
  make install DESTDIR=$TMP
  # We don't need these things:
  rm -Rf $TMP/usr/$LIBDIR/$APP-devel-$VERSION
  rm -Rf $TMP/usr/include
)
( cd $TMP/usr/$LIBDIR/$APP-$VERSION
  zcat $CWD/mimeTypes.rdf > defaults/profile/mimeTypes.rdf || exit 1
  zcat $CWD/firefox.moz_plugin_path.diff.gz \
    | sed -e "s#usr/lib#usr/$LIBDIR#g" \
    | patch -p1 --verbose --backup --suffix=.orig || exit 1
  # Clean up if the above patch was successful:
  # On my system, that patch is successful, but a bit fuzzy :/
  rm -f firefox.orig
) || exit 1

mkdir -p $TMP/usr/$LIBDIR/mozilla/plugins
mkdir -p $TMP/usr/share/applications
cat $CWD/mozilla-$APP.desktop > $TMP/usr/share/applications/mozilla-$APP.desktop
mkdir -p $TMP/usr/share/pixmaps
cat $CWD/$APP.png > $TMP/usr/share/pixmaps/$APP.png

# These files/directories are usually created if Firefox is run
# as root which technically should be never. Anyhow, if we don't
# see these items, put stubbies in place (whatever those are) to
# prevent startup errors:
( cd $TMP/usr/$LIBDIR/$APP-$VERSION
  if [ -d extensions/talkback\@mozilla.org ]; then
    if [ ! -r extensions/talkback\@mozilla.org/chrome.manifest ]; then
      echo > extensions/talkback\@mozilla.org/chrome.manifest
    fi
  fi
  if [ ! -d updates ]; then
    mkdir -p updates/0
  fi
)

# Install icons/imgs/license/slack-desc:
mkdir -p $TMP/usr/$LIBDIR/$APP-$VERSION/chrome/icons/default
install -m 644 $SRCDIR/other-licenses/branding/$APP/default16.png $TMP/usr/$LIBDIR/$APP-$VERSION/icons/
install -m 644 $SRCDIR/other-licenses/branding/$APP/default16.png $TMP/usr/$LIBDIR/$APP-$VERSION/chrome/icons/default/
install -p -c -m 644 $SRCDIR/LICENSE $TMP/usr/$LIBDIR/$APP-$VERSION/
mkdir $TMP/install
cat $CWD/slack-desc > $TMP/install/slack-desc

cd $TMP
# whip up the Slackpackage:
echo "Ready to zip up the slackpkg."
echo "1) su to root."
echo "2) Make sure you are in this folder: $TMP"
echo "3) Now, as root, type:"
echo "makepkg -l y -c y $PKGDIR/mozilla-$APP-$VERSION-$ARCH-$BUILD.tgz"
} # end of make_slackpkg()

#NOTE: Below here, uncomment whichever function-call you
# plan to use: either "make_slackpkg" or "make_tarball"
# You can probably do both if you have some reason to do so..

#make_tarball
make_slackpkg

#EOF
Have fun!
Sasha

craigevil 01-23-2010 03:10 AM
Nice.

Even 64 bit isn't really that hard to build.
http://www.mepis.org/docs/en/index.php/Compile_Firefox

GrapefruiTgirl 01-23-2010 10:16 AM
You're quite right -- building it plain and simple, is easy and pretty quick; but why bother doing that when you will (generally) end up with the same product that comes with your distro? :) It's the "optimizing for your system, and speeding the thing up" part that causes headaches. :)
Sure, this isn't rocket science, but the available docs re optimizing FF have a lot of holes in them.

Wait till 3.7+ comes out officially! There should be a significant difference over current versions, particularly in the mozilla releases for 64 bit.

Best regards,
Sasha

Shining Arcanine 04-18-2010 08:14 AM
I am a Gentoo Linux user and I wanted to try out a PGO build of firefox. I tried compiling Firefox myself from the official source tarball, but I had trouble doing that, so I tried your script and things worked after I made some minor changes. I was able to make some improvements to your script, so I thought I would post here to share them.

I modified your script to use the automated script that the Mozilla Foundation uses to generate nightly firefox builds. I also modified it to apply a small patch that enables x86_64 Trace Monkey support.

You can get the patch here:

http://aur.archlinux.org/packages/fi...cemonkey.patch

I tried doing a PGO-build of Firefox without that patch, but it was no faster than the system Firefox that Gentoo's package manager builds. I then modified the script to apply that patch and re-ran your build script. Sun Spider benchmarks went from 2069.8ms +/- 4.5% to 1090.6ms +/- 1.6%, which is a substantial improvement over the system Firefox, which for comparison purposes, scored 2011.0ms +/- 3.9%.

Here is a patch file that modifies your build script to implement the changes I made:

Code:
--- original.sh 2010-04-18 07:38:50.203135569 -0500
+++ build.sh    2010-04-18 08:04:19.223826684 -0500
@@ -66,6 +66,7 @@
                                  #  but may change depending on your locale.
 ARCH=x86_64                      # architecture.
 BUILD=1                          #
+PYTHON=/usr/bin/python
 
 
 # Download all the parts here?
@@ -73,7 +74,8 @@
 # make sure we have all the necessary pieces for
 # creating a slackpackage, and the source archive too:
 for i in firefox.png mimeTypes.rdf.gz firefox.moz_plugin_path.diff.gz \
-        mozilla-firefox.desktop slack-desc $APP-$VERSION.source.tar.bz2; do
+        mozilla-firefox.desktop slack-desc $APP-$VERSION.source.tar.bz2 \
+        enable-x86_64-tracemonkey.patch; do
  if [ ! -f &quot;$CWD/$i&quot; ]; then
  echo &quot;Error: $CWD/$i missing. Get it somewhere and try again.&quot;
  exit 1
@@ -107,6 +109,9 @@
 
 #/usr/local/bin/autoreconf || exit 1
 
+# Enable x86_64 Tracemonkey Support
+patch $SRCDIR/js/src/configure.in < $CWD/enable-x86_64-tracemonkey.patch || exit 1
+
 # Pat Volkerding says:
 # Fix a long standing bug that's prevented staying current on GTK+.
 # Thanks to the BLFS folks.  :-)
@@ -127,7 +132,10 @@
 mk_add_options MOZ_OBJDIR=$OBJDIR
 
 # startup script, for generating profile info for optimize:
-mk_add_options PROFILE_GEN_SCRIPT=$CWD/run-$APP.sh
+#mk_add_options PROFILE_GEN_SCRIPT=$CWD/run-$APP.sh
+
+# Automated script; comment this and uncomment the above for a manual script
+mk_add_options PROFILE_GEN_SCRIPT='${PYTHON} ${OBJDIR}/_profile/pgo/profileserver.py'
 
 # which mozilla app to build:
 ac_add_options --enable-application=$MOZAPP
Next I am going to modify Gentoo's ebuild for Firefox to apply these changes to it. Some other Gentoo users got it to work for Firefox 3.6.2, but since Firefox 3.6.3 is out, I am going to start from the official ebuild and use their work as a guide. Here is the thread if anyone is interested:

http://forums.gentoo.org/viewtopic-t-821325.html

GrapefruiTgirl 04-19-2010 10:59 PM
@ *Arcanine,

thanks for providing that update and patch and stuff. One of these days I need to rebuild mine again since FF has been updated a few times, and will look into the tracemonkey patch as it's reputed and reported to be a major improvement over the regular engine.

Cheers!
Sasha

onebuck 10-28-2010 09:10 AM
Hi,

sasha, I don't belive in resurrecting old threads but this one has intrigued me.

I'm curious as to the status of your PGO build for Firefox since FF is now at 3.6.11? What about updates for this script for Slackware 13.1_x86_64?

I've been reading your script;
Quote:
# PGO (Profile Guided Optimization) Firefox build
# for 64bit Slackware 13.0
# Works for me with FF 3.6 (freshly released Jan 22/2010)
# Script based on Slackware 13/64 SlackBuild and other
# examples from here and there, including mozilla.org.
#
# This whole build is a multi-stage clobber-build
# which takes the better part of 1.5 hours, including
# time spent playing with the half-built browser half
# way through the build to generate profiling info.
#
# FF 3.6 build depends on libnotify from Galago and
# some junk from your Wireless Tools package (yes, wtf?!)
Do you have links, notes & specifics on the underlined above?
I've been looking at working on PGO FF for a while. I would like to see how things will work with some of stuff for both stable & -current. Doing a lot of research lately, trying to get things together before really attempting.
Any help or assistance would be appreciated.

:hattip:

GrapefruiTgirl 10-28-2010 09:50 AM
Hi onebuck,

I have not built FF PGO on Slackware 13.1 at all; I have been using 64-current and just using the FF that comes with that. As such, the script is still as it was last time I used it.

However I am confident that you have enough experience/skill to accomplish the upgrade(s) to the script if you want to use it. :)
I just had a good read of the patch file above provided by *Arcanine, and what I would do right now if I wanted to upgrade my build script, would be (roughly):

1) to apply that patch to the script.

2) compare (diff?) the original 13.0 FF.slackbuild to the 13.1-current slackbuild (or most recent 13.1 slackbuild) and note any relevant differences and then see #3 next:

3) update the PGO script in any slackware-specific areas (configure --options, patches, etc..) that need adjustment per step #2

4) update the PGO script to use the current filenames of today's firefox source archive name that you download, & any related filenames for the other files necessary, as well as the final package name and version numbers, where necessary.

That should put it pretty close to being a working -current version. If you do this before I get around to it, and run into troubles, I'll be happy to try to assist you if I have an idea what the problem is. I'll leave it up to you if you start a new thread or not at that time, but if I happen to update the script & test it before hearing back from you on it, I'll post the updated version in here.

As for links & references of what I read while coming up with this, I do not believe I kept specific records of the sites that helped, other than the mozilla.org site mentioned in the script. I just Googled, and noted & tried bits & pieces taken from various sources, and by trial and error, figured out what worked. The information is not obscure-- Google returns most of the relevant stuff in its top 10 pages or so, with the right query.

Good luck!


All times are GMT -5. The time now is 07:19 PM.