how to reduce time of git pulling each time when you do a make world on Xen sourc

tkmsr · 02-26-2011, 11:18 AM

I am compiling xen from source and each time I do a make world
it basically gives some or the other error
my problem are not those errors ( I am trying to debug them)
but the problem is each time when I do a make world

Xen basically pulls things from git repository

Code:

   + rm -rf linux-2.6-pvops.git linux-2.6-pvops.git.tmp
   + mkdir linux-2.6-pvops.git.tmp
   + rmdir linux-2.6-pvops.git.tmp
   + git clone -o xen -n
git://git.kernel.org/pub/scm/linux/kernel/git/jeremy/xen.git
linux-2.6-pvops.git.tmp
   Initialized empty Git repository in
/usr/src/xen-4.0.1/linux-2.6-pvops.git.tmp/.git/
   remote: Counting objects: 1941611, done.
   remote: Compressing objects: 100% (319127/319127), done.
remote: Total 1941611 (delta 1614302), reused 1930655 (delta 1604595)
**Receiving objects: 20% (1941611/1941611), 98.17 MiB | 87 KiB/s, done.**

and if you notice the last line it is still consuming my bandwidth
pulling things from internet.How can I stop this step each time and
use existing git repository?
When I searched an answer to this question I came across a link
http://lists.xensource.com/archives/.../msg00476.html
http://web.archiveorange.com/archive...CfIzdpcmkWRz5w

but I am not able to clearly understand where Boris mentions

Quote:

make xen etc.
"Then clone JF's Git Repo, checkout git branch as you want and build
corresponding
kernel.That's all i always do."

Which kernel is he referring to is it Dom0?
because when I ran make xen it worked without any problem and I can see in my
/usr/src/xen-4.0.1/dist/install/boot

Quote:

xen-4.0.1.gz xen-4.0.gz xen-4.gz xen.gz xen-syms-4.0.1

but I am not clear with how to save this git pull each time.
I have given a link which I think is a solution to my problem but I am not able understand what they have mentioned.Can some one explain that.

andrewthomas · 03-06-2011, 06:50 PM

I don't use a xen kernel, but I compile git kernels all the time. You probably should post the script that you are using.

What you want to do is not remove the sources every build.

Quote:

Originally Posted by tkmsr

Code:

   + rm -rf linux-2.6-pvops.git linux-2.6-pvops.git.tmp
   + mkdir linux-2.6-pvops.git.tmp
   + rmdir linux-2.6-pvops.git.tmp
   + git clone -o xen -n
git://git.kernel.org/pub/scm/linux/kernel/git/jeremy/xen.git
linux-2.6-pvops.git.tmp
   Initialized empty Git repository in
/usr/src/xen-4.0.1/linux-2.6-pvops.git.tmp/.git/
   remote: Counting objects: 1941611, done.
   remote: Compressing objects: 100% (319127/319127), done.
remote: Total 1941611 (delta 1614302), reused 1930655 (delta 1604595)
**Receiving objects: 20% (1941611/1941611), 98.17 MiB | 87 KiB/s, done.**

You could have a build directory and clean sources directory.

I don't know if they rebase the xen kernel often, if they don't you could just
to begin

Code:

mkdir build
cd build 
git clone git://git.kernel.org/pub/scm/linux/kernel/git/jeremy/xen.git
git clone xen xen-build
cd xen-build 
... do your magic here

then to update

Code:

cd build
rm -rf xen-build
git pull
git clone xen xen-build
cd xen-build
...do whatever you need to do

As I said, I am don't really know much about the xen kernel.

For instance, I don't know what branch you are building (master?)

If you would post the script, you certainly will get some help.

It defeats the purpose of using git to get the kernel source if you are removing and downloading the entire source each build.

tkmsr · 03-06-2011, 10:39 PM

Quote:

Originally Posted by andrewthomas

I don't use a xen kernel, but I compile git kernels all the time. You probably should post the script that you are using.

Well I am not using any script.
I have some doubt with respect to usage of git.

Quote:

Originally Posted by andrewthomas

I don't know if they rebase the xen kernel often, if they don't you could just
to begin
mkdir build
cd build
git clone git://git.kernel.org/pub/scm/linux/kernel/git/jeremy/xen.git
git clone xen xen-build
cd xen-build
... do your magic here
then to update

With respect to above suggestion of your the Makefile that comes with Xen is here
http://lxr.xensource.com/lxr/source/Makefile

Quote:

Originally Posted by andrewthomas

cd build
rm -rf xen-build

Want to know if xen-build is a command or you just gave an example to me.

Quote:

Originally Posted by andrewthomas

git pull
git clone xen xen-build
cd xen-build
...do whatever you need to do[/CODE]

Quote:

Originally Posted by andrewthomas

As I said, I am don't really know much about the xen kernel.

For instance, I don't know what branch you are building (master?)

I really do not understand git what I found is following happens each time I do a make world

Code:

git clone git://git.kernel.org/pub/scm/linux/kernel/git/jeremy/xen.git linux-2.6-xen

and at each build it will download 900 Mb.Which is very costly for my internet.
Well it will help me if you can explain the above portion I have been trying to understand Xen and it has been very confusing.
What is the meaning of branches in git.

Quote:

git checkout -b xen/stable-2.6.32.x origin/xen/stable-2.6.32.x

The Xen page which talks such things is here
http://wiki.xensource.com/xenwiki/XenParavirtOps

Quote:

Originally Posted by andrewthomas

If you would post the script, you certainly will get some help.

Ok here is Xen source code
http://lxr.xensource.com/lxr/source
and what I found is the lines responsible for the pull are here
http://lxr.xensource.com/lxr/source/....git-clone#L25

Quote:

Originally Posted by andrewthomas

It defeats the purpose of using git to get the kernel source if you are removing and downloading the entire source each build.

You are right that is my problem I do not have such an internet connection to git clone 900 Mb each time when I build.
What I did is now by the time I saw your message I commented out following lines from
buidlconfig/src.git-clone directory of Xens tar ball.

Code:

  if ! [ -d $(LINUX_SRCDIR) ]; then \
 26                 rm -rf $(LINUX_SRCDIR) $(LINUX_SRCDIR).tmp; \
 27                 mkdir $(LINUX_SRCDIR).tmp; rmdir $(LINUX_SRCDIR).tmp; \
 28                 $(GIT) clone -o $(XEN_GIT_ORIGIN) -n $(XEN_LINUX_GIT_URL) $(LINUX_SRCDIR).tmp; \
 29                 (cd $(LINUX_SRCDIR).tmp; git checkout -b $(XEN_LINUX_GIT_LOCALBRANCH) $(XEN_LINUX_GITREV) ); \
 30                 mv $(LINUX_SRCDIR).tmp $(LINUX_SRCDIR); \
 31         fi

So now the git pull does not happen each time.But I do not know if above is the correct thing to do.

andrewthomas · 03-07-2011, 09:07 AM

The first question I have is why are you making world?

Quote:

@echo ' world - clean everything, delete guest kernel build'

this is why it cleans out the directory and you have to clone the entire git repo every time.

Downloading the git tree

Code:

$ git clone git://git.kernel.org/pub/scm/linux/kernel/git/jeremy/xen.git linux-2.6-xen
$ cd linux-2.6-xen

Changing the branch: You most probably want to use the "xen/stable-2.6.32.x" branch, so do this:

Code:

$ cd linux-2.6-xen
$ git reset --hard
$ git checkout -b xen/stable-2.6.32.x origin/xen/stable-2.6.32.x
Branch xen/stable-2.6.32.x set up to track remote branch refs/remotes/origin/xen/stable-2.6.32.x.
Switched to a new branch "xen/stable-2.6.32.x"
$ git pull
$ git log | less

Later when you want to update the tree use:

Code:

$ cd linux-2.6-xen
$ make clean
$ git pull

Example how to compile/build the pv_ops dom0 kernel:

Code:

cd linux-2.6-xen
make clean
cp -a .config .config-old
wget -O .config <http_url_to_config_file>
make oldconfig
make menuconfig (if you need to change something)
make bzImage
make modules
make modules_install

# in the following lines replace "version" with the actual kernel version you're compiling.
cp -a .config /boot/config-version
cp -a System.map /boot/System.map-version
cp -a arch/x86/boot/bzImage /boot/vmlinuz-version

# And then generate initrd/initramfs image for your dom0 kernel, example for Fedora/RHEL/CentOS:
mkinitrd -f /boot/initrd-version.img version

tkmsr · 03-07-2011, 11:09 AM

Quote:

Originally Posted by andrewthomas

The first question I have is why are you making world?
this is why it cleans out the directory and you have to clone the entire git repo every time.
Downloading the git tree

Ok thanks for this one I was not able to understand based on some blogs wiki etc I was doing it.

Quote:

Originally Posted by andrewthomas

Changing the branch: You most probably want to use the "xen/stable-2.6.32.x" branch, so do this:
$ cd linux-2.6-xen
$ git reset --hard
$ git checkout -b xen/stable-2.6.32.x origin/xen/stable-2.6.32.x
Branch xen/stable-2.6.32.x set up to track remote branch refs/remotes/origin/xen/stable-2.6.32.x.
Switched to a new branch "xen/stable-2.6.32.x"
$ git pull
$ git log | less
Later when you want to update the tree use:
$ cd linux-2.6-xen
$ make clean
$ git pull

Ok I am not that used to with git so I will search that out.

I am clear with kernel compile process.I was not clear with make world part that you made clear.
Basically I think process of compiling a pv-ops dom0 kernel is same as compiling a normal kernel.
Xen I need to compile separately.
I was not clear with compiling of Xen that is where every thing was giving me confusion as make world etc was mentioned here which gave rise to all the confusion http://wiki.xensource.com/xenwiki/Xen4.0