git repository over http?

git repository over http?

[peter.frings]

Good morning all, just a quicky.

I can’t seem to clone the git repository over http (from http://orgmode.org/org-mode.git). Is the server not supporting http, or is it just me?

Thanks,
Peter.
-- 
c++;         // this makes c bigger but returns the old value

Re: git repository over http?

[Bastien]

Hi Peter,

peter.frings@agfa.com writes:

> I can’t seem to clone the git repository over http (from
> http://orgmode.org/org-mode.git). Is the server not supporting http, or is
> it just me?

The server isn't supporting the http protocol -- you can use 
http://repo.or.cz/w/org-mode.git instead, which lags behind by
just ~1 hour.

HTH,

-- 
 Bastien

Re: git repository over http?

[Jonathan Leech-Pepin]

[-- Attachment #1: Type: text/plain, Size: 1521 bytes --]

Hello Bastien,

The server does seem to support HTTP for pulling, however it is very slow. I
have to use http from work because the firewall blocks the git:// protocol.
Times on my prompt are DST (-0400).

[10:22:25] ~/test$ git clone http://orgmode.org/org-mode.git
Cloning into org-mode...
[10:35:43] ~/test$ cd org-mode/
[10:36:46] ~/test/org-mode$ git log -1
commit 74cca9f57ee3896dad85d3b354d1d9db286e799b
Author: Nicolas Goaziou <n.goaziou@gmail.com>
Date: Thu Aug 18 15:44:46 2011 +0200

org-indent: silence byte-compiler
[10:36:57] ~/test/org-mode$ cat .git/config
[core]
repositoryformatversion = 0
filemode = true
bare = false
logallrefupdates = true
ignorecase = true
[remote "origin"]
fetch = +refs/heads/*:refs/remotes/origin/*
url = http://orgmode.org/org-mode.git
[branch "master"]
remote = origin
merge = refs/heads/master

http:// does work for cloning and pulling from the main Repo, however
there's nothing to indicate that anything is happening until the task is
complete.

Regards,

Jonathan

On Wed, Aug 17, 2011 at 3:52 AM, Bastien <bzg@altern.org> wrote:

> Hi Peter,
>
> peter.frings@agfa.com writes:
>
> > I can’t seem to clone the git repository over http (from
> > http://orgmode.org/org-mode.git). Is the server not supporting http, or
> is
> > it just me?
>
> The server isn't supporting the http protocol -- you can use
> http://repo.or.cz/w/org-mode.git instead, which lags behind by
> just ~1 hour.
>
> HTH,
>
> --
>  Bastien
>
>

[-- Attachment #2: Type: text/html, Size: 3699 bytes --]

Re: git repository over http?

[Bastien]

Hi Jonathan,

Jonathan Leech-Pepin <jonathan.leechpepin@gmail.com> writes:

> The server does seem to support HTTP for pulling, however it is very
> slow. 

Yes, I noticed this too.

> http:// does work for cloning and pulling from the main Repo, however
> there's nothing to indicate that anything is happening until the task
> is complete.

Yes.

Jason, is the slowliness something worth trying to fix?   
Can we get some indication that the repo is currently being
cloned?

Thanks!

-- 
 Bastien

Re: git repository over http?

[Jason Dunsmore]

Bastien <bzg@altern.org> writes:

> Hi Jonathan,
>
> Jonathan Leech-Pepin <jonathan.leechpepin@gmail.com> writes:
>
>> The server does seem to support HTTP for pulling, however it is very
>> slow. 
>
> Yes, I noticed this too.
>
>> http:// does work for cloning and pulling from the main Repo, however
>> there's nothing to indicate that anything is happening until the task
>> is complete.
>
> Yes.
>
> Jason, is the slowliness something worth trying to fix?   
> Can we get some indication that the repo is currently being
> cloned?

I ran some tests and found that the download speed is proportional to
the size of the repo:

--8<---------------cut here---------------start------------->8---
$ time /usr/bin/git clone http://orgmode.org/org-mode.git
Cloning into org-mode...

real    15m23.475s
user    0m30.434s
sys     0m7.588s

$ time /usr/bin/git clone git://orgmode.org/org-mode.git org-mode2
Cloning into org-mode2...
remote: Counting objects: 51864, done.
remote: Compressing objects: 100% (14898/14898), done.
remote: Total 51864 (delta 41315), reused 46129 (delta 36900)
Receiving objects: 100% (51864/51864), 53.00 MiB | 205 KiB/s, done.
Resolving deltas: 100% (41315/41315), done.

real    5m19.350s
user    0m19.341s
sys     0m2.576s

$ time /usr/bin/git clone http://repo.or.cz/r/org-mode.git org-mode3
Cloning into org-mode3...

real    7m15.211s
user    0m21.293s
sys     0m3.652s

$ du -sh org-mode*
201M    org-mode
63M     org-mode2
70M     org-mode3
--8<---------------cut here---------------end--------------->8---

If you clone via git://, it does some optimizations during the transfer,
whereas cloning via http:// does not.  If I do a "git gc" on the repo,
it reduces the size of the repo on the server and the time to download
via http://

--8<---------------cut here---------------start------------->8---
$ cp -r /home/orgmode/org-mode.git/ /var/www/orgmode.org/org-mode-copy.git
$ cd /var/www/orgmode.org/org-mode-copy.git
$ du -sh
221M    .
$ git gc
$ du -sh
65M     .

$ time /usr/bin/git clone http://orgmode.org/org-mode-copy.git
Cloning into org-mode-copy...

real    5m6.472s
user    0m18.797s
sys     0m2.300s

$ du -sh org-mode-copy
63M     org-mode-copy
--8<---------------cut here---------------end--------------->8---

I'd recommend using the repo.or.cz repo for http:// downloads.
repo.or.cz probably runs git-http-backend or does a "git gc"
occasionally.

Any user feedback for "git clone" would be up to the git client.  It
used to say "got... walk..." prior to 1.7.  Not sure why they removed
those messages in 1.7.

Regards,
Jason

Re: git repository over http?

[Bastien]

Hi Jason,

Jason Dunsmore <emacs-orgmode@dunsmor.com> writes:

> I ran some tests and found that the download speed is proportional to
> the size of the repo:

Good to know, thanks.

> If you clone via git://, it does some optimizations during the transfer,
> whereas cloning via http:// does not.  If I do a "git gc" on the repo,
> it reduces the size of the repo on the server and the time to download
> via http://

Okay -- can you "git gc" on the server?

> I'd recommend using the repo.or.cz repo for http:// downloads.

+1

Thanks for these informations!

-- 
 Bastien

Re: git repository over http?

[Achim Gratz]

Jason Dunsmore <emacs-orgmode@dunsmor.com> writes:
> If you clone via git://, it does some optimizations during the transfer,
> whereas cloning via http:// does not.  If I do a "git gc" on the repo,
> it reduces the size of the repo on the server and the time to download
> via http://

I'd suggest that the repo on the server should get a regular garbage
collect (daily, perhaps via cron) and on each release an aggressive
garbage collect.  It will be much easier on the server (reduced traffic
and fewer requests to serve).

Not only is the size of the repo much larger than it needs to be, the
server also has to deal with lots of small requests that can't be
efficiently transferred.  Cloning the repo via http protocol produces
190MByte on disk, a simple and fast 'git gc' gets that down to 63MByte,
just like cloning via git protocol (ironically, the server will do that
work on each clone as long as you do not do a garbage collect on the
repo itself).  An aggressive garbage collect takes a while longer, but
reduces the size of the repo down to 44MByte.

An http clone on repo.or.cz currently delivers 72MByte, so it appears
they do a regular garbage collect (or it simply comes courtesy of
mirroring).


Regards,
Achim.
-- 
+<[Q+ Matrix-12 WAVE#46+305 Neuron microQkb Andromeda XTk Blofeld]>+

Factory and User Sound Singles for Waldorf rackAttack:
http://Synth.Stromeko.net/Downloads.html#WaldorfSounds

Re: git repository over http?

[Jason Dunsmore]

Bastien <bzg@altern.org> writes:

> Hi Jason,
>
> Jason Dunsmore <emacs-orgmode@dunsmor.com> writes:
>
>> I ran some tests and found that the download speed is proportional to
>> the size of the repo:
>
> Good to know, thanks.
>
>> If you clone via git://, it does some optimizations during the transfer,
>> whereas cloning via http:// does not.  If I do a "git gc" on the repo,
>> it reduces the size of the repo on the server and the time to download
>> via http://
>
> Okay -- can you "git gc" on the server?

Okay, done:

--8<---------------cut here---------------start------------->8---
orgmode@org:~$ du -sh org-mode.git
225M    org-mode.git

orgmode@org:~/org-mode.git$ find objects/|wc -l
6523

orgmode@org:~/org-mode.git$ git gc
Counting objects: 52023, done.
Delta compression using up to 4 threads.
Compressing objects: 100% (14474/14474), done.
Writing objects: 100% (52023/52023), done.
Total 52023 (delta 41421), reused 46864 (delta 37483)
Removing duplicate objects: 100% (256/256), done.

orgmode@org:~/org-mode.git$ du -sh
56M     .

orgmode@org:~/org-mode.git$ find objects/|wc -l
66
--8<---------------cut here---------------end--------------->8---

I lowered the git auto-gc threshold from the default of 6700 (loose
objects) to 1000.


orgmode@org:~/org-mode.git$ git config gc.auto 1000


Regards,
Jason

Re: git repository over http?

[Bastien]

Jason Dunsmore <emacs-orgmode@dunsmor.com> writes:

>> Okay -- can you "git gc" on the server?
>
> Okay, done:

Great -- thanks a lot!

-- 
 Bastien