Re: [PATCH] (ELPA): Support for building Package Variants and Package Upload

[Jambunathan K <kjambunathan@gmail.com>]

Carsten Dominik <carsten.dominik@gmail.com> writes:

> Hi Jambunathan,
>
> thanks for this!
>
>> 2.2 Build Procedure
>> ====================
>>   1. make PKG_TAG=7.3
>>      This builds org-7.3.tar
>
> I guess this must be
>
>  make pkg PKG_TAG=...
>
> right?
>

Yes, I missed the target - 'pkg'. A small but a significant slip.

> I think it would be good to *not* use a name that is just like the
> normal org distribution files.  

There is a config variable for that :-). It is PKG_BNAME (as in 'Package
Base Name'). 


> I am distributing under the names
>
> org-7.3.tar.gz
> org-7.3.zip
>

I believe the question here is: How is an ELPA-tar differntiated from a
plain-old snapshot tar? From the user perspective, is there an element
of ambiguity and confusion?

As things stand today, ELPA-tarballs are plain tar files i.e., they are
not zipped. So the packages with above names can only be non-ELPA
tarballs [1]

The only difference between ELPA and non-ELPA tarballs is in the tree
structure. ELPA-tarballs *require* that lisp files and info files be in
the root directory [2].  This is not the case with plain old
tarballs. As it stands today, the 'contrib' files are not zipped in to
the ELPA tarball [3].

One approach of addressing a possible divide between ELPA and non-ELPA
tarballs is to distribute just a single tarball which is also
ELPA-tarball. This would require a modification in the tree-structure of
the conventional tarball and may be some changes to the Makefile.

A second approach of addressing the problem is to wait and watch. May be
by the time Emacs-24 gets out of the door, package manager would not be
too restrictive on where it wants to see the elisp and info files. And a
plain old tar file could as well work well with package manager.

> I am distributing under the names
>
> org-7.3.tar.gz
> org-7.3.zip
>
> or is this not a possible problem?

The advantage of using 'org' namespace for the ELPA tarballs is that it
would be deemed as an 'upgrade' to the builtin package 'org'. In that
sense there is a continuity. This privilege would be lost for packages
that doesn't share 'builtins' namespace [4].

As package are identified by their names, a org-snapshot-7.3.tar is
considered by package manager interface as different from org-7.3.tar.

AFAICS, packages get in to '~/.emacs.d/elpa/org-snapshot-7.3/...' and
'~/.emacs.d/elpa/org-7.3/...' and load-path is appropriately modified.

So a reasonably experienced user can have them installed 'in parallel'
and pick and choose what he uses by playing around with load-path. He
can also tie down the package manager not to activate the files, thus
minimizing the package manager functionality to that of a downloader,
compiler and unpacker but not as a 'loader' [5].


,----[ C-h f package-initialize RET ]
| package-initialize is an interactive Lisp function in `package.el'.
| 
| (package-initialize &optional NO-ACTIVATE)
| 
| Load Emacs Lisp packages, and activate them.
| The variable `package-load-list' controls which packages to load.
| If optional arg NO-ACTIVATE is non-nil, don't activate packages.
| 
| [back]
`----


> I am distributing under the names
>
> org-7.3.tar.gz
> org-7.3.zip

One another way to interpret the question is how would one distinguish
between tarballs that are 'official' (as in released directly by the
maintainer from his official website) and those that are 'unofficial'
(as in forks and derived works).

For the sake of recording my Makefile changes, as well as for the sake
of argument, I could potentially use the following setting:

PKG_BNAME=org
PKG_SEP=@
PKG_FLAVOUR=OpenOffice

and get a org@OpenOffice-20101112.tar.

This is where the notion of 'Variants/Flavours/Release Streams/' could
be useful. The string between '@' and 'the version number' could be
standardized as a variant. The advantage of variants is obvious [6].

[1] Possible that Package Manager might start supporting zipped files.

[2] This is a constraint only because 'package manager' is barely
    born. Who knows it can become a superman very soon ...

[3] This is so because just because it was done so. Package Manager
    provides support for packaging arbitrary data files - icons, sound
    files etc etc. There is no reason (apart from download size) which
    prevent one from including contrib files.

[4] I haven't experimented much with builtin packages, upgrades,
    downgrades etc. Take my remarks here with a pinch of salt. May be
    someone who has experimented with package manager could pitch in
    with additional insights here ... I am forming my opinion as I go
    along.

[5] In some sense package manager is a good 'eye candy'. It is not my
    intention to use the word in a negative sense but only as a
    convenient metaphor.

[6] This is something that I have proposed to emacs-devel. Variants etc
    are purely my own terminology i.e., it doesn't have any official
    sanction. See
 
    http://lists.gnu.org/archive/html/emacs-devel/2010-10/msg01026.html

    This is also one of the reasons why I had commented out 'PKG_SEP'
    and 'PKG_FLAVOURS'.

    One of biggest advantage of having variants is to support both
    snapshot and stable tarballs that could be simultaneously released
    with obvious advantages.
    

Jambunathan K.

>
> - Carsten
>
>
>>   2. make pkg
>>      This builds org-20101112.tar
>>
>> 3 Building Package Variants
>> ~~~~~~~~~~~~~~~~~~~~~~~~~~~~
>>
>> 3.1 Notes
>> ==========
>>    Modify/Uncomment PKG_SEP and PKG_FLAVOUR setting in Makefile as
>>    desired.
>>
>>    There is a choice of PKG_SEP. Pick what you like. Use of '-' is
>>    discouraged.
>>
>>    Assumption is:
>>    - stable releases are built with an explicit PKG_TAG
>>    - stable releases are non-flavored while the snapshot builds are
>>      flavored.
>>
>> 3.2 Build Procedure
>> ====================
>>   1. make PKG_TAG=7.3
>>      This builds org-7.3.tar
>>   2. make pkg
>>      This builds org@snaphsot-20101112.tar
>>
>> 4 Package Uploads
>> ~~~~~~~~~~~~~~~~~~
>>
>> 4.1 Notes
>> ==========
>>   - PKG_UPLOAD_DIR: directory where the package files are
>>     uploaded. i.e., it is the *local directory* behind the repo URL
>>     ([http://orgmode.org/pkg/daily/]). For eg, /var/www/pub/pkg/
>> daily/.
>>
>>     IMPORTANT NOTE: Remember to include the trailing '/' in the
>>     directory while setting this variable.
>>
>>   - PKG_EL_DIR: This is added to the load-path during BATCH
>>     compilation. The intention is two fold:
>>
>>     1. load package.el and package-x.el if the build machine is using
>>        Emacs-23 or less (which don't ship with package manager)
>>
>>     2. Override package.el and package-x.el that ship with Emacs with
>>        a custom or a patched up version.
>>
>>     IMPORTANT NOTE: The rule for 'pkg-upload' depends on my patch to
>>     Package Manager which is accepted but not committed yet because
>>     of the (postal?) delay in FSF copyright requirements.
>>
>>    [http://lists.gnu.org/archive/html/emacs-devel/2010-11/msg00026.html
>> ]
>>
>>
>> 4.2 Build Procedure
>> ====================
>>
>>   1. make PKG_EL_DIR="~/elisp" PKG_UPLOAD_DIR="~/packages/"
>> PKG_TAG=7.3 pkg
>>
>>      This builds org-7.3.tar and uploads the same to "~/packages"
>>      using package-x.el in "~/elisp/"
>>
>>   2. make PKG_EL_DIR="~/elisp" PKG_UPLOAD_DIR="~/packages/" pkg
>>
>>      This builds org-20101112.tar and uploads the same to
>>      "~/packages" using package-x.el in "~/elisp/"
>>
>> Jambunathan K.
>>
>> Attachments
>> ~~~~~~~~~~~
>>
>> From 06d227cc4d4e3ac86320690ebbe184bb804d9bf1 Mon Sep 17 00:00:00 2001
>> From: Jambunathan K <kjambunathan@gmail.com>
>> Date: Fri, 12 Nov 2010 16:32:51 +0530
>> Subject: [PATCH] (ELPA): Support for building Package Variants and
>> Package Upload.
>>
>> (PKG_TAG): Use PKG_TAG to control package flavour. If unspecified,
>> assume a snapshot build.
>>
>> (PKG_SEP, PKG_FLAVOUR): New config variables for building various
>> flavours of ELPA tarball. For eg., org-7.3.tar,
>> org@snapshot-20101112.tar, org-snapshot-20101112 etc.
>>
>> (PKG_FILES): Removed
>> (PKG_BASE_FILES): Files here end up in the root dir.
>> (PKG_DOC_FILES): Files here end up in the doc dir.
>>
>> (PKG_README_ORG_FILE): README file
>>
>> (PKG_EL_DIR): Load path for package.el and package-x.el. Useful if the
>> build machine runs an 'old' Emacs or if the build process requires a
>> patched up versions of these files.
>>
>> (PKG_UPLOAD_DIR): 'pub' dir of the webserver serving the ELPA tarball.
>>
>> (pkg-upload, pkg-readme): New targets subsumed within pkg target.
>> ---
>> diff --git a/Makefile b/Makefile
>> old mode 100644
>> new mode 100755
>> index 0b15abe..5c13d8b
>> --- a/Makefile
>> +++ b/Makefile
>> @@ -165,18 +165,44 @@ TEXIFILES   = doc/org.texi
>> INFOFILES   = doc/org
>>
>> # Package Manager (ELPA)
>> +
>> +# PKG_TAG =
>> +
>> +ifndef PKG_TAG
>> +# PKG_SEP = @
>> +# PKG_SEP = -
>> +# PKG_SEP = .
>> +# PKG_FLAVOUR = snapshot
>> PKG_TAG = $(shell date +%Y%m%d)
>> +endif
>> +
>> +PKG_BNAME = org
>> +PKG_QNAME = $(PKG_BNAME)$(PKG_SEP)$(PKG_FLAVOUR)
>> +PKG_FNAME = $(PKG_QNAME)-$(PKG_TAG)
>> +
>> PKG_DOC = "Outline-based notes management and organizer"
>> PKG_REQ = "nil"
>>
>> -PKG_FILES = $(LISPFILES0)		\
>> -            doc/dir doc/org		\
>> -            doc/pdflayout.sty		\
>> -            doc/org.pdf			\
>> -            doc/orgguide.pdf		\
>> -            doc/orgcard.tex		\
>> -            doc/orgcard.pdf		\
>> -            doc/orgcard_letter.pdf
>> +PKG_BASE_FILES = $(LISPFILES0)          \
>> +                 doc/dir doc/org
>> +
>> +PKG_DOC_FILES =                         \
>> +                doc/org.pdf             \
>> +                doc/org.html            \
>> +                doc/orgguide.pdf        \
>> +                doc/orgcard.txt         \
>> +                doc/orgcard.pdf         \
>> +                doc/orgcard_letter.pdf
>> +
>> +PKG_README_ORG_FILE = ORGWEBPAGE/Changes.org
>> +PKG_README_TXT_FILE = $(PKG_README_ORG_FILE:%.org=%.txt)
>> +
>> +# PKG_EL_DIR =
>> +# PKG_UPLOAD_DIR =
>> +
>> +ifdef PKG_EL_DIR
>> +BATCH_EXTRA = -eval "(setq load-path (cons (expand-file-name \"$
>> (PKG_EL_DIR)\") load-path))"
>> +endif
>>
>> .SUFFIXES: .el .elc .texi
>> SHELL = /bin/sh
>> @@ -372,16 +398,46 @@ distfile:
>> 	zip -r org-$(TAG).zip org-$(TAG)
>> 	gtar zcvf org-$(TAG).tar.gz org-$(TAG)
>>
>> -pkg:
>> +pkg:
>> 	@if [ "X$(PKG_TAG)" = "X" ]; then echo "*** No tag ***"; exit 1; fi
>> +	echo "Building $(PKG_FNAME).tar ..."
>> 	touch doc/org.texi doc/orgcard.tex # force update
>> 	${MAKE} info
>> 	${MAKE} doc
>> -	rm -rf org-$(PKG_TAG) org-$(PKG_TAG).tar
>> -	$(MKDIR) org-$(PKG_TAG)
>> -	cp -r $(PKG_FILES) org-$(PKG_TAG)
>> -	echo "(define-package \"org\" \"$(PKG_TAG)\" \"$(PKG_DOC)\" $
>> (PKG_REQ))" > org-$(PKG_TAG)/org-pkg.el
>> -	tar cf org-$(PKG_TAG).tar org-$(PKG_TAG) --remove-files
>> +	rm -rf $(PKG_FNAME) $(PKG_FNAME).tar
>> +	$(MKDIR) $(PKG_FNAME)
>> +	cp -r $(PKG_BASE_FILES) $(PKG_FNAME)
>> +	$(MKDIR) $(PKG_FNAME)/doc
>> +	cp -r $(PKG_DOC_FILES) $(PKG_FNAME)/doc
>> +	${MAKE} pkg-readme
>> +	echo "(define-package \"$(PKG_QNAME)\" \"$(PKG_TAG)\"
>> \"$(PKG_DOC)
>> \" $(PKG_REQ))" > $(PKG_FNAME)/$(PKG_QNAME)-pkg.el
>> +	tar cf $(PKG_FNAME).tar $(PKG_FNAME) --remove-files
>> +	${MAKE} pkg-upload
>> +
>> +pkg-readme:
>> +	$(MKDIR) $(PKG_FNAME)	
>> +	cp $(PKG_README_ORG_FILE) $(PKG_FNAME)/README.org
>> +	$(BATCH)						\
>> +	--eval "(require 'org-exp)"				\
>> +	--eval "(require 'org-ascii)"				\
>> +	--visit=$(PKG_FNAME)/README.org				\
>> +	--eval "(org-export-as-ascii 3 t)"
>> +	cp $(PKG_FNAME)/README.txt $(PKG_FNAME)/README
>> +	rm -f $(PKG_FNAME)/README.txt $(PKG_FNAME)/README.org
>> +
>> +pkg-upload:
>> +	@if [ "X$(PKG_UPLOAD_DIR)" = "X" ];					\
>> +	then echo "*** WARNING: Skipping Upload of $(PKG_FNAME).tar  ***";	\
>> +	else									\
>> +	echo "Uploading $(PKG_FNAME).tar to $(PKG_UPLOAD_DIR) ...";		\
>> +	$(MKDIR) $(PKG_UPLOAD_DIR);						\
>> +	$(BATCH) $(BATCH_EXTRA)							\
>> +	  --eval '(ignore)'							\
>> +	  --eval "(require 'package-x)"						\
>> +	  --eval '(setq package-archive-upload-base "$(PKG_UPLOAD_DIR)")'	\
>> +	  --eval '(setq package-update-news-on-upload nil)'			\
>> +	  --eval '(package-upload-file "$(PKG_FNAME).tar")';			\
>> +	fi
>>
>> makerelease:
>> 	@if [ "X$(TAG)" = "X" ]; then echo "*** No tag ***"; exit 1; fi
>> --
>> 1.7.2.3
>>
>> _______________________________________________
>> Emacs-orgmode mailing list
>> Please use `Reply All' to send replies to the list.
>> Emacs-orgmode@gnu.org
>> http://lists.gnu.org/mailman/listinfo/emacs-orgmode