About org-mode-tests and CI

About org-mode-tests and CI

[Simon Tournier]

Hi,

Recently, Bastien told me about using GNU Guix for some tests of Org.
Neat!  Then, Bastien pointed this org-mode-tests [1] effort.

Unrelated, Philip provides Guix recipes [2] for various old Emacs
versions.

Org and Guix are part of the GNU system.  Therefore, we could imagine to
bridge instead of relying on Debian. :-)

Moreover, what would be another advantage?  Run the exact same
computational environment from locally to SourceHut builds.

Two directions:

1. The SourceHut image of Guix [3] could be used but – and maybe I am
missing a point since I am not an expert about SourceHut CI – the state
(revision) of this image is not controlled and thus it requires
something like:

        image: guix
        tasks:
          - guix: |
              guix pull -C project/path/to/channels.scm

Well, I do not know how SourceHut is caching but somehow the .yml
configuration leads to always the same computational environment
(image), in which “make test” is run.  Therefore, the CI could spend
more time in computing again and again this fixed state than running the
Org test suite. :-)

2. Using [2], it appears to me almost straightforward to build
beforehand a Docker pack containing all the requirements; say emacs@26,
curl, gcc-toolchain, git, etc.  And this Docker pack would be built
using GNU Guix,

    guix pack -f docker -m manifest.scm

where the file manifest.scm lists all the requirements.  Using adequate
option as --save-provenance, this Docker pack can be inspectable [4] and
it could be stored to any Docker registery.

Hence, the line,

        image: debian/oldstable

or some images as,

        image: ubuntu/focal
        repositories:
          emacs: http://ppa.launchpad.net/kelleyk/emacs/ubuntu focal main 3FF0E01EEAAFC9CD

would be replaced by some images produced by “guix pack -f docker”
stored to some Docker registery.


All in all, I forked the project [1] but the SourceHuts build (CI)
requires some fee, right?  Well, let me know how we could test this
approach of using Guix as base for running Org test suite.

(The maintenance of such can be part of the story too. ;-))


Last, without putting the cart before the horse, I think this work could
be a kind of preliminary proof-of-concept for testing Emacs packages
(ELPA, MELPA, etc.).

Cheers,
simon

1: https://git.sr.ht/~bzg/org-mode-tests
2: https://git.sr.ht/~pkal/guix-emacs-historical
3: https://man.sr.ht/~dhruvin/builds.sr.ht-guix-cookbook/
4: https://hpc.guix.info/blog/2021/10/when-docker-images-become-fixed-point/

Re: About org-mode-tests and CI

[Philip Kaludercic]

Simon Tournier <zimon.toutoune@gmail.com> writes:

> Hi,
>
> Recently, Bastien told me about using GNU Guix for some tests of Org.
> Neat!  Then, Bastien pointed this org-mode-tests [1] effort.
>
> Unrelated, Philip provides Guix recipes [2] for various old Emacs
> versions.

I cannot find the discussion right now, but I should point out that for
the intent of testing, if the code depends on any external packages, it
will also be necessary to provide packages for every version so that the
compiled byte code, to avoid byte-code incompatibility.

Maybe https://yhetil.org/guix-devel/87y21n3x99.fsf@posteo.net/ could be
of use?

> Org and Guix are part of the GNU system.  Therefore, we could imagine to
> bridge instead of relying on Debian. :-)
>
> Moreover, what would be another advantage?  Run the exact same
> computational environment from locally to SourceHut builds.
>
> Two directions:
>
> 1. The SourceHut image of Guix [3] could be used but – and maybe I am
> missing a point since I am not an expert about SourceHut CI – the state
> (revision) of this image is not controlled and thus it requires
> something like:
>
>         image: guix
>         tasks:
>           - guix: |
>               guix pull -C project/path/to/channels.scm
>
> Well, I do not know how SourceHut is caching but somehow the .yml
> configuration leads to always the same computational environment
> (image), in which “make test” is run.  Therefore, the CI could spend
> more time in computing again and again this fixed state than running the
> Org test suite. :-)

This is also what I intend to do one day (in my case for Compat), but I
can imagine that if a substitute server existed with all the old
versions of Emacs, then that could be pinged for tests and that would
make testing a lot faster.

I believe that a number of packages on MELPA and GitHub use Nix to the
same effect.

But perhaps the SourceHut CI integration with Guix just requires more
work?  I can't say, because I am neither a SourceHut CI nor a Guix
export :/

> 2. Using [2], it appears to me almost straightforward to build
> beforehand a Docker pack containing all the requirements; say emacs@26,
> curl, gcc-toolchain, git, etc.  And this Docker pack would be built
> using GNU Guix,
>
>     guix pack -f docker -m manifest.scm
>
> where the file manifest.scm lists all the requirements.  Using adequate
> option as --save-provenance, this Docker pack can be inspectable [4] and
> it could be stored to any Docker registery.
>
> Hence, the line,
>
>         image: debian/oldstable
>
> or some images as,
>
>         image: ubuntu/focal
>         repositories:
>           emacs: http://ppa.launchpad.net/kelleyk/emacs/ubuntu focal main 3FF0E01EEAAFC9CD
>
> would be replaced by some images produced by “guix pack -f docker”
> stored to some Docker registery.
>
>
> All in all, I forked the project [1] but the SourceHuts build (CI)
> requires some fee, right?  Well, let me know how we could test this
> approach of using Guix as base for running Org test suite.
>
> (The maintenance of such can be part of the story too. ;-))
>
>
> Last, without putting the cart before the horse, I think this work could
> be a kind of preliminary proof-of-concept for testing Emacs packages
> (ELPA, MELPA, etc.).
>
> Cheers,
> simon
>
> 1: https://git.sr.ht/~bzg/org-mode-tests
> 2: https://git.sr.ht/~pkal/guix-emacs-historical
> 3: https://man.sr.ht/~dhruvin/builds.sr.ht-guix-cookbook/
> 4: https://hpc.guix.info/blog/2021/10/when-docker-images-become-fixed-point/

Re: About org-mode-tests and CI

[Ihor Radchenko]

Simon Tournier <zimon.toutoune@gmail.com> writes:

> Recently, Bastien told me about using GNU Guix for some tests of Org.
> Neat!  Then, Bastien pointed this org-mode-tests [1] effort.
>
> Unrelated, Philip provides Guix recipes [2] for various old Emacs
> versions.
>
> Org and Guix are part of the GNU system.  Therefore, we could imagine to
> bridge instead of relying on Debian. :-)
>
> Moreover, what would be another advantage?  Run the exact same
> computational environment from locally to SourceHut builds.

I am not sure what you mean here.

Also, note that org-mode-tests has /local folder running local podman
builds with reproducible environment. See
https://git.sr.ht/~bzg/org-mode-tests/#usage

> Two directions:
>
> 1. The SourceHut image of Guix [3] could be used but – and maybe I am
> missing a point since I am not an expert about SourceHut CI – the state
> (revision) of this image is not controlled and thus it requires
> something like:
>
>         image: guix
>         tasks:
>           - guix: |
>               guix pull -C project/path/to/channels.scm
>
> Well, I do not know how SourceHut is caching but somehow the .yml
> configuration leads to always the same computational environment
> (image), in which “make test” is run.  Therefore, the CI could spend
> more time in computing again and again this fixed state than running the
> Org test suite. :-)

I do not think that is it dramatically different from other images
(debian/ubuntu). Need to install non-default packages one way or
another.

> 2. Using [2], it appears to me almost straightforward to build
> beforehand a Docker pack containing all the requirements; say emacs@26,
> curl, gcc-toolchain, git, etc.  And this Docker pack would be built
> using GNU Guix,
>
>     guix pack -f docker -m manifest.scm
>
> where the file manifest.scm lists all the requirements.  Using adequate
> option as --save-provenance, this Docker pack can be inspectable [4] and
> it could be stored to any Docker registery.
>
> Hence, the line,
>
>         image: debian/oldstable
>
> or some images as,
>
>         image: ubuntu/focal
>         repositories:
>           emacs: http://ppa.launchpad.net/kelleyk/emacs/ubuntu focal main 3FF0E01EEAAFC9CD
>
> would be replaced by some images produced by “guix pack -f docker”
> stored to some Docker registery.

I guess it could be fine. Though I am not sure how the images are going
to be upgraded then as the required packages (Elisp and system) are
updated to new versions.

> All in all, I forked the project [1] but the SourceHuts build (CI)
> requires some fee, right?  Well, let me know how we could test this
> approach of using Guix as base for running Org test suite.
>
> (The maintenance of such can be part of the story too. ;-))

I think Bastien will contact you about this. He has access to the build
servers.

> Last, without putting the cart before the horse, I think this work could
> be a kind of preliminary proof-of-concept for testing Emacs packages
> (ELPA, MELPA, etc.).

Yep. Or maybe even Emacs itself? I am not sure if Emacs is currently
using CI. Just recall talks about it.

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Re: About org-mode-tests and CI

[Bastien]

Thanks Simon for bringing this up!

Our first need is to have someone volunteering for maintaining our
current test infrastructure - Christian is helping with this right
now but cannot afford to become the maintainer right now.

Philip, is this something you would consider?

If so, that'd be very helpful. Let me know your SourceHut username
and I'll give you access to https://git.sr.ht/~bzg/org-mode-tests/

I am a paid user on SourceHut and I'm happy pulling org-mode-tests
from the orgmode.org server to build from the latest repo manifests.

Once we solve this maintenance issue, we might consider evolving
the tests so that they use Guix instead of Debian if it provides a
real benefit for our needs.

In the meantime, this test infrastructure has already proven very
useful IMHO.

Thanks,

-- 
 Bastien

Re: About org-mode-tests and CI

[Philip Kaludercic]

Bastien <bzg@gnu.org> writes:

> Thanks Simon for bringing this up!
>
> Our first need is to have someone volunteering for maintaining our
> current test infrastructure - Christian is helping with this right
> now but cannot afford to become the maintainer right now.
>
> Philip, is this something you would consider?

I am sorry, I know too little about Org and use it too superficially to
be of much use here.

> If so, that'd be very helpful. Let me know your SourceHut username
> and I'll give you access to https://git.sr.ht/~bzg/org-mode-tests/
>
> I am a paid user on SourceHut and I'm happy pulling org-mode-tests
> from the orgmode.org server to build from the latest repo manifests.
>
> Once we solve this maintenance issue, we might consider evolving
> the tests so that they use Guix instead of Debian if it provides a
> real benefit for our needs.
>
> In the meantime, this test infrastructure has already proven very
> useful IMHO.
>
> Thanks,

Re: About org-mode-tests and CI

[Bastien]

Philip Kaludercic <philipk@posteo.net> writes:

> I am sorry, I know too little about Org and use it too superficially to
> be of much use here.

No problem, and thanks for the super quick feedback!

-- 
 Bastien