A requires/provides approach to linking source code blocks

A requires/provides approach to linking source code blocks

[Marko Schuetz-Schmuck]

[-- Attachment #1: Type: text/plain, Size: 843 bytes --]

Dear All,

AFAIU in the current support for literate programming I can establish
sequence between blocks by either tangling the entire file whereby the
blocks are written to the source code file in the sequence in which they
appear in the org-mode file or I can name the blocks and use noweb
linking to explicitly state the precise sequence in which the blocks
appear in the source code file.

I would find it useful to have a more declarative way for specifying
sequence. I imagine e.g. using "#+REQUIRES:" and "#+PROVIDES:" to
capture dependency and then have the exporter compute a sequence
satisfying these. I would think that this makes the maintenance of the
dependencies more convenient.

I'd be interested in feedback on the idea.

Please include my email address in the reply as I am not subscribed to
the list.

Best regards,

Marko

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 223 bytes --]

Re: A requires/provides approach to linking source code blocks

[Tim Cross]

Marko Schuetz-Schmuck <MarkoSchuetz@web.de> writes:

> [[PGP Signed Part:Undecided]]
> Dear All,
>
> AFAIU in the current support for literate programming I can establish
> sequence between blocks by either tangling the entire file whereby the
> blocks are written to the source code file in the sequence in which they
> appear in the org-mode file or I can name the blocks and use noweb
> linking to explicitly state the precise sequence in which the blocks
> appear in the source code file.
>
> I would find it useful to have a more declarative way for specifying
> sequence. I imagine e.g. using "#+REQUIRES:" and "#+PROVIDES:" to
> capture dependency and then have the exporter compute a sequence
> satisfying these. I would think that this makes the maintenance of the
> dependencies more convenient.
>
> I'd be interested in feedback on the idea.
>
> Please include my email address in the reply as I am not subscribed to
> the list.
>

My concern here is with the additional complexity. This is already a
somewhat complex aspect of org mode and the behaviour you describe can
effectively be done using noweb, although as you say, not as
declarative in style.

I'm not sure the added complexity is worth the additional maintenance
overhead and I would be concerned regarding how much confusion this
could cause i.e. someone uses both noweb and declarative
provide/require. These additions are also likely to hamper any efforts
to improve performance - a topic which comes up on a fairly regular
basis. 

This could just be me, but recently, I'm becoming very concerned about
the growth of additional features and options in org mode. Many of the
new things being requested are for relatively narrow use cases and
represent functionality which can largely be achieved with existing
features. At the same time, there also seems to be a growing
frequency in patches which introduce regression bugs and patches to fix
patches.

This is all beginning to feel like we are running very close to the
tipping point at which time we will have something that is so complex
that only a very few people are able to maintain the code base and keep
the system stable. New maintainers are discouraged because of the code
complexity. We could end up in a position where really important issues
cannot be addressed or addressed in a timely manner because of the
overall complexity of the code base and time it takes to fix and test
and dependence on a very few number of maintainers who are already
swamped with work. 

At some point, I think we may want to consider a temporary freeze on new
functionality and spend a few months just focusing on bug fixes and code
base improvements or re-factoring. It might also be worthwhile providing
some guidelines or criteria/procedures for assessing proposed new
features to avoid a perception of new features being accepted/rejected
based on personality, loudness of voice or some other real or perceived
and irrelevant basis.


-- 
Tim Cross

Re: A requires/provides approach to linking source code blocks

[Stefan Nobis]

Marko Schuetz-Schmuck <MarkoSchuetz@web.de> writes:

> I would find it useful to have a more declarative way for specifying
> sequence. I imagine e.g. using "#+REQUIRES:" and "#+PROVIDES:" to
> capture dependency and then have the exporter compute a sequence
> satisfying these.

I would say that declaring an explicit ordering of blocks via noweb is
quite declarative.

The tensions seems more to be between implicit and explicit ordering.
I'm quite a fan of letting the computer compute things, but in the
context of literate programming I tend to prefer explicit over
implicit. IMHO that's the main point of literate programming: Be more
explicit and document all the details. Try not to hide important
aspects.

If you have so many blocks that maintaining the order of the
concatenation seems like a burden and tiresome I would argue that you
are doing it wrong. Because in the end you need quite a good
understanding of the final order of the blocks or else maintaining the
implicit order via require/provide will also get out of hand.

I also second Tims concerns about the additional complexity and
feature creep.

-- 
Until the next mail...,
Stefan.

Re: Slowing down new features

[Timothy]

[-- Attachment #1: Type: text/plain, Size: 2302 bytes --]

Hi Tim,

> This is all beginning to feel like we are running very close to the
> tipping point at which time we will have something that is so complex
> that only a very few people are able to maintain the code base and keep
> the system stable. New maintainers are discouraged because of the code
> complexity. We could end up in a position where really important issues
> cannot be addressed or addressed in a timely manner because of the
> overall complexity of the code base and time it takes to fix and test
> and dependence on a very few number of maintainers who are already
> swamped with work.
>
> At some point, I think we may want to consider a temporary freeze on new
> functionality and spend a few months just focusing on bug fixes and code
> base improvements or re-factoring. It might also be worthwhile providing
> some guidelines or criteria/procedures for assessing proposed new
> features to avoid a perception of new features being accepted/rejected
> based on personality, loudness of voice or some other real or perceived
> and irrelevant basis.

I’ve got some changes that improve existing features that I know I’d like to
push (and would be happy to maintain), but I get the same feeling as you that
overall that we’ve been working on a lot of new things and it would be worth
taking some time to refine what’s here instead of adding more.

That said, I’m not sure a hard freeze would be the best way to approach this,
I’d be more of a fan of just a shift in focus and maybe deferring some ideas to
be considered later.

――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――

FWIW, these are the new things I’d like to push:
• ob-julia
• inline src block fortification
• smart generated export preamble
• a new LaTeX src block syntax highlighting option (this actually isn’t that
  much code)

These are all currently maturing in my config, and in particular the “smart
generated export preamble” and new LaTeX syntax highlighting option have been
massaged into a state I’m very happy with over a number of months.

All the best,
*Timothy*

Re: A requires/provides approach to linking source code blocks

[Maxim Nikulin]

On 09/07/2021 02:32, Tim Cross wrote:
> Marko Schuetz-Schmuck writes:
> 
>> I would find it useful to have a more declarative way for specifying
>> sequence. I imagine e.g. using "#+REQUIRES:" and "#+PROVIDES:" to
>> capture dependency and then have the exporter compute a sequence
>> satisfying these. I would think that this makes the maintenance of the
>> dependencies more convenient.
> 
> My concern here is with the additional complexity. This is already a
> somewhat complex aspect of org mode and the behaviour you describe can
> effectively be done using noweb, although as you say, not as
> declarative in style.

There was a feature request for dependencies between code blocks to run 
them in proper order a half of a year ago.

I think, it is better to try such feature as an extension of org, as a 
separate package. I suspect that some non-obvious issue may arise. 
Likely, to be convenient, it will be desired to autofill 
requires/provides attributes using some tools related to particular 
language.

The question is whether org code is organized in such way, so extensions 
of this kind are possible without code duplication. Maybe API could be 
adjusted. On the other hand, even maintaining of stable semi-internal 
API sometimes is significant burden.

P.S. I am reading the mail list through NNTP news.gmain.io. I have 
noticed that personal copies sometimes arrives without mail list 
address. Sorry if you will have to manually add it to reply.

Re: A requires/provides approach to linking source code blocks

[Berry, Charles via General discussions about Org-mode.]

Tim,

> On Jul 8, 2021, at 12:32 PM, Tim Cross <theophilusx@gmail.com> wrote:
> 
> My concern here is with the additional complexity. This is already a
> somewhat complex aspect of org mode and the behaviour you describe can
> effectively be done using noweb, although as you say, not as
> declarative in style.


This (and the rest of what you said) is very well taken.

I share your concern that adding features to an already rich feature suite will make babel very difficult to penetrate.

Best,
Chuck

p.s. Extracting code from src blocks need not depend on `org-babel-tangle' and friends. Custom exporters have to potential to render code using attribute keys and other export features.  And such exporters have the virtue of not adding complexity to the org code base.