Re: A requires/provides approach to linking source code blocks

Re: A requires/provides approach to linking source code blocks

[autofrettage]

Tim wrote:
> This could just be me, but recently, I'm becoming very concerned
> about the growth of additional features and options in org mode.

Count me in. I have been mostly been hanging around in the shadows, but this is serious enough for me to wave a flag on the right side.

I would go as far as saying that several suggestions have been so niche, as to be labeled feature bloats. They can be made available as user-added extensions through melpa, but should stay outside org itself.

just my ¢2

Rasmus

Re: A requires/provides approach to linking source code blocks

[Alexander Adolf]

[-- Attachment #1: Type: text/plain, Size: 426 bytes --]


> On 9. Jul 2021, at 19:01, autofrettage <autofrettage@protonmail.ch> wrote:
> 
> Tim wrote:
>> This could just be me, but recently, I'm becoming very concerned
>> about the growth of additional features and options in org mode.
> 
> Count me in. I have been mostly been hanging around in the shadows, but this is serious enough for me to wave a flag on the right side.
> […]

Same concern here. 

  — Alex

[-- Attachment #2: smime.p7s --]
[-- Type: application/pkcs7-signature, Size: 1944 bytes --]

Re: A requires/provides approach to linking source code blocks

[Tom Gillespie]

We have been receiving many new feature suggestions and requests
coming in for org babel. I think that Tim's suggestion is the right
one. Nearly all of these need to be implemented as an extension first
and tested independently. Further, even if this is done, it should be
clear that there is zero expectation that such extensions will be
incorporated.

Once I wrap up the formal grammar for org, one of the next things I
plan to work on is a clear specification for org babel. This is
critical because so many of the suggestions that come in deal with
individuals' specific problems and thus fail to account for how such
features interact with existing features and how the newly proposed
feature would block some other features in the future, confuse users,
etc. Such suggestions also often fail to account for increased
complexity, nor have they been exposed to a sufficient number of
examples to reveal fundamental ambiguities in how they could be
interpreted. The issues with variable behavior between ob langs for
:pre :post :prologue :epilogue etc. are already enough to keep us busy
for quite some time.

With regard to this thread in particular, it is of some interest, but
there are fundamental issues, including the fact that certain
languages (e.g. racket) expect module code to exist somewhere on the
file system. There are ways around many of these issues, in fact there
are likely many ways around any individual issue, so org babel needs
to systematically consider the issues and provide a clear
specification, or at least a guide for how such cases should be
handled.

To give an example from one of my continual pain points: I start
writing python or racket in an org src block and then I want it to be
a library so that it can be reused by other code both inside and
outside the org file without having to resort to noweb.

What is the best way to handle this? I don't know. Right now I tangle
things and resort to awful hacks for the reuse-in-this-org-file case, but
I'm guessing there is a better generic solution which would allow _any_
org block to be exported as a library instead of nowebbed in.

Before jumping for any particular suggestion for how to handle this
we need to explore the diversity of cases that various ob langs
present, so that we can find a solution that will work for all of
them. After all, packaging code to a library for reuse is an
extremely common pattern that org babel should be able to
abstract, but it is a major undertaking, not just the addition of a
keyword here and there.

In short I suggest that we issue a general moratorium on new org babel
feature suggestions until we can stabilize what we already have and
provide a clear specification for correct behavior. Until we have that spec
we could encourage users to create extensions that implement those
features.

Best,
Tom


PS The other next thing that I am working on might be another way out
for this particular feature request. Namely, it is simplifying and
extending org keyword syntax so that new keywords (with options) and
associated keywords can be specified using keyword syntax within a
single org file. This would make it possible to get useful high level
keyword behavior in a single file without burdening the core
implementation with more special cases for associated keywords, and it
would allow users to write small elisp functions that could do some of
what is suggested here, all without need to add anything to the core
org implementation.

Re: A requires/provides approach to linking source code blocks

[George Mauer]

[-- Attachment #1: Type: text/plain, Size: 4848 bytes --]

In implementing an extension you might consider doing it as a generalized
form of what I did with ob-racket (https://github.com/togakangaroo/ob-racket
).

I think it is best to rely on the import/require/include mechanism of the
language you're using. Pretty much all of them support adjacent files. The
problem with code block execution is that it expands the block and writes
it to a temporary directory in a non-predictable location. If you want to
reference other blocks, you need them to *also* write into files in the
same folder.

What I do in ob-racket is support a header :adjacent-file which allows you
to pass in the name of another block and then writes it into the same
folder as the file to be executed with a predictable name (the name of the
block)

So something like this

  #+begin_src racket :adjacent-file stacker-reader-expander.rkt
    #lang reader "./stacker-reader-expander.rkt"

Will make sure the block named `stacker-reader-expander.rkt` is written
into that same temporary directory with that same name. There's a few rough
edges but for the most part it works well. Would love to see someone take
this and generalize it into its own extension.

On Tue, Jul 13, 2021 at 2:18 PM Tom Gillespie <tgbugs@gmail.com> wrote:

> We have been receiving many new feature suggestions and requests
> coming in for org babel. I think that Tim's suggestion is the right
> one. Nearly all of these need to be implemented as an extension first
> and tested independently. Further, even if this is done, it should be
> clear that there is zero expectation that such extensions will be
> incorporated.
>
> Once I wrap up the formal grammar for org, one of the next things I
> plan to work on is a clear specification for org babel. This is
> critical because so many of the suggestions that come in deal with
> individuals' specific problems and thus fail to account for how such
> features interact with existing features and how the newly proposed
> feature would block some other features in the future, confuse users,
> etc. Such suggestions also often fail to account for increased
> complexity, nor have they been exposed to a sufficient number of
> examples to reveal fundamental ambiguities in how they could be
> interpreted. The issues with variable behavior between ob langs for
> :pre :post :prologue :epilogue etc. are already enough to keep us busy
> for quite some time.
>
> With regard to this thread in particular, it is of some interest, but
> there are fundamental issues, including the fact that certain
> languages (e.g. racket) expect module code to exist somewhere on the
> file system. There are ways around many of these issues, in fact there
> are likely many ways around any individual issue, so org babel needs
> to systematically consider the issues and provide a clear
> specification, or at least a guide for how such cases should be
> handled.
>
> To give an example from one of my continual pain points: I start
> writing python or racket in an org src block and then I want it to be
> a library so that it can be reused by other code both inside and
> outside the org file without having to resort to noweb.
>
> What is the best way to handle this? I don't know. Right now I tangle
> things and resort to awful hacks for the reuse-in-this-org-file case, but
> I'm guessing there is a better generic solution which would allow _any_
> org block to be exported as a library instead of nowebbed in.
>
> Before jumping for any particular suggestion for how to handle this
> we need to explore the diversity of cases that various ob langs
> present, so that we can find a solution that will work for all of
> them. After all, packaging code to a library for reuse is an
> extremely common pattern that org babel should be able to
> abstract, but it is a major undertaking, not just the addition of a
> keyword here and there.
>
> In short I suggest that we issue a general moratorium on new org babel
> feature suggestions until we can stabilize what we already have and
> provide a clear specification for correct behavior. Until we have that spec
> we could encourage users to create extensions that implement those
> features.
>
> Best,
> Tom
>
>
> PS The other next thing that I am working on might be another way out
> for this particular feature request. Namely, it is simplifying and
> extending org keyword syntax so that new keywords (with options) and
> associated keywords can be specified using keyword syntax within a
> single org file. This would make it possible to get useful high level
> keyword behavior in a single file without burdening the core
> implementation with more special cases for associated keywords, and it
> would allow users to write small elisp functions that could do some of
> what is suggested here, all without need to add anything to the core
> org implementation.
>
>

[-- Attachment #2: Type: text/html, Size: 5553 bytes --]

Re: A requires/provides approach to linking source code blocks

[Tim Cross]

Tom Gillespie <tgbugs@gmail.com> writes:

[snip]

>
> Once I wrap up the formal grammar for org, one of the next things I
> plan to work on is a clear specification for org babel. This is
> critical because so many of the suggestions that come in deal with
> individuals' specific problems and thus fail to account for how such
> features interact with existing features and how the newly proposed
> feature would block some other features in the future, confuse users,
> etc. Such suggestions also often fail to account for increased
> complexity, nor have they been exposed to a sufficient number of
> examples to reveal fundamental ambiguities in how they could be
> interpreted. The issues with variable behavior between ob langs for
> :pre :post :prologue :epilogue etc. are already enough to keep us busy
> for quite some time.
>

Yes, that clearly summs up my concerns as well. Often, a 'fix' for some
problem can seem straight-forward, possibly even trivial, when
considered only within one person's use case/workflow. Part of what
makes org so powerful is the level of flexibility it supports. However,
this flexibility makes it extremely difficult to consider, or even know
about, all the existing use cases. 

I suspect that once we have some formal specifications for babel, the
next step will be to develop some good unit tests to verify these
specifications. This would at least highlight/alert developers to
unforeseen impact from changes and alert them to things they may
not have considered. 

> With regard to this thread in particular, it is of some interest, but
> there are fundamental issues, including the fact that certain
> languages (e.g. racket) expect module code to exist somewhere on the
> file system. There are ways around many of these issues, in fact there
> are likely many ways around any individual issue, so org babel needs
> to systematically consider the issues and provide a clear
> specification, or at least a guide for how such cases should be
> handled.
>
> To give an example from one of my continual pain points: I start
> writing python or racket in an org src block and then I want it to be
> a library so that it can be reused by other code both inside and
> outside the org file without having to resort to noweb.
>

This is an interesting point. I think a number of languages have
challenges here. I run into very similar issues with Clojure.

To some extent, I think this is a grey area within the literate
programming paradigm. The original literate programming model was
developed at a time when most languages were compiled rather then
interpreted. You generated source code, compiled it and then ran it.

These days, many 'modern' languages are based around an interpreter and
concepts like 'just in time' compilation. In org mode, things become
even more complex because in addition to generation (tangling) of code,
we also want to have evaluation of code blocks, plus we have added the
concept of 'sessions'.

> What is the best way to handle this? I don't know. Right now I tangle
> things and resort to awful hacks for the reuse-in-this-org-file case, but
> I'm guessing there is a better generic solution which would allow _any_
> org block to be exported as a library instead of nowebbed in.
>
> Before jumping for any particular suggestion for how to handle this
> we need to explore the diversity of cases that various ob langs
> present, so that we can find a solution that will work for all of
> them. After all, packaging code to a library for reuse is an
> extremely common pattern that org babel should be able to
> abstract, but it is a major undertaking, not just the addition of a
> keyword here and there.
>

Agree. I'm not convinced we really understand the requirements here and
more exploration and specification is required. The more we add 'simple'
extensions, options, keywords etc the more likely it is we will make
this a much harder process and will likely result in even larger
'breakage' once we do define a clearer specification.

> In short I suggest that we issue a general moratorium on new org babel
> feature suggestions until we can stabilize what we already have and
> provide a clear specification for correct behavior. Until we have that spec
> we could encourage users to create extensions that implement those
> features.
>

Yes. While it might sound harsh or overly limiting, I do think such a
moratorium may be required. We may be able to lift it once we have some
core specifications in place. We would still accept bug fixes (though we
may need to provide some clarity on what is a bug fix and what is a
feature enhancement/change - I regularly see posts flagged as bug fixes
which are actually feature enhancements or extensions). 
>
> PS The other next thing that I am working on might be another way out
> for this particular feature request. Namely, it is simplifying and
> extending org keyword syntax so that new keywords (with options) and
> associated keywords can be specified using keyword syntax within a
> single org file. This would make it possible to get useful high level
> keyword behavior in a single file without burdening the core
> implementation with more special cases for associated keywords, and it
> would allow users to write small elisp functions that could do some of
> what is suggested here, all without need to add anything to the core
> org implementation.

Sounds interesting. To some extent, recognising there may be at least
two types of code blocks (library code, executable/driver code) and at
least two types of languages (compiled/interpreted) may help identify
base code block permutation and required options, keywords etc. However,
we are unlikely to find a definitive set which supports all use
cases/workflows and the ability to easily extend/customise behaviours
would be very useful. 
-- 
Tim Cross

Re: A requires/provides approach to linking source code blocks

[Samuel Wales]

one thing about org that i think has been making it complex, in
addition to number of features, is non-orthogonality.

On 7/13/21, Tim Cross <theophilusx@gmail.com> wrote:
>
> Tom Gillespie <tgbugs@gmail.com> writes:
>
> [snip]
>
>>
>> Once I wrap up the formal grammar for org, one of the next things I
>> plan to work on is a clear specification for org babel. This is
>> critical because so many of the suggestions that come in deal with
>> individuals' specific problems and thus fail to account for how such
>> features interact with existing features and how the newly proposed
>> feature would block some other features in the future, confuse users,
>> etc. Such suggestions also often fail to account for increased
>> complexity, nor have they been exposed to a sufficient number of
>> examples to reveal fundamental ambiguities in how they could be
>> interpreted. The issues with variable behavior between ob langs for
>> :pre :post :prologue :epilogue etc. are already enough to keep us busy
>> for quite some time.
>>
>
> Yes, that clearly summs up my concerns as well. Often, a 'fix' for some
> problem can seem straight-forward, possibly even trivial, when
> considered only within one person's use case/workflow. Part of what
> makes org so powerful is the level of flexibility it supports. However,
> this flexibility makes it extremely difficult to consider, or even know
> about, all the existing use cases.
>
> I suspect that once we have some formal specifications for babel, the
> next step will be to develop some good unit tests to verify these
> specifications. This would at least highlight/alert developers to
> unforeseen impact from changes and alert them to things they may
> not have considered.
>
>> With regard to this thread in particular, it is of some interest, but
>> there are fundamental issues, including the fact that certain
>> languages (e.g. racket) expect module code to exist somewhere on the
>> file system. There are ways around many of these issues, in fact there
>> are likely many ways around any individual issue, so org babel needs
>> to systematically consider the issues and provide a clear
>> specification, or at least a guide for how such cases should be
>> handled.
>>
>> To give an example from one of my continual pain points: I start
>> writing python or racket in an org src block and then I want it to be
>> a library so that it can be reused by other code both inside and
>> outside the org file without having to resort to noweb.
>>
>
> This is an interesting point. I think a number of languages have
> challenges here. I run into very similar issues with Clojure.
>
> To some extent, I think this is a grey area within the literate
> programming paradigm. The original literate programming model was
> developed at a time when most languages were compiled rather then
> interpreted. You generated source code, compiled it and then ran it.
>
> These days, many 'modern' languages are based around an interpreter and
> concepts like 'just in time' compilation. In org mode, things become
> even more complex because in addition to generation (tangling) of code,
> we also want to have evaluation of code blocks, plus we have added the
> concept of 'sessions'.
>
>> What is the best way to handle this? I don't know. Right now I tangle
>> things and resort to awful hacks for the reuse-in-this-org-file case, but
>> I'm guessing there is a better generic solution which would allow _any_
>> org block to be exported as a library instead of nowebbed in.
>>
>> Before jumping for any particular suggestion for how to handle this
>> we need to explore the diversity of cases that various ob langs
>> present, so that we can find a solution that will work for all of
>> them. After all, packaging code to a library for reuse is an
>> extremely common pattern that org babel should be able to
>> abstract, but it is a major undertaking, not just the addition of a
>> keyword here and there.
>>
>
> Agree. I'm not convinced we really understand the requirements here and
> more exploration and specification is required. The more we add 'simple'
> extensions, options, keywords etc the more likely it is we will make
> this a much harder process and will likely result in even larger
> 'breakage' once we do define a clearer specification.
>
>> In short I suggest that we issue a general moratorium on new org babel
>> feature suggestions until we can stabilize what we already have and
>> provide a clear specification for correct behavior. Until we have that
>> spec
>> we could encourage users to create extensions that implement those
>> features.
>>
>
> Yes. While it might sound harsh or overly limiting, I do think such a
> moratorium may be required. We may be able to lift it once we have some
> core specifications in place. We would still accept bug fixes (though we
> may need to provide some clarity on what is a bug fix and what is a
> feature enhancement/change - I regularly see posts flagged as bug fixes
> which are actually feature enhancements or extensions).
>>
>> PS The other next thing that I am working on might be another way out
>> for this particular feature request. Namely, it is simplifying and
>> extending org keyword syntax so that new keywords (with options) and
>> associated keywords can be specified using keyword syntax within a
>> single org file. This would make it possible to get useful high level
>> keyword behavior in a single file without burdening the core
>> implementation with more special cases for associated keywords, and it
>> would allow users to write small elisp functions that could do some of
>> what is suggested here, all without need to add anything to the core
>> org implementation.
>
> Sounds interesting. To some extent, recognising there may be at least
> two types of code blocks (library code, executable/driver code) and at
> least two types of languages (compiled/interpreted) may help identify
> base code block permutation and required options, keywords etc. However,
> we are unlikely to find a definitive set which supports all use
> cases/workflows and the ability to easily extend/customise behaviours
> would be very useful.
> --
> Tim Cross
>
>


-- 
The Kafka Pandemic

Please learn what misopathy is.
https://thekafkapandemic.blogspot.com/2013/10/why-some-diseases-are-wronged.html

A requires/provides approach to linking source code blocks

[Marko Schuetz-Schmuck]

[-- Attachment #1: Type: text/plain, Size: 843 bytes --]

Dear All,

AFAIU in the current support for literate programming I can establish
sequence between blocks by either tangling the entire file whereby the
blocks are written to the source code file in the sequence in which they
appear in the org-mode file or I can name the blocks and use noweb
linking to explicitly state the precise sequence in which the blocks
appear in the source code file.

I would find it useful to have a more declarative way for specifying
sequence. I imagine e.g. using "#+REQUIRES:" and "#+PROVIDES:" to
capture dependency and then have the exporter compute a sequence
satisfying these. I would think that this makes the maintenance of the
dependencies more convenient.

I'd be interested in feedback on the idea.

Please include my email address in the reply as I am not subscribed to
the list.

Best regards,

Marko

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 223 bytes --]

Re: A requires/provides approach to linking source code blocks

[Tim Cross]

Marko Schuetz-Schmuck <MarkoSchuetz@web.de> writes:

> [[PGP Signed Part:Undecided]]
> Dear All,
>
> AFAIU in the current support for literate programming I can establish
> sequence between blocks by either tangling the entire file whereby the
> blocks are written to the source code file in the sequence in which they
> appear in the org-mode file or I can name the blocks and use noweb
> linking to explicitly state the precise sequence in which the blocks
> appear in the source code file.
>
> I would find it useful to have a more declarative way for specifying
> sequence. I imagine e.g. using "#+REQUIRES:" and "#+PROVIDES:" to
> capture dependency and then have the exporter compute a sequence
> satisfying these. I would think that this makes the maintenance of the
> dependencies more convenient.
>
> I'd be interested in feedback on the idea.
>
> Please include my email address in the reply as I am not subscribed to
> the list.
>

My concern here is with the additional complexity. This is already a
somewhat complex aspect of org mode and the behaviour you describe can
effectively be done using noweb, although as you say, not as
declarative in style.

I'm not sure the added complexity is worth the additional maintenance
overhead and I would be concerned regarding how much confusion this
could cause i.e. someone uses both noweb and declarative
provide/require. These additions are also likely to hamper any efforts
to improve performance - a topic which comes up on a fairly regular
basis. 

This could just be me, but recently, I'm becoming very concerned about
the growth of additional features and options in org mode. Many of the
new things being requested are for relatively narrow use cases and
represent functionality which can largely be achieved with existing
features. At the same time, there also seems to be a growing
frequency in patches which introduce regression bugs and patches to fix
patches.

This is all beginning to feel like we are running very close to the
tipping point at which time we will have something that is so complex
that only a very few people are able to maintain the code base and keep
the system stable. New maintainers are discouraged because of the code
complexity. We could end up in a position where really important issues
cannot be addressed or addressed in a timely manner because of the
overall complexity of the code base and time it takes to fix and test
and dependence on a very few number of maintainers who are already
swamped with work. 

At some point, I think we may want to consider a temporary freeze on new
functionality and spend a few months just focusing on bug fixes and code
base improvements or re-factoring. It might also be worthwhile providing
some guidelines or criteria/procedures for assessing proposed new
features to avoid a perception of new features being accepted/rejected
based on personality, loudness of voice or some other real or perceived
and irrelevant basis.


-- 
Tim Cross

Re: A requires/provides approach to linking source code blocks

[Maxim Nikulin]

On 09/07/2021 02:32, Tim Cross wrote:
> Marko Schuetz-Schmuck writes:
> 
>> I would find it useful to have a more declarative way for specifying
>> sequence. I imagine e.g. using "#+REQUIRES:" and "#+PROVIDES:" to
>> capture dependency and then have the exporter compute a sequence
>> satisfying these. I would think that this makes the maintenance of the
>> dependencies more convenient.
> 
> My concern here is with the additional complexity. This is already a
> somewhat complex aspect of org mode and the behaviour you describe can
> effectively be done using noweb, although as you say, not as
> declarative in style.

There was a feature request for dependencies between code blocks to run 
them in proper order a half of a year ago.

I think, it is better to try such feature as an extension of org, as a 
separate package. I suspect that some non-obvious issue may arise. 
Likely, to be convenient, it will be desired to autofill 
requires/provides attributes using some tools related to particular 
language.

The question is whether org code is organized in such way, so extensions 
of this kind are possible without code duplication. Maybe API could be 
adjusted. On the other hand, even maintaining of stable semi-internal 
API sometimes is significant burden.

P.S. I am reading the mail list through NNTP news.gmain.io. I have 
noticed that personal copies sometimes arrives without mail list 
address. Sorry if you will have to manually add it to reply.

Re: A requires/provides approach to linking source code blocks

[Berry, Charles via General discussions about Org-mode.]

Tim,

> On Jul 8, 2021, at 12:32 PM, Tim Cross <theophilusx@gmail.com> wrote:
> 
> My concern here is with the additional complexity. This is already a
> somewhat complex aspect of org mode and the behaviour you describe can
> effectively be done using noweb, although as you say, not as
> declarative in style.


This (and the rest of what you said) is very well taken.

I share your concern that adding features to an already rich feature suite will make babel very difficult to penetrate.

Best,
Chuck

p.s. Extracting code from src blocks need not depend on `org-babel-tangle' and friends. Custom exporters have to potential to render code using attribute keys and other export features.  And such exporters have the virtue of not adding complexity to the org code base.

Re: A requires/provides approach to linking source code blocks

[Stefan Nobis]

Marko Schuetz-Schmuck <MarkoSchuetz@web.de> writes:

> I would find it useful to have a more declarative way for specifying
> sequence. I imagine e.g. using "#+REQUIRES:" and "#+PROVIDES:" to
> capture dependency and then have the exporter compute a sequence
> satisfying these.

I would say that declaring an explicit ordering of blocks via noweb is
quite declarative.

The tensions seems more to be between implicit and explicit ordering.
I'm quite a fan of letting the computer compute things, but in the
context of literate programming I tend to prefer explicit over
implicit. IMHO that's the main point of literate programming: Be more
explicit and document all the details. Try not to hide important
aspects.

If you have so many blocks that maintaining the order of the
concatenation seems like a burden and tiresome I would argue that you
are doing it wrong. Because in the end you need quite a good
understanding of the final order of the blocks or else maintaining the
implicit order via require/provide will also get out of hand.

I also second Tims concerns about the additional complexity and
feature creep.

-- 
Until the next mail...,
Stefan.