Re: A requires/provides approach to linking source code blocks

[Tim Cross <theophilusx@gmail.com>]

Tom Gillespie <tgbugs@gmail.com> writes:

[snip]

>
> Once I wrap up the formal grammar for org, one of the next things I
> plan to work on is a clear specification for org babel. This is
> critical because so many of the suggestions that come in deal with
> individuals' specific problems and thus fail to account for how such
> features interact with existing features and how the newly proposed
> feature would block some other features in the future, confuse users,
> etc. Such suggestions also often fail to account for increased
> complexity, nor have they been exposed to a sufficient number of
> examples to reveal fundamental ambiguities in how they could be
> interpreted. The issues with variable behavior between ob langs for
> :pre :post :prologue :epilogue etc. are already enough to keep us busy
> for quite some time.
>

Yes, that clearly summs up my concerns as well. Often, a 'fix' for some
problem can seem straight-forward, possibly even trivial, when
considered only within one person's use case/workflow. Part of what
makes org so powerful is the level of flexibility it supports. However,
this flexibility makes it extremely difficult to consider, or even know
about, all the existing use cases. 

I suspect that once we have some formal specifications for babel, the
next step will be to develop some good unit tests to verify these
specifications. This would at least highlight/alert developers to
unforeseen impact from changes and alert them to things they may
not have considered. 

> With regard to this thread in particular, it is of some interest, but
> there are fundamental issues, including the fact that certain
> languages (e.g. racket) expect module code to exist somewhere on the
> file system. There are ways around many of these issues, in fact there
> are likely many ways around any individual issue, so org babel needs
> to systematically consider the issues and provide a clear
> specification, or at least a guide for how such cases should be
> handled.
>
> To give an example from one of my continual pain points: I start
> writing python or racket in an org src block and then I want it to be
> a library so that it can be reused by other code both inside and
> outside the org file without having to resort to noweb.
>

This is an interesting point. I think a number of languages have
challenges here. I run into very similar issues with Clojure.

To some extent, I think this is a grey area within the literate
programming paradigm. The original literate programming model was
developed at a time when most languages were compiled rather then
interpreted. You generated source code, compiled it and then ran it.

These days, many 'modern' languages are based around an interpreter and
concepts like 'just in time' compilation. In org mode, things become
even more complex because in addition to generation (tangling) of code,
we also want to have evaluation of code blocks, plus we have added the
concept of 'sessions'.

> What is the best way to handle this? I don't know. Right now I tangle
> things and resort to awful hacks for the reuse-in-this-org-file case, but
> I'm guessing there is a better generic solution which would allow _any_
> org block to be exported as a library instead of nowebbed in.
>
> Before jumping for any particular suggestion for how to handle this
> we need to explore the diversity of cases that various ob langs
> present, so that we can find a solution that will work for all of
> them. After all, packaging code to a library for reuse is an
> extremely common pattern that org babel should be able to
> abstract, but it is a major undertaking, not just the addition of a
> keyword here and there.
>

Agree. I'm not convinced we really understand the requirements here and
more exploration and specification is required. The more we add 'simple'
extensions, options, keywords etc the more likely it is we will make
this a much harder process and will likely result in even larger
'breakage' once we do define a clearer specification.

> In short I suggest that we issue a general moratorium on new org babel
> feature suggestions until we can stabilize what we already have and
> provide a clear specification for correct behavior. Until we have that spec
> we could encourage users to create extensions that implement those
> features.
>

Yes. While it might sound harsh or overly limiting, I do think such a
moratorium may be required. We may be able to lift it once we have some
core specifications in place. We would still accept bug fixes (though we
may need to provide some clarity on what is a bug fix and what is a
feature enhancement/change - I regularly see posts flagged as bug fixes
which are actually feature enhancements or extensions). 
>
> PS The other next thing that I am working on might be another way out
> for this particular feature request. Namely, it is simplifying and
> extending org keyword syntax so that new keywords (with options) and
> associated keywords can be specified using keyword syntax within a
> single org file. This would make it possible to get useful high level
> keyword behavior in a single file without burdening the core
> implementation with more special cases for associated keywords, and it
> would allow users to write small elisp functions that could do some of
> what is suggested here, all without need to add anything to the core
> org implementation.

Sounds interesting. To some extent, recognising there may be at least
two types of code blocks (library code, executable/driver code) and at
least two types of languages (compiled/interpreted) may help identify
base code block permutation and required options, keywords etc. However,
we are unlikely to find a definitive set which supports all use
cases/workflows and the ability to easily extend/customise behaviours
would be very useful. 
-- 
Tim Cross