Multiple named code blocks

Multiple named code blocks

[mooss]

[-- Attachment #1: Type: text/plain, Size: 2967 bytes --]

Hi,

I have been using org-mode for almost three years and I loved it so much that I started working on a literate programming tool based on it.
One particular technique that I use is having multiple named code blocks, like so:

#+begin_src perl :noweb yes :results output
<<Before foo>>
sub foo {
<<In foo>>
}
foo;
#+end_src

#+name: Before foo
#+begin_src perl
print "Before foo definition.\n";
#+end_src

#+name: Before foo
#+begin_src perl
my $value = 'Inside foo call';
#+end_src

#+name: In foo
#+begin_src perl
print $value . "\n";
#+end_src

This technique worked without issue until I recently updated Emacs and org-mode with it.
I do not know the version of org-mode I was using before, but this was with Emacs 26.3 and I upgraded to Emacs 27.1 with org-mode 9.4 according to the information at the top of elpa/org-plus-contrib-20201116/org.el.

Before the update, the code block after expansion (obtained with org-babel-expand-src-block via the C-c C-v C-v shortcut) looked like this:
#+begin_src perl
print "Before foo definition.\n";
my $value = 'Inside foo call';
sub foo {
print $value . "\n";
}
foo;
#+end_src

Now the definition of $value is gone:
#+begin_src perl
print "Before foo definition.\n";
sub foo {
print $value . "\n";
}
foo;
#+end_src

I have looked at the info manual so I realise that according to "15.2 Structure of Code Blocks": "For duplicate names, Org mode’s behavior is undefined" so it follows that:
- Up until now, I was incorrectly assuming that duplicated named code blocks were supposed to result in them being concatenated in the noweb expansion phase.
- This is not a bug report, org-mode is working as documented.

I find this technique pretty useful for two reasons:
1. Importing packages right when they are needed.
2. Declaring variables in a broader scope than the one where they are first used.
Here is an short example of this kind of situation:

#+begin_src perl :noweb no
# Expansion of <<Variable declarations>>:
my $even_counter = 0;
my @array = (4, 8, 15, 16, 23, 42);
# A
# lot
# of
# other
# code
# [...]
foreach my $n (@array) {
# Expansion of <<Array processing>>:
$even_counter++ if $n % 2 == 0;
}
print "$even_counter";
#+end_src

In this example, $even_counter could not have been declared on the spot.
Of course, this example is too basic to really paint the usefulness of this technique but an actual example would be too long, the goal here is just to explain the general idea.

With all that being said I would suggest to define the behaviour for multiple named code blocks as resulting in a concatenation of the code blocks, in the order of their apparition.
If you agree about defining this behaviour but think adapting the implementation is of low priority, I could try to implement it myself though I have little experience in emacs-lisp development beyond basic configuration and no experience whatsoever in contributing to FOSS, but I'm willing to start in both domains.

Best regards,
Félix

[-- Attachment #2: Type: text/html, Size: 4673 bytes --]

Re: Multiple named code blocks

[Tom Gillespie]

Hi Félix,
   I think that it is probably not a good idea to implicitly
concatenate blocks that share the same name. There are a number of
major downsides. One reason is that all the other parts of org-mode
assume that there is only a single block with that name, or rather
have undefined behavior if there is more than one, so all the other
reference resolving infrastructure will not work as expected and it
will be hard to navigate to additional blocks. The concatenation
behavior you describe does work if you specify the same file for
tangling, however that uses completely different code. Further, if we
were to define behavior for what should happen if there are multiple
blocks or sections with the same name it would probably either be to
signal an error (if you are in the more immutable camp), that the
first named block is the "canonical" block (current implementation),
or that the last defined block is the "canonical" block (if you want
org to be more like a programming language). The current
implementation follows the first named block, but the reason why the
manual states that it is undefined is because even that behavior
should not be relied upon (thus why an error is probably the most
friendly thing to do). Consider also that if you have different blocks
by the same name in different sections and you reorder the sections
the order in which they are nowebbed in will change. Given that that
behavior can be actively dangerous (imagine a block with cd
some-folder followed by a block rm contents/ -r getting reordered),
there are major downsides to trying to guess how to concatenate
multiple blocks and trying to specify restrictions on where and how
using the same name is allowed or can be safely used is not something
that anyone would want to try to do. Best!
Tom

Re: Multiple named code blocks

[mooss]

Hi Tom,

The downsides you mention are making perfect sense, especially
the cd and rm examples, I did not thought about that and I agree
that this would be a dangerous default.

Another possibility would be to shield this behaviour behind a
header argument, for example by adding a "dimension" to the
"noweb" argument.
What I mean by "dimension" is that to my knowledge, the header-
arg "noweb" only has a, let's say, "when" dimension that can take
one value among "yes" "no" and "no-export", whereas "results" can
have several dimensions, hence the validity of ":results drawer
replace" as a header-arg.
This new dimension could be called the "duplicated" dimension and
take values among "contatenate", "first" or "last".
With this scheme, one could use the header-arg ":noweb yes
concatenate" to always expand noweb inclusions and handle
multiple code blocks via concatenation.

I can see a couple of potential problems with this approach:
 1. It might be better to avoid adding header-args left and right
    to keep complexity in check and this feature is arguably
    quite niche.
 2. This would solidify the idea that multiple code blocks
    sharing the same name is good practice in org documents,
    which it is not, as you explained.

Yet another option would be to not rely on the names but on
another header-arg like "addto" that instruct the noweb resolver
to concatenate the current block to another named code block.
Reusing the previous example,

#+name: Before foo
#+begin_src perl
my $value = 'Inside foo call';
#+end_src

Would become:

#+begin_src perl :addto "Before foo"
my $value = 'Inside foo call';
#+end_src

Yet another approach would be to add a "directive" (that's probably not the right terminology), like so:

#+addto: Before foo
#+begin_src perl
my $value = 'Inside foo call';
#+end_src

The last two approaches would solve problem 2) but not problem
1).
There would also be the problem of what to do when a code block
is "added to" before its definition, what should be done ?
Throwing an error, appending or prepending ?

Personally I would prefer the last approach, even though the
prepending problem remains to be solved, because the other
approaches are polluting the header arguments and look out of
place there.
In any case, I think that a mechanism to concatenate to an
existing code block is a valuable feature for a literate
programming system.

On another note, that is a bit embarrassing but I'm not too sure
about how I am supposed to respond to this email list, I just
clicked on "Reply all" in my webmail client and this results in a
mail specifically addressed to you Tom and a CC to the list.
That is also what is suggested by the "Reply instructions:"
section of the list, but I just want to be sure that I am not
disrespecting a rule or custom that escapes me.

Best regards,
Félix

Re: Multiple named code blocks

[Greg Minshall]

Félix,

i ran into this restriction a while ago.  on this list i was helped, and
ended up using the suggestion to instead put my common bits in a
property in the subtree for a given "name"

***** aggregate.R
       :PROPERTIES:
       :header-args+: :tangle build/package/covid.19.data/R/aggregate.R
       :header-args+: :noweb-ref aggregates
       :END:


then, the source blocks themselves have no name and are very plain

#+begin_src R


i don't know if that will be of any help.

cheers, Greg

Re: Multiple named code blocks

[mooss]

Hi Greg,

I don't know why I missed this, it is exactly what I needed.
Thank you for your help.

Best regards,
Félix

Re: Multiple named code blocks

[Diego Zamboni]

[-- Attachment #1: Type: text/plain, Size: 3586 bytes --]

Hi Felix,

You need to use the noweb-ref header argument instead of #+NAME, then the
block are all concatenated on output.

Best,
--Diego


On Sat, 28 Nov 2020 at 23:35, mooss <mooss@protonmail.com> wrote:

> Hi,
>
> I have been using org-mode for almost three years and I loved it so much
> that I started working on a literate programming tool based on it.
> One particular technique that I use is having multiple named code blocks,
> like so:
>
> #+begin_src perl :noweb yes :results output
> <<Before foo>>
> sub foo {
>     <<In foo>>
> }
> foo;
> #+end_src
>
> #+name: Before foo
> #+begin_src perl
> print "Before foo definition.\n";
> #+end_src
>
> #+name: Before foo
> #+begin_src perl
> my $value = 'Inside foo call';
> #+end_src
>
> #+name: In foo
> #+begin_src perl
> print $value . "\n";
> #+end_src
>
> This technique worked without issue until I recently updated Emacs and
> org-mode with it.
> I do not know the version of org-mode I was using before, but this was
> with Emacs 26.3 and I upgraded to Emacs 27.1 with org-mode 9.4 according to
> the information at the top of elpa/org-plus-contrib-20201116/org.el.
>
> Before the update, the code block after expansion (obtained with
> org-babel-expand-src-block via the C-c C-v C-v shortcut) looked like this:
> #+begin_src perl
> print "Before foo definition.\n";
> my $value = 'Inside foo call';
> sub foo {
>     print $value . "\n";
> }
> foo;
> #+end_src
>
> Now the definition of $value is gone:
> #+begin_src perl
> print "Before foo definition.\n";
> sub foo {
>     print $value . "\n";
> }
> foo;
> #+end_src
>
> I have looked at the info manual so I realise that according to "15.2
> Structure of Code Blocks": "For duplicate names, Org mode’s behavior is
> undefined" so it follows that:
> - Up until now, I was incorrectly assuming that duplicated named code
> blocks were supposed to result in them being concatenated in the noweb
> expansion phase.
> - This is not a bug report, org-mode is working as documented.
>
> I find this technique pretty useful for two reasons:
> 1. Importing packages right when they are needed.
> 2. Declaring variables in a broader scope than the one where they are
> first used.
>     Here is an short example of this kind of situation:
>
>     #+begin_src perl :noweb no
>     # Expansion of <<Variable declarations>>:
>     my $even_counter = 0;
>     my @array = (4, 8, 15, 16, 23, 42);
>     # A
>     # lot
>     # of
>     # other
>     # code
>     # [...]
>     foreach my $n (@array) {
>         # Expansion of <<Array processing>>:
>         $even_counter++ if $n % 2 == 0;
>     }
>     print "$even_counter";
>     #+end_src
>
>     In this example, $even_counter could not have been declared on the
> spot.
>     Of course, this example is too basic to really paint the usefulness of
> this technique but an actual example would be too long, the goal here is
> just to explain the general idea.
>
> With all that being said I would suggest to define the behaviour for
> multiple named code blocks as resulting in a concatenation of the code
> blocks, in the order of their apparition.
> If you agree about defining this behaviour but think adapting the
> implementation is of low priority, I could try to implement it myself
> though I have little experience in emacs-lisp development beyond basic
> configuration and no experience whatsoever in contributing to FOSS, but I'm
> willing to start in both domains.
>
> Best regards,
> Félix
>
>

[-- Attachment #2: Type: text/html, Size: 5107 bytes --]

Re: Multiple named code blocks

[mooss]

Hi Diego,

This feature does indeed what I need, I will update my code with it.
Thank you for your help.

Best regards,
Félix