[BUG] Inconsistency in src block hiding

[BUG] Inconsistency in src block hiding

[Nicolas Goaziou]

Hello,

In the following example (latest Org), with point at "|", when TAB is
pressed, block gets hidden at the "name" level.

--8<---------------cut here---------------start------------->8---
|#+name: test
#+begin_src emacs-lisp
"test"
#+end_src
--8<---------------cut here---------------end--------------->8---

It is because `org-babel-result-regexp' is matched in
`org-babel-hide-result-toggle-maybe'. But, should it be the case?

In particular, if I look at `org-babel-data-names', upon which
`org-babel-result-regexp' is built, I'm not sure that "name" keyword
should define a result line. As such, I would guess that it should be
removed from that list.


Regards,

-- 
Nicolas Goaziou

Re: [BUG] Inconsistency in src block hiding

[Eric Schulte]

Nicolas Goaziou <n.goaziou@gmail.com> writes:

> Hello,
>
> In the following example (latest Org), with point at "|", when TAB is
> pressed, block gets hidden at the "name" level.
>
> |#+name: test
> #+begin_src emacs-lisp
> "test"
> #+end_src
>
> It is because `org-babel-result-regexp' is matched in
> `org-babel-hide-result-toggle-maybe'. But, should it be the case?
>
> In particular, if I look at `org-babel-data-names', upon which
> `org-babel-result-regexp' is built, I'm not sure that "name" keyword
> should define a result line. As such, I would guess that it should be
> removed from that list.
>

Hi Nicolas,

"name" is and should be an element of the `org-babel-data-names' list as
it is the preferred way to name data in an Org-mode file, e.g.,

#+name: foo
- 1
- 2
- 3

The only reason that "tblname" and "results" are included in the list
are because "tblname" is used by other parts of Org-mode, and "results"
was retained because it was felt that a "name" line without an actual
name (e.g., as the results of an un-named code block) would look funny.

I don't view the "results style" hiding as inconsistent, however if
others do, it wouldn't be difficult to insert a check for a special case
in the code that hides results to inhibit the behavior if the data
following "name" happens to be a code block.

Best -- Eric

>
>
> Regards,

-- 
Eric Schulte
http://cs.unm.edu/~eschulte/

Re: [BUG] Inconsistency in src block hiding

[Nicolas Goaziou]

Eric Schulte <schulte.eric@gmail.com> writes:

> "name" is and should be an element of the `org-babel-data-names' list as
> it is the preferred way to name data in an Org-mode file, e.g.,
>
> #+name: foo
> - 1
> - 2
> - 3

I agree.

> The only reason that "tblname" and "results" are included in the list
> are because "tblname" is used by other parts of Org-mode, and "results"
> was retained because it was felt that a "name" line without an actual
> name (e.g., as the results of an un-named code block) would look funny.
>
> I don't view the "results style" hiding as inconsistent

It is inconsistent when keywords stack on top of each other. If you have
only a "#+name:" keyword, block with fold at the "#+name:" level. If you
have both "#+name:" and, for example "#+results" below, block will fold
at the "#+result:" level and TAB will not respond at "#+name:". If you
have, from top to bottom, "name", "results" "header", nothing will fold.
In all those cases, I think a consistent behaviour could be to hide the
block, with any number of keywords above, and TAB pressed at any of
them.

I'm not sure that "#+results:" or "#+name:" keywords should allow to
hide whole parts of the buffer. I realize that toggling "#+results:"
visibility has been in core for a while. But now that every Org blob can
have a "#+name" attached to it, one can hide almost anything in the
buffer.

Until now we could hide contents with headlines, blocks, drawers, and
items (with an option to prevent it). And we had a global mechanism to
handle visibility toggling just fine (namely S-TAB with different
numbers of prefixes). But hiding independently each list, table or
paragraph with no possibility to make them appear in groups just doesn't
sound right.

Hence, I suggest to think again about it: we can live happily with
outlines, blocks and drawers as folding entities.

Moreover, there is another problem related to that.

`org-export-blocks-preprocess' will remove all "#+name:" keywords in the
buffer.  It mustn't: again "#+name:" is a general naming mechanism for
almost any Org syntax. It may/will be also used for other purpose than
Babel. That information is important even after blocks have been
processed.


Regards,

-- 
Nicolas Goaziou

Re: [BUG] Inconsistency in src block hiding

[Eric Schulte]

Nicolas Goaziou <n.goaziou@gmail.com> writes:

> Eric Schulte <schulte.eric@gmail.com> writes:
>
>> "name" is and should be an element of the `org-babel-data-names' list as
>> it is the preferred way to name data in an Org-mode file, e.g.,
>>
>> #+name: foo
>> - 1
>> - 2
>> - 3
>
> I agree.
>
>> The only reason that "tblname" and "results" are included in the list
>> are because "tblname" is used by other parts of Org-mode, and "results"
>> was retained because it was felt that a "name" line without an actual
>> name (e.g., as the results of an un-named code block) would look funny.
>>
>> I don't view the "results style" hiding as inconsistent
>
> It is inconsistent when keywords stack on top of each other. If you have
> only a "#+name:" keyword, block with fold at the "#+name:" level. If you
> have both "#+name:" and, for example "#+results" below, block will fold
> at the "#+result:" level and TAB will not respond at "#+name:".

Can you provide an example, I'm thinking that what you describe may not
be legal syntax.  For example

#+name: one
#+results: two
- 1
- 2

doesn't make any more sense than

#+name: one
#+name: two
- 1
- 2

> If you have, from top to bottom, "name", "results" "header", nothing
> will fold.  In all those cases, I think a consistent behaviour could
> be to hide the block, with any number of keywords above, and TAB
> pressed at any of them.
>

Yes, I would agree, the hiding should be smart enough to find the whole
unit and hide it.  I'll take a look at the code.

>
> I'm not sure that "#+results:" or "#+name:" keywords should allow to
> hide whole parts of the buffer. I realize that toggling "#+results:"
> visibility has been in core for a while. But now that every Org blob can
> have a "#+name" attached to it, one can hide almost anything in the
> buffer.
>
> Until now we could hide contents with headlines, blocks, drawers, and
> items (with an option to prevent it). And we had a global mechanism to
> handle visibility toggling just fine (namely S-TAB with different
> numbers of prefixes). But hiding independently each list, table or
> paragraph with no possibility to make them appear in groups just doesn't
> sound right.
>
> Hence, I suggest to think again about it: we can live happily with
> outlines, blocks and drawers as folding entities.
>

The hiding was added because code blocks occasionally generate *huge*
results which make it impossible to read further down in the buffer.
Hiding such large results can be very useful when you want to keep the
data in-buffer, but still want to be able to read down the page.

Is there a way to bring the hiding of results more in-line with the
other methods of hiding in Org-mode?  Should S-Tab be made to toggle the
hidden states of named entities as well as outline levels?

>
> Moreover, there is another problem related to that.
>
> `org-export-blocks-preprocess' will remove all "#+name:" keywords in the
> buffer.  It mustn't: again "#+name:" is a general naming mechanism for
> almost any Org syntax. It may/will be also used for other purpose than
> Babel. That information is important even after blocks have been
> processed.
>

I'll take a look at this and submit a patch.

Best -- Eric

>
>
> Regards,

-- 
Eric Schulte
http://cs.unm.edu/~eschulte/

Re: [BUG] Inconsistency in src block hiding

[Eric Schulte]

>> If you have, from top to bottom, "name", "results" "header", nothing
>> will fold.  In all those cases, I think a consistent behaviour could
>> be to hide the block, with any number of keywords above, and TAB
>> pressed at any of them.
>>
>
> Yes, I would agree, the hiding should be smart enough to find the whole
> unit and hide it.  I'll take a look at the code.
>

I've just pushed up a fix for this issue.

>> `org-export-blocks-preprocess' will remove all "#+name:" keywords in the
>> buffer.  It mustn't: again "#+name:" is a general naming mechanism for
>> almost any Org syntax. It may/will be also used for other purpose than
>> Babel. That information is important even after blocks have been
>> processed.
>>
>
> I'll take a look at this and submit a patch.
>

And this one as well

Cheers,

-- 
Eric Schulte
http://cs.unm.edu/~eschulte/

Re: [BUG] Inconsistency in src block hiding

[Nicolas Goaziou]

Hello,

Eric Schulte <schulte.eric@gmail.com> writes:

> Nicolas Goaziou <n.goaziou@gmail.com> writes:
>
>> It is inconsistent when keywords stack on top of each other. If you have
>> only a "#+name:" keyword, block with fold at the "#+name:" level. If you
>> have both "#+name:" and, for example "#+results" below, block will fold
>> at the "#+result:" level and TAB will not respond at "#+name:".
>
> Can you provide an example, I'm thinking that what you describe may not
> be legal syntax.

Consider the following cases:

--8<---------------cut here---------------start------------->8---
#+name: one-more
#+header: :var k=2
#+begin_src emacs-lisp
(1+ k)
#+end_src

#+header: :var k=2
#+name: one-more
#+begin_src emacs-lisp
(1+ k)
#+end_src

#+attr_html: :textarea t :height 10 :width 40
#+name: unique-name
#+begin_example
Edit me!
#+end_example
--8<---------------cut here---------------end--------------->8---

Note that the second case doesn't appear to be legal, as executing the
block errors out with "Symbol's value as variable is void: k". I don't
think that there should be any imposed order in affiliated keywords.

Anyway, in the first case, block will be hidden only when TAB is pressed
on the "#+begin_src" line. In the second case, block can be hidden when
TAB is pressed on both the "#+name:" and the "#+begin_" line, with two
different results. That's confusing.

Only lines below "#+begin_" should be hidden, with TAB pressed on any
keyword. Affiliated keywords should always be visible.


>> If you have, from top to bottom, "name", "results" "header", nothing
>> will fold.  In all those cases, I think a consistent behaviour could
>> be to hide the block, with any number of keywords above, and TAB
>> pressed at any of them.
>>
>
> Yes, I would agree, the hiding should be smart enough to find the whole
> unit and hide it.  I'll take a look at the code.

Or rely on Org Elements... *coughs*

>> I'm not sure that "#+results:" or "#+name:" keywords should allow to
>> hide whole parts of the buffer. I realize that toggling "#+results:"
>> visibility has been in core for a while. But now that every Org blob can
>> have a "#+name" attached to it, one can hide almost anything in the
>> buffer.
>>
>> Until now we could hide contents with headlines, blocks, drawers, and
>> items (with an option to prevent it). And we had a global mechanism to
>> handle visibility toggling just fine (namely S-TAB with different
>> numbers of prefixes). But hiding independently each list, table or
>> paragraph with no possibility to make them appear in groups just doesn't
>> sound right.
>>
>> Hence, I suggest to think again about it: we can live happily with
>> outlines, blocks and drawers as folding entities.
>>
>
> The hiding was added because code blocks occasionally generate *huge*
> results which make it impossible to read further down in the buffer.
> Hiding such large results can be very useful when you want to keep the
> data in-buffer, but still want to be able to read down the page.

Then wraps a drawer around the result. Their purpose is to hide
arbitrary large parts of a buffer. Why inventing yet another way?

> Is there a way to bring the hiding of results more in-line with the
> other methods of hiding in Org-mode?  Should S-Tab be made to toggle the
> hidden states of named entities as well as outline levels?

Again, drawers are in-line with standard hiding methods. Though, their
behaviour with regards to export needs to be changed (i.e. by default
simply export contents of the drawer instead of ignoring it).

I think we should drop any "#+result:" or "#+name:" hiding and take
another route. It's not their job anyway.

>> `org-export-blocks-preprocess' will remove all "#+name:" keywords in the
>> buffer.  It mustn't: again "#+name:" is a general naming mechanism for
>> almost any Org syntax. It may/will be also used for other purpose than
>> Babel. That information is important even after blocks have been
>> processed.
>>
>
> I'll take a look at this and submit a patch.

You took care of the problem even before I could thank you for thinking
about fixing it.


Regards,

-- 
Nicolas Goaziou

Re: [BUG] Inconsistency in src block hiding

[Eric Schulte]

>
> Consider the following cases:
>
> #+name: one-more
> #+header: :var k=2
> #+begin_src emacs-lisp
> (1+ k)
> #+end_src
>
> #+header: :var k=2
> #+name: one-more
> #+begin_src emacs-lisp
> (1+ k)
> #+end_src
>
> #+attr_html: :textarea t :height 10 :width 40
> #+name: unique-name
> #+begin_example
> Edit me!
> #+end_example
>
> Note that the second case doesn't appear to be legal, as executing the
> block errors out with "Symbol's value as variable is void: k". I don't
> think that there should be any imposed order in affiliated keywords.
>

I would say that the first and third should be valid constructions, but
the second should not.  This is for two reasons.

1. As you point out the second is not currently supported, and generally
   changing the code should be avoided.

2. More importantly, unlike the #+attr_html line in the third example,
   the #+header line *only* applies to code blocks and not to any other
   Org-mode element.  Given that I think it is part of the code block in
   a more intimate way than #+name is part of the code block, and thus
   should not be separated from the code block by a name.  In effect I
   think of the code block (including the #+header line) as an Org-mode
   object, and the #+name line as something applied to that element as a
   whole.

I suppose I would yield to you on this point as you actually have a
systematic way of defining Org-mode objects and keywords (org-elements),
where as all I have is intuition and the vague notion that option two
above "looks ugly" to me.

Although, that said I don't really look forward to the task of changing
the code to allow any arbitrary sequence of #+name and #+header
arguments prefixing a code block as it will complicate the current code
significantly.

>
> Anyway, in the first case, block will be hidden only when TAB is pressed
> on the "#+begin_src" line.

In the current code it will also be hidden if TAB is pressed on the
#+name line.

> In the second case, block can be hidden when TAB is pressed on both
> the "#+name:" and the "#+begin_" line, with two different
> results. That's confusing.
>

I did work out a version of this patch in which when tab was pressed on
the #+name line, only the code block was hidden, however in my
conception where names can be applied to any Org-mode object, and
intervening #+header lines are part of the code block it seemed more
consistent to hide the whole object following the name, so e.g., hiding
this with TAB on #+name

#+name: one-more
#+header: :var k=2
#+begin_src emacs-lisp
(1+ k)
#+end_src

should result in this (as it does now)

#+name: one-more...

rather than this

#+name: one-more
#+header: :var k=2
#+begin_src emacs-lisp...

>
> Only lines below "#+begin_" should be hidden, with TAB pressed on any
> keyword. Affiliated keywords should always be visible.
>

I see your view, and it is consistent, it is just different from mine.

>
>
>>> If you have, from top to bottom, "name", "results" "header", nothing
>>> will fold.  In all those cases, I think a consistent behaviour could
>>> be to hide the block, with any number of keywords above, and TAB
>>> pressed at any of them.
>>>
>>
>> Yes, I would agree, the hiding should be smart enough to find the whole
>> unit and hide it.  I'll take a look at the code.
>
> Or rely on Org Elements... *coughs*
>

:) I agree, Ideally I would love to only manipulate the list provided by
org-element.el, however now I am more concerned with changes before the
upcoming Emacs merge before the release of Emacs24.  So relying on
org-elements is not currently an option.

>
>>> I'm not sure that "#+results:" or "#+name:" keywords should allow to
>>> hide whole parts of the buffer. I realize that toggling "#+results:"
>>> visibility has been in core for a while. But now that every Org blob can
>>> have a "#+name" attached to it, one can hide almost anything in the
>>> buffer.
>>>
>>> Until now we could hide contents with headlines, blocks, drawers, and
>>> items (with an option to prevent it). And we had a global mechanism to
>>> handle visibility toggling just fine (namely S-TAB with different
>>> numbers of prefixes). But hiding independently each list, table or
>>> paragraph with no possibility to make them appear in groups just doesn't
>>> sound right.
>>>
>>> Hence, I suggest to think again about it: we can live happily with
>>> outlines, blocks and drawers as folding entities.
>>>
>>
>> The hiding was added because code blocks occasionally generate *huge*
>> results which make it impossible to read further down in the buffer.
>> Hiding such large results can be very useful when you want to keep the
>> data in-buffer, but still want to be able to read down the page.
>
> Then wraps a drawer around the result. Their purpose is to hide
> arbitrary large parts of a buffer. Why inventing yet another way?
>

I like this idea, and would be happy to strip all hiding functionality
away from results... It it possible to wrap all result types (e.g.,
scalar, raw org text, tables, blocks) in drawers?

>
>> Is there a way to bring the hiding of results more in-line with the
>> other methods of hiding in Org-mode?  Should S-Tab be made to toggle the
>> hidden states of named entities as well as outline levels?
>
> Again, drawers are in-line with standard hiding methods. Though, their
> behaviour with regards to export needs to be changed (i.e. by default
> simply export contents of the drawer instead of ignoring it).
>
> I think we should drop any "#+result:" or "#+name:" hiding and take
> another route. It's not their job anyway.
>

Fair enough, I like this idea.

Cheers,

>
>>> `org-export-blocks-preprocess' will remove all "#+name:" keywords in the
>>> buffer.  It mustn't: again "#+name:" is a general naming mechanism for
>>> almost any Org syntax. It may/will be also used for other purpose than
>>> Babel. That information is important even after blocks have been
>>> processed.
>>>
>>
>> I'll take a look at this and submit a patch.
>
> You took care of the problem even before I could thank you for thinking
> about fixing it.
>
>
> Regards,

-- 
Eric Schulte
http://cs.unm.edu/~eschulte/

Re: [BUG] Inconsistency in src block hiding

[Nicolas Goaziou]

Eric Schulte <schulte.eric@gmail.com> writes:

>> Consider the following cases:
>>
>> #+name: one-more
>> #+header: :var k=2
>> #+begin_src emacs-lisp
>> (1+ k)
>> #+end_src
>>
>> #+header: :var k=2
>> #+name: one-more
>> #+begin_src emacs-lisp
>> (1+ k)
>> #+end_src
>>
>> #+attr_html: :textarea t :height 10 :width 40
>> #+name: unique-name
>> #+begin_example
>> Edit me!
>> #+end_example
>>
>> Note that the second case doesn't appear to be legal, as executing the
>> block errors out with "Symbol's value as variable is void: k". I don't
>> think that there should be any imposed order in affiliated keywords.
>>
>
> I would say that the first and third should be valid constructions, but
> the second should not.  This is for two reasons.
>
> 1. As you point out the second is not currently supported, and generally
>    changing the code should be avoided.

I'm just thinking about a super-set of what's currently supported, that
is to allow any order instead of a static one. So that's not really
a problem.

> 2. More importantly, unlike the #+attr_html line in the third example,
>    the #+header line *only* applies to code blocks and not to any other
>    Org-mode element.

For now, you may be right. But I consider "#+header" as a continuation
of the information specified at the block beginning line. With that in
mind, there is at least one other block type that allows information
there: example-block. So an "#+header" keyword could be as appropriate
for it. I'm also thinking about Jambunathan's "#+begin_annotation".

> I suppose I would yield to you on this point as you actually have
> a systematic way of defining Org-mode objects and keywords
> (org-elements), where as all I have is intuition and the vague notion
> that option two above "looks ugly" to me.

> Although, that said I don't really look forward to the task of
> changing the code to allow any arbitrary sequence of #+name
> and #+header arguments prefixing a code block as it will complicate
> the current code significantly.

Not necessarily (see below).

>> In the second case, block can be hidden when TAB is pressed on both
>> the "#+name:" and the "#+begin_" line, with two different
>> results. That's confusing.

> I did work out a version of this patch in which when tab was pressed on
> the #+name line, only the code block was hidden, however in my
> conception where names can be applied to any Org-mode object, and
> intervening #+header lines are part of the code block it seemed more
> consistent to hide the whole object following the name, so e.g., hiding
> this with TAB on #+name
>
>
> #+name: one-more
> #+header: :var k=2
> #+begin_src emacs-lisp
> (1+ k)
> #+end_src
>
> should result in this (as it does now)
>
>
> #+name: one-more...
>
>
> rather than this
>
>
> #+name: one-more
> #+header: :var k=2
> #+begin_src emacs-lisp...

Well, if we lived in a perfect world where anyone could smartly name
things the first time they encounter them, I'll be on your side. But,
the names I choose are often rather vague, so I'd always prefer to both
see the name and a have a glimpse of the named object. Thus, when the
wind is southerly, I can know a src-block from an example-block.

That also explains why I dislike the "#+name:" hiding idea: lists,
paragraphs, tables will all look the same. Also, it could potentially
increase a lot the number of overlays used in a buffer, which isn't good
performance-wise.

>> Only lines below "#+begin_" should be hidden, with TAB pressed on any
>> keyword. Affiliated keywords should always be visible.
>>
>
> I see your view, and it is consistent, it is just different from mine.
>

I agree. I try nonetheless to make you see mine as more consistent than
yours ;)

>>>> If you have, from top to bottom, "name", "results" "header", nothing
>>>> will fold.  In all those cases, I think a consistent behaviour could
>>>> be to hide the block, with any number of keywords above, and TAB
>>>> pressed at any of them.
>>>>
>>>
>>> Yes, I would agree, the hiding should be smart enough to find the whole
>>> unit and hide it.  I'll take a look at the code.
>>
>> Or rely on Org Elements... *coughs*
>>
>
> :) I agree, Ideally I would love to only manipulate the list provided by
> org-element.el, however now I am more concerned with changes before the
> upcoming Emacs merge before the release of Emacs24.  So relying on
> org-elements is not currently an option.

But borrowing parts of it is allowed in the game.

More specifically, you need to copy and rename a single function
`org-element-collect-affiliated-keywords', and four variables
(`org-element--affiliated-re', `org-element-keyword-translation-alist',
`org-element-parsed-keywords' and `org-element-dual-keywords'.)

Now, in the following example, which will hurt your eyes, 

#+header: :var k=3 :exports code
#+headers: :var k=2
#+caption: One more time.
#+name: one-more-time
#+begin_src emacs-lisp
(1+ k)
#+end_src

calling (org-element-collect-affiliated-keywords) on the "#+begin_src"
line will result in:

(1 (:name "one-more-time" 
    :caption ("One more time.")
    :header (":var k=3 :exports code" ":var k=2")))

1 being the position at the beginning of the first "#+header:" line.

So you get both the beginning position of the block (because "#+caption"
and "#+name", being affiliated keywords, belong to the block in the same
way as "#+header:" does) and all the info you need. And that, again,
with only one function, not the full library (if you don't want to use
the variables, you can pass their value as arguments to the function).

>>>> I'm not sure that "#+results:" or "#+name:" keywords should allow to
>>>> hide whole parts of the buffer. I realize that toggling "#+results:"
>>>> visibility has been in core for a while. But now that every Org blob can
>>>> have a "#+name" attached to it, one can hide almost anything in the
>>>> buffer.
>>>>
>>>> Until now we could hide contents with headlines, blocks, drawers, and
>>>> items (with an option to prevent it). And we had a global mechanism to
>>>> handle visibility toggling just fine (namely S-TAB with different
>>>> numbers of prefixes). But hiding independently each list, table or
>>>> paragraph with no possibility to make them appear in groups just doesn't
>>>> sound right.
>>>>
>>>> Hence, I suggest to think again about it: we can live happily with
>>>> outlines, blocks and drawers as folding entities.
>>>>
>>>
>>> The hiding was added because code blocks occasionally generate *huge*
>>> results which make it impossible to read further down in the buffer.
>>> Hiding such large results can be very useful when you want to keep the
>>> data in-buffer, but still want to be able to read down the page.
>>
>> Then wraps a drawer around the result. Their purpose is to hide
>> arbitrary large parts of a buffer. Why inventing yet another way?
>>
>
> I like this idea, and would be happy to strip all hiding functionality
> away from results... Is it possible to wrap all result types (e.g.,
> scalar, raw org text, tables, blocks) in drawers?

Everything fits in a drawer, excepted another drawer, or an headline. It
only needs to be defined in `org-drawers' or on the "#+drawers:" buffer
property.

When I want to center the output of a code block, I wrap it into
a center-block. If I really want to hide it, I should be able to do the
same with a drawer. But in both case, that's to the user discretion. So
don't bother providing an option to automatically do this.

>>> Is there a way to bring the hiding of results more in-line with the
>>> other methods of hiding in Org-mode?  Should S-Tab be made to toggle the
>>> hidden states of named entities as well as outline levels?
>>
>> Again, drawers are in-line with standard hiding methods. Though, their
>> behaviour with regards to export needs to be changed (i.e. by default
>> simply export contents of the drawer instead of ignoring it).
>>
>> I think we should drop any "#+result:" or "#+name:" hiding and take
>> another route. It's not their job anyway.
>>
>
> Fair enough, I like this idea.

So let's work towards drawers rehabilitation.


Regards,

-- 
Nicolas Goaziou

Re: [BUG] Inconsistency in src block hiding

[Eric Schulte]

[-- Attachment #1: Type: text/plain, Size: 908 bytes --]

>>>
>>> Again, drawers are in-line with standard hiding methods. Though, their
>>> behaviour with regards to export needs to be changed (i.e. by default
>>> simply export contents of the drawer instead of ignoring it).
>>>
>>> I think we should drop any "#+result:" or "#+name:" hiding and take
>>> another route. It's not their job anyway.
>>>
>>
>> Fair enough, I like this idea.
>
> So let's work towards drawers rehabilitation.
>

The attached patch entirely removes the #+name and #+results based
hiding.  Note that the existing "wrap" argument to the ":results" header
argument will wrap results in a block which allows easy tab-based result
hiding.

As this is a relatively large change I hesitate to apply it outright,
although I do agree that it leads to simpler more consistent behavior.
If general consensus seems to support the application of this patch then
I will be happy to apply it.

Best,


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: 0001-remove-name-and-result-hiding.patch --]
[-- Type: text/x-diff, Size: 4538 bytes --]

From 8eec0e67d0e9ea703f0449310ca4db8c600e880f Mon Sep 17 00:00:00 2001
From: Eric Schulte <schulte.eric@gmail.com>
Date: Mon, 28 Nov 2011 00:43:35 -0700
Subject: [PATCH] remove #+name and #+result hiding

Given that arbitrary regions may already be hidden using a drawer the
ability to hide any named entity is redundant.  In addition the name
based hiding does not mix well with other keywords.

Thanks to Nicolas Goaziou for raising this issue and suggesting the
removal of #+name based hiding.
---
 lisp/ob.el |   83 ------------------------------------------------------------
 1 files changed, 0 insertions(+), 83 deletions(-)

diff --git a/lisp/ob.el b/lisp/ob.el
index dab40f4..5815fe6 100644
--- a/lisp/ob.el
+++ b/lisp/ob.el
@@ -975,89 +975,6 @@ This can be called with C-c C-c."
     (when hash (kill-new hash) (message hash))))
 (add-hook 'org-ctrl-c-ctrl-c-hook 'org-babel-hash-at-point)
 
-(defun org-babel-result-hide-spec ()
-  "Hide portions of results lines.
-Add `org-babel-hide-result' as an invisibility spec for hiding
-portions of results lines."
-  (add-to-invisibility-spec '(org-babel-hide-result . t)))
-(add-hook 'org-mode-hook 'org-babel-result-hide-spec)
-
-(defvar org-babel-hide-result-overlays nil
-  "Overlays hiding results.")
-
-(defun org-babel-result-hide-all ()
-  "Fold all results in the current buffer."
-  (interactive)
-  (org-babel-show-result-all)
-  (save-excursion
-    (while (re-search-forward org-babel-result-regexp nil t)
-      (save-excursion (goto-char (match-beginning 0))
-                      (org-babel-hide-result-toggle-maybe)))))
-
-(defun org-babel-show-result-all ()
-  "Unfold all results in the current buffer."
-  (mapc 'delete-overlay org-babel-hide-result-overlays)
-  (setq org-babel-hide-result-overlays nil))
-
-;;;###autoload
-(defun org-babel-hide-result-toggle-maybe ()
-  "Toggle visibility of result at point."
-  (interactive)
-  (let ((case-fold-search t))
-    (if (save-excursion
-          (beginning-of-line 1)
-          (looking-at org-babel-result-regexp))
-        (progn (org-babel-hide-result-toggle)
-               t) ;; to signal that we took action
-      nil))) ;; to signal that we did not
-
-(defun org-babel-hide-result-toggle (&optional force)
-  "Toggle the visibility of the current result."
-  (interactive)
-  (save-excursion
-    (beginning-of-line)
-    (if (re-search-forward org-babel-result-regexp nil t)
-        (let ((start (progn (beginning-of-line 2) (- (point) 1)))
-	      (end (progn
-		     (while (looking-at org-babel-multi-line-header-regexp)
-		       (forward-line 1))
-		     (goto-char (- (org-babel-result-end) 1)) (point)))
-	      ov)
-          (if (memq t (mapcar (lambda (overlay)
-                                (eq (overlay-get overlay 'invisible)
-				    'org-babel-hide-result))
-                              (overlays-at start)))
-              (if (or (not force) (eq force 'off))
-                  (mapc (lambda (ov)
-                          (when (member ov org-babel-hide-result-overlays)
-                            (setq org-babel-hide-result-overlays
-                                  (delq ov org-babel-hide-result-overlays)))
-                          (when (eq (overlay-get ov 'invisible)
-                                    'org-babel-hide-result)
-                            (delete-overlay ov)))
-                        (overlays-at start)))
-            (setq ov (make-overlay start end))
-            (overlay-put ov 'invisible 'org-babel-hide-result)
-            ;; make the block accessible to isearch
-            (overlay-put
-             ov 'isearch-open-invisible
-             (lambda (ov)
-               (when (member ov org-babel-hide-result-overlays)
-                 (setq org-babel-hide-result-overlays
-                       (delq ov org-babel-hide-result-overlays)))
-               (when (eq (overlay-get ov 'invisible)
-                         'org-babel-hide-result)
-                 (delete-overlay ov))))
-            (push ov org-babel-hide-result-overlays)))
-      (error "Not looking at a result line"))))
-
-;; org-tab-after-check-for-cycling-hook
-(add-hook 'org-tab-first-hook 'org-babel-hide-result-toggle-maybe)
-;; Remove overlays when changing major mode
-(add-hook 'org-mode-hook
-	  (lambda () (org-add-hook 'change-major-mode-hook
-				   'org-babel-show-result-all 'append 'local)))
-
 (defvar org-file-properties)
 (defun org-babel-params-from-properties (&optional lang)
   "Retrieve parameters specified as properties.
-- 
1.7.4.1


[-- Attachment #3: Type: text/plain, Size: 47 bytes --]


-- 
Eric Schulte
http://cs.unm.edu/~eschulte/

Re: [BUG] Inconsistency in src block hiding

[Bastien]

Eric Schulte <schulte.eric@gmail.com> writes:

> The attached patch entirely removes the #+name and #+results based
> hiding.  Note that the existing "wrap" argument to the ":results" header
> argument will wrap results in a block which allows easy tab-based result
> hiding.
>
> As this is a relatively large change I hesitate to apply it outright,
> although I do agree that it leads to simpler more consistent behavior.
> If general consensus seems to support the application of this patch then
> I will be happy to apply it.

Has there been any more feedback on this?  

Erik, is it safe to apply the patch or shall we wait more?

Thanks,

-- 
 Bastien

Re: [BUG] Inconsistency in src block hiding

[Nicolas Goaziou]

Bastien <bzg@altern.org> writes:

> Eric Schulte <schulte.eric@gmail.com> writes:
>
>> The attached patch entirely removes the #+name and #+results based
>> hiding.  Note that the existing "wrap" argument to the ":results" header
>> argument will wrap results in a block which allows easy tab-based result
>> hiding.

I didn't notice it before, but the wrap option from :results property
could also be removed. Unlike to every other block "#+begin_results"
doesn't add any meaning. Its sole purpose is to group some Org data,
which is the job of drawers.

Moreover, it's already easy to install results in a block (I use it to
center results). You only need to put the results line in an appropriate
location and evaluate the code block, like in the following:

--8<---------------cut here---------------start------------->8---
#+name: four
#+begin_src emacs-lisp
(+ 2 2)
#+end_src

#+begin_center
#+results: four
#+end_center
--8<---------------cut here---------------end--------------->8---

Note that the wrap argument may be used instead to wrap results in
a :RESULTS: drawer, but that default drawer would have then to be added
to the `org-drawers' list, and, in the end, wouldn't be very practical.

In my opinion, we should forget any will to automatically group results
(because there is no option more interesting than others) and let the
user decide what suits him the best.


Regards,

-- 
Nicolas Goaziou

Re: [BUG] Inconsistency in src block hiding

[Eric Schulte]

Nicolas Goaziou <n.goaziou@gmail.com> writes:

> Bastien <bzg@altern.org> writes:
>
>> Eric Schulte <schulte.eric@gmail.com> writes:
>>
>>> The attached patch entirely removes the #+name and #+results based
>>> hiding.  Note that the existing "wrap" argument to the ":results" header
>>> argument will wrap results in a block which allows easy tab-based result
>>> hiding.
>
> I didn't notice it before, but the wrap option from :results property
> could also be removed. Unlike to every other block "#+begin_results"
> doesn't add any meaning. Its sole purpose is to group some Org data,
> which is the job of drawers.
>
> Moreover, it's already easy to install results in a block (I use it to
> center results). You only need to put the results line in an appropriate
> location and evaluate the code block, like in the following:
>
> #+name: four
> #+begin_src emacs-lisp
> (+ 2 2)
> #+end_src
>
> #+begin_center
> #+results: four
> #+end_center
>

One major utility of the wrap :results header argument is that it allows
blocks of raw Org-mode text returned by a code block to be automatically
replaced on every execution.  This is possible because the wrapping
block is used to delimit the result in the Org-mode file.  This is not
possible with your solution above.

>
> Note that the wrap argument may be used instead to wrap results in
> a :RESULTS: drawer, but that default drawer would have then to be added
> to the `org-drawers' list, and, in the end, wouldn't be very practical.
>

I agree a :RESULTS: drawer would be a more meaningful way to wrap code
block results.  I don't see why such a solution is not practical.  I
just pushed up a change that defaults to wrapping results in new RESULTS
drawers rather than blocks.

>
> In my opinion, we should forget any will to automatically group results
> (because there is no option more interesting than others) and let the
> user decide what suits him the best.
>

I would disagree because of the result delimitation functionality
mentioned above.

Best,

-- 
Eric Schulte
http://cs.unm.edu/~eschulte/

Re: [BUG] Inconsistency in src block hiding

[Eric Schulte]

Bastien <bzg@altern.org> writes:

> Eric Schulte <schulte.eric@gmail.com> writes:
>
>> The attached patch entirely removes the #+name and #+results based
>> hiding.  Note that the existing "wrap" argument to the ":results" header
>> argument will wrap results in a block which allows easy tab-based result
>> hiding.
>>
>> As this is a relatively large change I hesitate to apply it outright,
>> although I do agree that it leads to simpler more consistent behavior.
>> If general consensus seems to support the application of this patch then
>> I will be happy to apply it.
>
> Has there been any more feedback on this?  
>
> Erik, is it safe to apply the patch or shall we wait more?
>

I've received no more feedback on this patch.  It should be safe as it
adds no new code but simply removes some questionable code.  I will
apply this patch now.

Best,

-- 
Eric Schulte
http://cs.unm.edu/~eschulte/

Re: [BUG] Inconsistency in src block hiding

[Bastien]

Eric Schulte <eric.schulte@gmx.com> writes:

> I've received no more feedback on this patch.  It should be safe as it
> adds no new code but simply removes some questionable code.  I will
> apply this patch now.

Seen - thanks!

-- 
 Bastien

Re: [BUG] Inconsistency in src block hiding

[Bernt Hansen]

Eric Schulte <eric.schulte@gmx.com> writes:

> Bastien <bzg@altern.org> writes:
>
>> Eric Schulte <schulte.eric@gmail.com> writes:
>>
>>> The attached patch entirely removes the #+name and #+results based
>>> hiding.  Note that the existing "wrap" argument to the ":results" header
>>> argument will wrap results in a block which allows easy tab-based result
>>> hiding.
>>>
>>> As this is a relatively large change I hesitate to apply it outright,
>>> although I do agree that it leads to simpler more consistent behavior.
>>> If general consensus seems to support the application of this patch then
>>> I will be happy to apply it.
>>
>> Has there been any more feedback on this?  
>>
>> Erik, is it safe to apply the patch or shall we wait more?
>>
>
> I've received no more feedback on this patch.  It should be safe as it
> adds no new code but simply removes some questionable code.  I will
> apply this patch now.
>
> Best,

So I'm (very) late to this party...  but today I decided to track down
why I can't hide my PlantUML graphics with TAB anymore... and it's this
commit:

--8<---------------cut here---------------start------------->8---
commit 8a8a56c27715740053d8a7ea2ba6411b781a22f1
Author: Eric Schulte <schulte.eric@gmail.com>
Date:   Mon Nov 28 00:43:35 2011 -0700

    remove #+name and #+result hiding
    
    Given that arbitrary regions may already be hidden using a drawer the
    ability to hide any named entity is redundant.  In addition the name
    based hiding does not mix well with other keywords.
    
    Thanks to Nicolas Goaziou for raising this issue and suggesting the
    removal of #+name based hiding.
--8<---------------cut here---------------end--------------->8---

I actually miss being able to hide single (large) activity diagrams at
work since they tend to be larger than my screen when displayed in
Emacs.  I can use M-x org-toggle-inline-images but that works globally
on all images at once.

Is there an easy way to wrap my resulting PlantUML images so I can use
TAB to hide them - or is this not supported anymore?

I tried :results wrap but that didn't work for me.  If I add RESULTS to
my list of drawers then I can hide the block with TAB but I can't export
my diagrams to HTML anymore which isn't very satisfying.

Is there still a way to hide results output with the current master?

Regards,
Bernt

Re: [BUG] Inconsistency in src block hiding

[Nicolas Goaziou]

Hello,

Bernt Hansen <bernt@norang.ca> writes:

> I tried :results wrap but that didn't work for me.  If I add RESULTS to
> my list of drawers then I can hide the block with TAB but I can't export
> my diagrams to HTML anymore which isn't very satisfying.

Why can't you? Wouldn't it be related to drawers configuration
(org-export-with-drawers for example)?

> Is there still a way to hide results output with the current master?

Yes, within a drawer.


Regards,

-- 
Nicolas Goaziou

Re: [BUG] Inconsistency in src block hiding

[Bernt Hansen]

Nicolas Goaziou <n.goaziou@gmail.com> writes:

> Hello,
>
> Bernt Hansen <bernt@norang.ca> writes:
>
>> I tried :results wrap but that didn't work for me.  If I add RESULTS to
>> my list of drawers then I can hide the block with TAB but I can't export
>> my diagrams to HTML anymore which isn't very satisfying.
>
> Why can't you? Wouldn't it be related to drawers configuration
> (org-export-with-drawers for example)?

Yes... but I don't think I can configure which drawers I get, and I
don't want my LOGBOOK drawer with all my clock lines in my export.

-Bernt
>
>> Is there still a way to hide results output with the current master?
>
> Yes, within a drawer.
>
>
> Regards,

Re: [BUG] Inconsistency in src block hiding

[Leo Alekseyev]

>> Why can't you? Wouldn't it be related to drawers configuration
>> (org-export-with-drawers for example)?
>
> Yes... but I don't think I can configure which drawers I get, and I
> don't want my LOGBOOK drawer with all my clock lines in my export.
>
> -Bernt
>>
>>> Is there still a way to hide results output with the current master?
>>
>> Yes, within a drawer.

Yes, but is it possible to hide the drawer name in the HTML export?
If drawers are designed to be used as an organization device (and not
an outlining device), the drawer delimiters shouldn't be exported to
HTML.

Re: [BUG] Inconsistency in src block hiding

[Rick Frankel]

On 18.01.2012 05:45, Leo Alekseyev wrote:
>>> Why can't you? Wouldn't it be related to drawers configuration
>>> (org-export-with-drawers for example)?
>>
>> Yes... but I don't think I can configure which drawers I get, and I
>> don't want my LOGBOOK drawer with all my clock lines in my export.
>>
>> -Bernt
>>>
>>>> Is there still a way to hide results output with the current 
>>>> master?
>>>
>>> Yes, within a drawer.
>
> Yes, but is it possible to hide the drawer name in the HTML export?
> If drawers are designed to be used as an organization device (and not
> an outlining device), the drawer delimiters shouldn't be exported to
> HTML.

It's worse than that. Once wrapped in a RESULTS drawer, the output is 
no
longer export.

I aggree w/ Bernt, this has been a major annoyance for me, in fact, i 
was
going to post it as a regession (not having seen the original post).

I use org-mode for generating a lot of self-documenting code (w/o 
noweb). I
generate large images and data tables which i want in the export, but
don't normally want to see while working in the source file.

rick

Re: [BUG] Inconsistency in src block hiding

[Eric Schulte]

[-- Attachment #1: Type: text/plain, Size: 1608 bytes --]

Rick Frankel <rick@rickster.com> writes:

> On 18.01.2012 05:45, Leo Alekseyev wrote:
>>>> Why can't you? Wouldn't it be related to drawers configuration
>>>> (org-export-with-drawers for example)?
>>>
>>> Yes... but I don't think I can configure which drawers I get, and I
>>> don't want my LOGBOOK drawer with all my clock lines in my export.
>>>
>>> -Bernt
>>>>
>>>>> Is there still a way to hide results output with the current
>>>>> master?
>>>>
>>>> Yes, within a drawer.
>>
>> Yes, but is it possible to hide the drawer name in the HTML export?
>> If drawers are designed to be used as an organization device (and not
>> an outlining device), the drawer delimiters shouldn't be exported to
>> HTML.
>
> It's worse than that. Once wrapped in a RESULTS drawer, the output is
> no longer export.
>
> I aggree w/ Bernt, this has been a major annoyance for me, in fact, i
> was going to post it as a regession (not having seen the original
> post).
>
> I use org-mode for generating a lot of self-documenting code (w/o
> noweb). I generate large images and data tables which i want in the
> export, but don't normally want to see while working in the source
> file.
>

Well maybe we should roll back this change.  It was an attempt to bring
code block syntax more /inline/ with the rest of Org-mode, but if it
proves to cause usability problems I think that trumps clean design.

I'll wait to see if Nicolas has a solution which is both functional and
conforms to the Org-mode wide syntax norms.  In the mean time the
solution used in the following Org-mode file should provide the desired
functionality.


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: folding-results.org --]
[-- Type: text/x-org, Size: 183 bytes --]

#+Property: wrap results

#+BEGIN_SRC emacs-lisp :exports results
  '((1 2 3 4)
    (5 6 7 8))
#+END_SRC

#+results:
#+BEGIN_results
| 1 | 2 | 3 | 4 |
| 5 | 6 | 7 | 8 |
#+END_results

[-- Attachment #3: Type: text/plain, Size: 66 bytes --]


Best,

>
> rick
>

-- 
Eric Schulte
http://cs.unm.edu/~eschulte/

Re: [BUG] Inconsistency in src block hiding

[Nicolas Goaziou]

[-- Attachment #1: Type: text/plain, Size: 1128 bytes --]

Hello,

Eric Schulte <eric.schulte@gmx.com> writes:

> Well maybe we should roll back this change.

Please don't.  _That_ would be a regression.

> I'll wait to see if Nicolas has a solution which is both functional and
> conforms to the Org-mode wide syntax norms.

The problem comes from the current exporter, which isn't neutral about
drawers, by default.

I think that a temporary fix should be to:

  1. Change default drawer formatting function to the following
     (neutral):

     #+begin_src emacs-lisp
     (defun org-export-format-drawer (name contents)
       "Export contents of a drawer as-is.
     Property drawers are ignored."
       (if (string= "PROPERTIES" name) "" contents))
     #+end_src

  2. Handle drawers earlier in org-export-preprocess-string function,
     i.e. just before footnote handling, so its contents can benefit
     from further modifications.

  3. Allow drawers in export by default (excepted PROPERTIES
     drawers). That is change `org-export-with-drawers' default value to
     t.

I'm packaging this in the following quick patch, highly untested.


Regards,

-- 
Nicolas Goaziou

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: patch for drawers --]
[-- Type: text/x-patch, Size: 3138 bytes --]

From 0c15bf694f8051eb58fd131868059460f28f2e0d Mon Sep 17 00:00:00 2001
From: Nicolas Goaziou <n.goaziou@gmail.com>
Date: Wed, 18 Jan 2012 18:34:11 +0100
Subject: [PATCH] org-exp: Set neutral behaviour towards drawers

* lisp/org-exp.el (org-export-with-drawers): Change default value so
  all drawers are exportable as a default.
(org-export-preprocess-string): Handle drawers earlier in the
  preprocess, so their contents can be modified further.
(org-export-format-drawer): Change default behaviour contents of
  drawers are exported as Org code.  As a special case, property
  drawers are still ignored.
---
 lisp/org-exp.el |   26 ++++++++------------------
 1 files changed, 8 insertions(+), 18 deletions(-)

diff --git a/lisp/org-exp.el b/lisp/org-exp.el
index c7e1a94..d9b0a3e 100644
--- a/lisp/org-exp.el
+++ b/lisp/org-exp.el
@@ -406,7 +406,7 @@ This option can also be set with the +OPTIONS line, e.g. \"tags:nil\"."
 	  (const :tag "Not in TOC" not-in-toc)
 	  (const :tag "On" t)))
 
-(defcustom org-export-with-drawers nil
+(defcustom org-export-with-drawers t
   "Non-nil means export with drawers like the property drawer.
 When t, all drawers are exported.  This may also be a list of
 drawer names to export."
@@ -1156,6 +1156,10 @@ on this string to produce the exported version."
       ;; Get rid of tasks, depending on configuration
       (org-export-remove-tasks (plist-get parameters :tasks))
 
+      ;; Get rid of drawers
+      (org-export-remove-or-extract-drawers
+       drawers (plist-get parameters :drawers))
+
       ;; Prepare footnotes for export.  During that process, footnotes
       ;; actually included in the exported part of the buffer go
       ;; though some transformations:
@@ -1209,10 +1213,6 @@ on this string to produce the exported version."
       ;; Find HTML special classes for headlines
       (org-export-remember-html-container-classes)
 
-      ;; Get rid of drawers
-      (org-export-remove-or-extract-drawers
-       drawers (plist-get parameters :drawers))
-
       ;; Get the correct stuff before the first headline
       (when (plist-get parameters :skip-before-1st-heading)
 	(goto-char (point-min))
@@ -1500,19 +1500,9 @@ EXP-DRAWERS will be removed."
 				  name content))
 	   (insert content)))))))
 
-(defun org-export-format-drawer (name content)
-  "Format the content of a drawer as a colon example."
-  (if (string-match "[ \t]+\\'" content)
-      (setq content (substring content (match-beginning 0))))
-  (while (string-match "\\`[ \t]*\n" content)
-    (setq content (substring content (match-end 0))))
-  (setq content (org-remove-indentation content))
-  (setq content (concat ": " (mapconcat 'identity
-					(org-split-string content "\n")
-					"\n: ")
-			"\n"))
-  (setq content (concat " : " (upcase name) "\n" content))
-  (org-add-props content nil 'org-protected t))
+(defun org-export-format-drawer (name contents)
+  "Export contents of a drawer as-is."
+  (if (string= "PROPERTIES" name) "" contents))
 
 (defun org-export-handle-export-tags (select-tags exclude-tags)
   "Modify the buffer, honoring SELECT-TAGS and EXCLUDE-TAGS.
-- 
1.7.8.3

Re: [BUG] Inconsistency in src block hiding

[Martyn Jago]

Hi

Nicolas Goaziou <n.goaziou@gmail.com> writes:

> Hello,
>
> Eric Schulte <eric.schulte@gmx.com> writes:
>
>> Well maybe we should roll back this change.
>
> Please don't.  _That_ would be a regression.
>

These changes /have/ caused a software regression, and should be
reverted immediately, since:

- they change current expected and implemented behavior to the cost of
  users expectations and current use, with no prior discussion and
  agreement on behavior changes 

It has been proven that to achieve the required consistency set out by
this thread /without/ breaking current expectations is less than
straight-forward, and the changes should therefore be moved to
EXPERIMENTAL - at least until the changes are proven not to break
current expectations.

Much of org-mode is currently inconsistent - and while it is highly
desirable to improve on that situation, it should not become the major
decision to change code at the cost of regression - org-mode is a very
practical system used successfully by lots of (very appreciative)
people, and while all users appreciate bug fixes / enhancements in the
fast-moving repository that is org-mode (thanks Carsten), clear
regression such as this should be avoided in master at all costs. 

At least that is my understanding of the development model as
established by Carsten.

If this is /not/ the case, then many people are wasting their time
writing regression tests IMHO.

Best, Martyn


[...]

Re: [BUG] Inconsistency in src block hiding

[Eric Schulte]

Martyn Jago <martyn.jago@btinternet.com> writes:

> Hi
>
> Nicolas Goaziou <n.goaziou@gmail.com> writes:
>
>> Hello,
>>
>> Eric Schulte <eric.schulte@gmx.com> writes:
>>
>>> Well maybe we should roll back this change.
>>
>> Please don't.  _That_ would be a regression.
>>
>
> These changes /have/ caused a software regression, and should be
> reverted immediately, since:
>
> - they change current expected and implemented behavior to the cost of
>   users expectations and current use, with no prior discussion and
>   agreement on behavior changes 
>

I'm inclined to agree.  I've just reverted the removal of results
folding pending further discussion.  I apologize for the inconvenience.

>
> It has been proven that to achieve the required consistency set out by
> this thread /without/ breaking current expectations is less than
> straight-forward, and the changes should therefore be moved to
> EXPERIMENTAL - at least until the changes are proven not to break
> current expectations.
>
> Much of org-mode is currently inconsistent - and while it is highly
> desirable to improve on that situation, it should not become the major
> decision to change code at the cost of regression - org-mode is a very
> practical system used successfully by lots of (very appreciative)
> people, and while all users appreciate bug fixes / enhancements in the
> fast-moving repository that is org-mode (thanks Carsten), clear
> regression such as this should be avoided in master at all costs. 
>

My adviser is fond of saying something along the lines of "programming
languages should let you break the rules".  To the extent that Org-mode
is a document programming language its rules should be flexible as well.

Best,

>
> At least that is my understanding of the development model as
> established by Carsten.
>
> If this is /not/ the case, then many people are wasting their time
> writing regression tests IMHO.
>
> Best, Martyn
>
>
> [...]
>
>
>

-- 
Eric Schulte
http://cs.unm.edu/~eschulte/

Re: [BUG] Inconsistency in src block hiding

[Rick Frankel]

On 19.01.2012 07:10, Martyn Jago wrote:
> Hi
>
> Nicolas Goaziou <n.goaziou@gmail.com> writes:
>
>> Hello,
>>
>> Eric Schulte <eric.schulte@gmx.com> writes:
>>
>>> Well maybe we should roll back this change.
>>
>> Please don't.  _That_ would be a regression.
>>
>
> These changes /have/ caused a software regression, and should be
> reverted immediately, since:

Also, the exporting of a specific drawers seems to be inconsistent w/
the existing semantics for drawers in general...

rick

Re: [BUG] Inconsistency in src block hiding

[Nicolas Goaziou]

Hello,

Martyn Jago <martyn.jago@btinternet.com> writes:

> These changes /have/ caused a software regression, and should be
> reverted immediately, since:
>
> - they change current expected and implemented behavior to the cost of
>   users expectations and current use, with no prior discussion and
>   agreement on behavior changes 

Surprisingly, I do remember discussing about this change not so long
ago. Thus, I think the "no prior discussion" is fallacious.

To sum it up, in that discussion, I had suggested that allowing to fold
keywords wasn't a good idea since:

  1. Every element could get a #+name keyword, so everything would be
     eventually foldable (and think about the mess to unfold piece-wise,
     or everything).

  2. Drawers would have consequently no more purpose (properties drawer
     excepted).

> It has been proven that to achieve the required consistency set out by
> this thread /without/ breaking current expectations is less than
> straight-forward, and the changes should therefore be moved to
> EXPERIMENTAL - at least until the changes are proven not to break
> current expectations.
>
> Much of org-mode is currently inconsistent - and while it is highly
> desirable to improve on that situation, it should not become the major
> decision to change code at the cost of regression - org-mode is a very
> practical system used successfully by lots of (very appreciative)
> people, and while all users appreciate bug fixes / enhancements in the
> fast-moving repository that is org-mode (thanks Carsten), clear
> regression such as this should be avoided in master at all costs. 

As a side note, master isn't maint, which is the place for stability and
also bug fixing.  On the other hand, if a change doesn't hit master
branch, it won't get enough testing.

Now, there is no regression here: it has never been specified that
"#+results" keywords must allow to fold code. Folding is about
headlines, blocks, drawers and, optionally, items. Period.

Relying on "#+keyword" folding is a mistake. It would be better to
improve and correct drawers export (one of the things I address in the
new exporter). I talked in this thread about drawers
rehabilitation. Reverting this change will postpone this goal or, worse,
bury it.


Regards,

-- 
Nicolas Goaziou

Re: [BUG] Inconsistency in src block hiding

[Eric Schulte]

Nicolas Goaziou <n.goaziou@gmail.com> writes:

> Hello,
>
> Eric Schulte <eric.schulte@gmx.com> writes:
>
>> Well maybe we should roll back this change.
>
> Please don't.  _That_ would be a regression.
>
>> I'll wait to see if Nicolas has a solution which is both functional and
>> conforms to the Org-mode wide syntax norms.
>
> The problem comes from the current exporter, which isn't neutral about
> drawers, by default.
>
> I think that a temporary fix should be to:
>
>   1. Change default drawer formatting function to the following
>      (neutral):
>
>      #+begin_src emacs-lisp
>      (defun org-export-format-drawer (name contents)
>        "Export contents of a drawer as-is.
>      Property drawers are ignored."
>        (if (string= "PROPERTIES" name) "" contents))
>      #+end_src
>
>   2. Handle drawers earlier in org-export-preprocess-string function,
>      i.e. just before footnote handling, so its contents can benefit
>      from further modifications.
>
>   3. Allow drawers in export by default (excepted PROPERTIES
>      drawers). That is change `org-export-with-drawers' default value to
>      t.
>
> I'm packaging this in the following quick patch, highly untested.
>

Thanks for taking the time to collect these changes into a patch,
however I believe the changes you describe present /new/ behavior (e.g.,
new export semantics for drawers), rather than a bug repair.  For the
time being I am going to bring back results folding until an acceptable
alternative can be agreed upon and implemented.

Best,

>
>
> Regards,
>
> -- 
> Nicolas Goaziou
> From 0c15bf694f8051eb58fd131868059460f28f2e0d Mon Sep 17 00:00:00 2001
> From: Nicolas Goaziou <n.goaziou@gmail.com>
> Date: Wed, 18 Jan 2012 18:34:11 +0100
> Subject: [PATCH] org-exp: Set neutral behaviour towards drawers
>
> * lisp/org-exp.el (org-export-with-drawers): Change default value so
>   all drawers are exportable as a default.
> (org-export-preprocess-string): Handle drawers earlier in the
>   preprocess, so their contents can be modified further.
> (org-export-format-drawer): Change default behaviour contents of
>   drawers are exported as Org code.  As a special case, property
>   drawers are still ignored.
> ---
>  lisp/org-exp.el |   26 ++++++++------------------
>  1 files changed, 8 insertions(+), 18 deletions(-)
>
> diff --git a/lisp/org-exp.el b/lisp/org-exp.el
> index c7e1a94..d9b0a3e 100644
> --- a/lisp/org-exp.el
> +++ b/lisp/org-exp.el
> @@ -406,7 +406,7 @@ This option can also be set with the +OPTIONS line, e.g. \"tags:nil\"."
>  	  (const :tag "Not in TOC" not-in-toc)
>  	  (const :tag "On" t)))
>  
> -(defcustom org-export-with-drawers nil
> +(defcustom org-export-with-drawers t
>    "Non-nil means export with drawers like the property drawer.
>  When t, all drawers are exported.  This may also be a list of
>  drawer names to export."
> @@ -1156,6 +1156,10 @@ on this string to produce the exported version."
>        ;; Get rid of tasks, depending on configuration
>        (org-export-remove-tasks (plist-get parameters :tasks))
>  
> +      ;; Get rid of drawers
> +      (org-export-remove-or-extract-drawers
> +       drawers (plist-get parameters :drawers))
> +
>        ;; Prepare footnotes for export.  During that process, footnotes
>        ;; actually included in the exported part of the buffer go
>        ;; though some transformations:
> @@ -1209,10 +1213,6 @@ on this string to produce the exported version."
>        ;; Find HTML special classes for headlines
>        (org-export-remember-html-container-classes)
>  
> -      ;; Get rid of drawers
> -      (org-export-remove-or-extract-drawers
> -       drawers (plist-get parameters :drawers))
> -
>        ;; Get the correct stuff before the first headline
>        (when (plist-get parameters :skip-before-1st-heading)
>  	(goto-char (point-min))
> @@ -1500,19 +1500,9 @@ EXP-DRAWERS will be removed."
>  				  name content))
>  	   (insert content)))))))
>  
> -(defun org-export-format-drawer (name content)
> -  "Format the content of a drawer as a colon example."
> -  (if (string-match "[ \t]+\\'" content)
> -      (setq content (substring content (match-beginning 0))))
> -  (while (string-match "\\`[ \t]*\n" content)
> -    (setq content (substring content (match-end 0))))
> -  (setq content (org-remove-indentation content))
> -  (setq content (concat ": " (mapconcat 'identity
> -					(org-split-string content "\n")
> -					"\n: ")
> -			"\n"))
> -  (setq content (concat " : " (upcase name) "\n" content))
> -  (org-add-props content nil 'org-protected t))
> +(defun org-export-format-drawer (name contents)
> +  "Export contents of a drawer as-is."
> +  (if (string= "PROPERTIES" name) "" contents))
>  
>  (defun org-export-handle-export-tags (select-tags exclude-tags)
>    "Modify the buffer, honoring SELECT-TAGS and EXCLUDE-TAGS.

-- 
Eric Schulte
http://cs.unm.edu/~eschulte/

Re: [BUG] Inconsistency in src block hiding

[Nicolas Goaziou]

Hello,

Eric Schulte <eric.schulte@gmx.com> writes:

> Thanks for taking the time to collect these changes into a patch,
> however I believe the changes you describe present /new/ behavior (e.g.,
> new export semantics for drawers), rather than a bug repair.

I'd rather say that its intent is to fix an old bug: incomplete
specifications of drawers. Also, I don't add export semantics for
drawers: I remove any, by default.

Again, drawers allow to fold any part of an Org buffer without adding
semantics to its contents. It's a more general solution than
keywords. But you already know that.

> For the time being I am going to bring back results folding until an
> acceptable alternative can be agreed upon and implemented.

There is already an acceptable alternative. I sincerely hope that
reverting this won't make it impossible to be implemented later.


Regards,

-- 
Nicolas Goaziou

Re: [BUG] Inconsistency in src block hiding

[Eric Schulte]

Nicolas Goaziou <n.goaziou@gmail.com> writes:

> Hello,
>
> Eric Schulte <eric.schulte@gmx.com> writes:
>
>> Thanks for taking the time to collect these changes into a patch,
>> however I believe the changes you describe present /new/ behavior (e.g.,
>> new export semantics for drawers), rather than a bug repair.
>
> I'd rather say that its intent is to fix an old bug: incomplete
> specifications of drawers. Also, I don't add export semantics for
> drawers: I remove any, by default.
>

I apologize if I come across as argumentative, but I disagree with the
statement above.  The tag-line to the "Drawers" section in the manual is
"Tucking stuff away" which I think is often how drawers are used.
Changing the default drawer export behavior from "don't export" to "do
export" would be surprising, would break many existing work flows, and
would likely make drawers less useful.

In general I think the Org-mode specification is best defined by how
Org-mode is used and how it may be more easily and intuitively used in
the future.  Org-mode doesn't currently have a formal specification, and
I think that is a good thing.  Formal specification don't prevent bugs,
they just move them from the code to the spec.

Incidentally this is also why I consider the loss of result folding to
be a regression because it breaks existing usage patterns.

>
> Again, drawers allow to fold any part of an Org buffer without adding
> semantics to its contents. It's a more general solution than
> keywords. But you already know that.
>
>> For the time being I am going to bring back results folding until an
>> acceptable alternative can be agreed upon and implemented.
>
> There is already an acceptable alternative. I sincerely hope that
> reverting this won't make it impossible to be implemented later.
>

There is no rule against reverting a revision :), so we could always do
that if there is a consensus on list.

To my mind a better path moving forward would be to change the behavior
of the :RESULTS: drawer so that it is exported but *not* to change the
default drawer export behavior.  This way with a :wrap header argument
the code block results could be hidden with tab but would still be
exported.

   PRO: allows hiding code block results with tab, makes it clear where
        results begin and end, uses drawers for hiding which is what
        they are designed for, avoids the potential for hiding anything
        with a name

   CON: more syntactic weight around results, changes the existing
        default behavior, makes the "RESULTS" drawer a special type of
        drawer

There is likely a better option but this is the best that comes to mind.
Personally I am also content with the current behavior in which anything
under a #+name: may be hidden.

Cheers,

>
>
> Regards,

-- 
Eric Schulte
http://cs.unm.edu/~eschulte/

Re: [BUG] Inconsistency in src block hiding

[Leo Alekseyev]

> statement above.  The tag-line to the "Drawers" section in the manual is
> "Tucking stuff away" which I think is often how drawers are used.
> Changing the default drawer export behavior from "don't export" to "do
> export" would be surprising, would break many existing work flows, and
> would likely make drawers less useful.
>
> In general I think the Org-mode specification is best defined by how
> Org-mode is used and how it may be more easily and intuitively used in
> the future.  Org-mode doesn't currently have a formal specification, and
> I think that is a good thing.  Formal specification don't prevent bugs,
> they just move them from the code to the spec.

"Tucking stuff away" can mean different things to different users.
Personally, I have treated them purely as an organizational device for
supplementary information (I have :DETAILS: drawers all over my org
files).  The problem is that I may or may not want this supplementary
information in the export, and will really vary from case to case.
(Personally, in most cases, I do want to export that information --
but not always!)  Furthermore, assuming that I _do_ want drawers
exported, I may or may not want the drawer markup to be exported, i.e.
if drawers are used purely for organizing the presentation of
information, the drawer markup doesn't belong in the export.  On the
other hand, in certain cases one might want to denote the information
as supplementary, either by exporting drawer markup (or, more likely,
by putting drawer contents inside something like a <code> block).

If I were designing this behavior from scratch, I would allow for
maximum flexibility by
(1) creating e.g. org-drawers-to-export variable which could take on
the values nil (don't export), 'all (all drawers except :PROPERTIES:)
exported, or a list of drawer names to export
(2) introducing drawer flags that would control the export and display
behavior of individual drawers.  For instance, something like
:FOO: -vis -export
stuff...
:END:
would indicate that this drawer is to be kept unfolded and exported by default.
(3) controlling whether the drawer contents are separated out from the
rest of the contents by some markups (<hr>'s or  a <code> block)

I'm not sure how easy and/or practical any of this would be.  My
general philosophy is that if a behavior isn't specified, many
possible (sensible) behaviors should be considered and accommodated.

In that sense, hiding #+name: blocks is a good thing, because it
increases the amount of allowed sensible usage patterns.  If we were
to take it away, I think it would be necessary to compensate for this
by increasing the amount of allowed sensible usage patterns of the
drawers, kind of along the lines of what I described above.

--Leo

Re: [BUG] Inconsistency in src block hiding

[Jambunathan K]

Leo Alekseyev <dnquark@gmail.com> writes:

> "Tucking stuff away" can mean different things to different users.
> Personally, I have treated them purely as an organizational device for
> supplementary information (I have :DETAILS: drawers all over my org
> files).

(Out of Context)

Drawer contents => Marginalia[1]? or Presentation Notes?

It could also be useful as side-bar notes that have interesting trivia
like "Ever seen a subliminal arrow in the Fedex logo?"

Footnotes: 
[1]  http://en.wikipedia.org/wiki/Marginalia
-- 

Re: [BUG] Inconsistency in src block hiding

[Andreas Leha]

Eric Schulte <eric.schulte@gmx.com> writes:

[...]

>
> To my mind a better path moving forward would be to change the behavior
> of the :RESULTS: drawer so that it is exported but *not* to change the
> default drawer export behavior.  This way with a :wrap header argument
> the code block results could be hidden with tab but would still be
> exported.
>
>    PRO: allows hiding code block results with tab, makes it clear where
>         results begin and end, uses drawers for hiding which is what
>         they are designed for, avoids the potential for hiding anything
>         with a name
>
>    CON: more syntactic weight around results, changes the existing
>         default behavior, makes the "RESULTS" drawer a special type of
>         drawer
>
> There is likely a better option but this is the best that comes to mind.
> Personally I am also content with the current behavior in which anything
> under a #+name: may be hidden.
>

Coming late to this thread, I just want to say, that I am on the "PRO"
side for special results drawers.

The fact that org mode uses drawers for wrapping results is an
internal/technical choice that - I think - the user should not need to know.

In the current state, in case the user switches result wrapping on,
the results block disappears from the export, until the user has figured out,
that exporting drawers has to be enabled now (with all drawers being
exported...).

So, I think, there *is* a difference between drawers as used in other
places and drawers used for results, which are imposed onto the user by
org mode.  Thus, the CON just reflects this difference and is no real
"CON".

Just my two cents.

- Andreas

Re: [BUG] Inconsistency in src block hiding

[Nicolas Goaziou]

Hello,

Eric Schulte <eric.schulte@gmx.com> writes:

> Nicolas Goaziou <n.goaziou@gmail.com> writes:

> Changing the default drawer export behavior from "don't export" to "do
> export" would be surprising

Probably at first, but not for too long.

> would break many existing work flows

Not at all, since it's just a d:nil away from old behaviour.

Note that the current behavour is _not_ a d:t away from what I want to
implement. Indeed, even with d:t, you still need to change default
handler for drawers (org-export-format-drawer-function) as the default
one (org-export-format-drawer) will change drawer's contents into
a fixed-width area. For that task, an example block would have achieved
the same (hiding + verbatim).

> would likely make drawers less useful.

Certainly not, but I will that explain later.

> In general I think the Org-mode specification is best defined by how
> Org-mode is used and how it may be more easily and intuitively used in
> the future.  Org-mode doesn't currently have a formal specification, and
> I think that is a good thing.  Formal specification don't prevent bugs,
> they just move them from the code to the spec.

Then we may be running into problems, since my objective, with
org-element, is to provide a formal specification and normalization of
Org syntax. Along the way, some parts (hopefully few) in Org syntax will
need to be defined or re-defined.

Because only a very small number of persons can pretend to know every
part of Org syntax and even less (if any) to actually use them, we
definitely need guidelines to offer relevant future improvements.

Some current uses just don't fit in a global view: a recent example was
given by the caption syntax, which was heavily LaTeX oriented.

> To my mind a better path moving forward would be to change the behavior
> of the :RESULTS: drawer so that it is exported but *not* to change the
> default drawer export behavior.  This way with a :wrap header argument
> the code block results could be hidden with tab but would still be
> exported.
>
>    PRO: allows hiding code block results with tab, makes it clear where
>         results begin and end, uses drawers for hiding which is what
>         they are designed for, avoids the potential for hiding anything
>         with a name
>
>    CON: more syntactic weight around results, changes the existing
>         default behavior, makes the "RESULTS" drawer a special type of
>         drawer
>
> There is likely a better option but this is the best that comes to mind.
> Personally I am also content with the current behavior in which anything
> under a #+name: may be hidden.

Drawers should not add any semantics to the contents they are hiding.
They should be, as a default, neutral entities useful to hide stuff in
an Org buffer but certainly not to interfere with export.

1. There are already many ways to remove arbitrary contents from export.
   Among them, one can find the :noexport: tag, comment blocks...
   There's absolutely no need to add more.

2. They are better handled with regards to visibility cycling, which in
   not the case of single keywords. Allowing to hide every single Org
   syntax with a #+name: keyword could potentially be a mess.

3. By essence, drawers are better suited for hiding stuff than keywords,
   since they allow to group any number of elements, whereas keywords
   only apply to one keyword at a time.

4. Drawers are flexible. All major back-ends allow to configure
   behaviour of drawers with regards to export. You can always decide to
   keep :FOO: drawers transparent but remove any :BAR: drawer. It will
   be even simpler with the new export engine.

On the other hand, I don't think that adding another special type for
drawers is the way to go. In general, adding new syntax should be done
with parsimony. In this case, it unnecessarilyrestricts possibilities:
why enforce a special behaviour for :RESULTS: since (point 4) you can
choose it?

I still vote for neutral drawers.


Regards,

-- 
Nicolas Goaziou

Re: [BUG] Inconsistency in src block hiding

[Nicolas Goaziou]

Hello,

Eric Schulte <eric.schulte@gmx.com> writes:

> To my mind a better path moving forward would be to change the behavior
> of the :RESULTS: drawer so that it is exported but *not* to change the
> default drawer export behavior.  This way with a :wrap header argument
> the code block results could be hidden with tab but would still be
> exported.
>
>    PRO: allows hiding code block results with tab, makes it clear where
>         results begin and end, uses drawers for hiding which is what
>         they are designed for, avoids the potential for hiding anything
>         with a name
>
>    CON: more syntactic weight around results, changes the existing
>         default behavior, makes the "RESULTS" drawer a special type of
>         drawer

While implementing a recent patch about drawers insertion, I realized my
current view about drawers had a flaw. Indeed, while I had correctly put
aside properties drawers, which are very different from regular drawers,
I had overlooked special drawers like LOGBOOK and CLOCK.

Those drawers are different from regular drawers, since they are
attached to an headline (this is not the case of a RESULTS drawer), and
as such should be classified in another category.

Therefore, I suggest the following fixed behaviour with regards to
export for drawers:

- Properties drawers :: Still ignored in export, independently on "d"
     option or `org-export-with-drawers' value. Obviously, back-end can
     ignore this specification, but it should be followed by major ones.

- Special drawers :: Not exported by default. Though, their export can
     be configured with a new "sd" option item (i.e. "sd:t") or
     `org-export-with-special-drawers' variable. This category only
     includes LOGBOOK[1] and CLOCK drawers at the moment.

     If their export is activated, the special
     `org-backend-format-drawer-function' (i.e.
     `org-e-latex-format-drawer-function') can allow to tweak their
     output.
     
- Regular drawers :: Exported by default. "d:nil" turns that off. This
     category includes RESULTS drawers and every user created drawer.
     `org-backend-format-drawer-function' still can help tweaking their
     output. Thus, it can be used to filter out some types of drawers.

With that model, drawers will be able to fill a niche by allowing to
hide data in an Org buffer while still wanting to export it.

As a reminder, currently with "d:nil", drawers are redundant
with #+begin_comment blocks, and with "d:t" they are redundant
with #+begin_example blocks. "d:'(some names)" is just a mix of comment
and example blocks.

Note that it has the same advantages as your suggestion.

What do you think?


Regards,

[1] Or whatever the user specified in `org-log-into-drawer'.

-- 
Nicolas Goaziou

Re: [BUG] Inconsistency in src block hiding

[Nick Dokos]

Bernt Hansen <bernt@norang.ca> wrote:

> Nicolas Goaziou <n.goaziou@gmail.com> writes:
> 
> > Hello,
> >
> > Bernt Hansen <bernt@norang.ca> writes:
> >
> >> I tried :results wrap but that didn't work for me.  If I add RESULTS to
> >> my list of drawers then I can hide the block with TAB but I can't export
> >> my diagrams to HTML anymore which isn't very satisfying.
> >
> > Why can't you? Wouldn't it be related to drawers configuration
> > (org-export-with-drawers for example)?
> 
> Yes... but I don't think I can configure which drawers I get, and I
> don't want my LOGBOOK drawer with all my clock lines in my export.
> 

The following exports JUNK drawers but not GARBAGE drawers - isn't that exactly
what you want? or am I misunderstanding?

--8<---------------cut here---------------start------------->8---
#+DRAWERS: JUNK GARBAGE
#+OPTIONS: d:("JUNK")

* foo
hunoz
  :GARBAGE:
  what garbage is hidden here
  :END:
bar
  :JUNK:
  Hide some junk in here.
  :END:
baz
--8<---------------cut here---------------end--------------->8---

Nick

Re: [BUG] Inconsistency in src block hiding

[Bernt Hansen]

Nick Dokos <nicholas.dokos@hp.com> writes:

> Bernt Hansen <bernt@norang.ca> wrote:
>
>> Nicolas Goaziou <n.goaziou@gmail.com> writes:
>> 
>> > Hello,
>> >
>> > Bernt Hansen <bernt@norang.ca> writes:
>> >
>> >> I tried :results wrap but that didn't work for me.  If I add RESULTS to
>> >> my list of drawers then I can hide the block with TAB but I can't export
>> >> my diagrams to HTML anymore which isn't very satisfying.
>> >
>> > Why can't you? Wouldn't it be related to drawers configuration
>> > (org-export-with-drawers for example)?
>> 
>> Yes... but I don't think I can configure which drawers I get, and I
>> don't want my LOGBOOK drawer with all my clock lines in my export.
>> 
>
> The following exports JUNK drawers but not GARBAGE drawers - isn't that exactly
> what you want? or am I misunderstanding?
>
> #+DRAWERS: JUNK GARBAGE
> #+OPTIONS: d:("JUNK")
>
> * foo
> hunoz
>   :GARBAGE:
>   what garbage is hidden here
>   :END:
> bar
>   :JUNK:
>   Hide some junk in here.
>   :END:
> baz
>
> Nick

Well sort of... except it was a wrapped PlantUML graphic which in a
drawer doesn't display inline in my HTML export. :)

I admit I didn't try hard to fix it though.

-Bernt