Bug: property drawers within code blocks interfere [8.2.2 (8.2.2-elpa @ /home/tod/.emacs.d/.cask/24.3.50.1/elpa/org-20131108/)]

Bug: property drawers within code blocks interfere [8.2.2 (8.2.2-elpa @ /home/tod/.emacs.d/.cask/24.3.50.1/elpa/org-20131108/)]

[Tod Middlebrook]

Remember to cover the basics, that is, what you expected to happen and
what in fact did happen.  You don't know how to make a good report?  See

     http://orgmode.org/manual/Feedback.html#Feedback

Your bug report will be posted to the Org-mode mailing list.
------------------------------------------------------------------------
(org-version)"8.2.2"
c.f. http://osdir.com/ml/emacs-orgmode-gnu/2010-08/msg01248.html through
http://osdir.com/ml/emacs-orgmode-gnu/2010-08/msg01279.html, and http://blog.gmane.org/gmane.emacs.orgmode/page=4.

The bug below prevents me from easily using dependencies in org-dotemacs.

To reproduce,
start with this entry:

*** stuff for bug report
#+BEGIN_SRC emacs-lisp
  (setq org-capture-templates
        (quote
         (
          ("c" "Contacts" entry (file+headline "~/my-stuff/file.org" "Contacts")
           "* %^{Name: }
          :PROPERTIES: 
          :EMAIL: %^{Email}
          :PHONE: %^{Phone number}
          :END:
          %?
          "))))
#+END_SRC

Then do C-c C-x p EMAIL [RET] TestValue, and get the same block, with
the properties drawer folded. When expanded, there is:

*** stuff for bug report
#+BEGIN_SRC emacs-lisp
    (setq org-capture-templates
          (quote
           (
            ("c" "Contacts" entry (file+headline "~/my-stuff/file.org" "Contacts")
             "* %^{Name: }
            :PROPERTIES: 
  :EMAIL:    TestValue
            :PHONE: %^{Phone number}
            :END:
            %?
            "))))
#+END_SRC

Emacs  : GNU Emacs 24.3.50.1 (x86_64-unknown-linux-gnu, GTK+ Version 3.4.2)
 of 2013-11-10 on tod-ThinkPad-T430u
Package: Org-mode version 8.2.2 (8.2.2-elpa @ /home/tod/.emacs.d/.cask/24.3.50.1/elpa/org-20131108/)

Re: Bug: property drawers within code blocks interfere [8.2.2 (8.2.2-elpa @ /home/tod/.emacs.d/.cask/24.3.50.1/elpa/org-20131108/)]

[Bastien]

Hi Tod,

Tod Middlebrook <todmiddlebrook@gmail.com> writes:

> The bug below prevents me from easily using dependencies in org-dotemacs.
>
> To reproduce,
> start with this entry:
>
> *** stuff for bug report
> #+BEGIN_SRC emacs-lisp
>   (setq org-capture-templates
>         (quote
>          (
>           ("c" "Contacts" entry (file+headline "~/my-stuff/file.org" "Contacts")
>            "* %^{Name: }
>           :PROPERTIES: 
>           :EMAIL: %^{Email}
>           :PHONE: %^{Phone number}
>           :END:
>           %?
>           "))))
> #+END_SRC
>
> Then do C-c C-x p EMAIL [RET] TestValue, and get the same block, with
> the properties drawer folded. When expanded, there is:
>
> *** stuff for bug report
> #+BEGIN_SRC emacs-lisp
>     (setq org-capture-templates
>           (quote
>            (
>             ("c" "Contacts" entry (file+headline "~/my-stuff/file.org" "Contacts")
>              "* %^{Name: }
>             :PROPERTIES: 
>   :EMAIL:    TestValue
>             :PHONE: %^{Phone number}
>             :END:
>             %?
>             "))))
> #+END_SRC

I'm not sure to understand what the problem is exactly: if the problem
is that `C-x C-c p' works in the context of source code blocks, we can
easily fix it.  If the problems is that such properties are matched in
contexts where they should not, we need more information about when
you observe the wrong behavior, i.e. in what context do you see the
properties taken into account while you expect them to be ignored?

Thanks in advance for further details,

-- 
 Bastien

Re: Bug: property drawers within code blocks interfere [8.2.2 (8.2.2-elpa @ /home/tod/.emacs.d/.cask/24.3.50.1/elpa/org-20131108/)]

[Tod Middlebrook]

Bastien,
When there is no property drawer before the code block, C-c C-x p
affects the code block and either doesn't create a property drawer or it
leaves the 'real' property drawer unaffected.

An example for the second case, where org-set-property matches the code
block, despite an existing property drawer that happens to be below
it. The wrong "NAME" is matched, and "DEPENDS" isn't matched at
all. This could throw off for example org-dotemacs, where it would
tangle code blocks in the wrong order without the DEPENDS matching.
* stuff for bug report
#+BEGIN_SRC emacs-lisp
  
  (setq org-capture-templates
        (quote
         (
          ("c" "Contacts" entry (file+headline "~/my-stuff/file.org" "Contacts")
           "* %^{Name: }
          :PROPERTIES: 
          :EMAIL: %^{Email}
          :PHONE: %^{Phone number}
          :NAME: %^{Full Name}
          :END:
          %?
          "))))
#+END_SRC
:PROPERTIES:
:NAME: org-capture-templates
:DEPENDS: org
:END: 

Thanks,
Tod

bzg@gnu.org writes:

> Hi Tod,
>
> Tod Middlebrook <todmiddlebrook@gmail.com> writes:
>
>> The bug below prevents me from easily using dependencies in org-dotemacs.
>>
>> To reproduce,
>> start with this entry:
>>
>> *** stuff for bug report
>> #+BEGIN_SRC emacs-lisp
>>   (setq org-capture-templates
>>         (quote
>>          (
>>           ("c" "Contacts" entry (file+headline "~/my-stuff/file.org" "Contacts")
>>            "* %^{Name: }
>>           :PROPERTIES: 
>>           :EMAIL: %^{Email}
>>           :PHONE: %^{Phone number}
>>           :END:
>>           %?
>>           "))))
>> #+END_SRC
>>
>> Then do C-c C-x p EMAIL [RET] TestValue, and get the same block, with
>> the properties drawer folded. When expanded, there is:
>>
>> *** stuff for bug report
>> #+BEGIN_SRC emacs-lisp
>>     (setq org-capture-templates
>>           (quote
>>            (
>>             ("c" "Contacts" entry (file+headline "~/my-stuff/file.org" "Contacts")
>>              "* %^{Name: }
>>             :PROPERTIES: 
>>   :EMAIL:    TestValue
>>             :PHONE: %^{Phone number}
>>             :END:
>>             %?
>>             "))))
>> #+END_SRC
>
> I'm not sure to understand what the problem is exactly: if the problem
> is that `C-x C-c p' works in the context of source code blocks, we can
> easily fix it.  If the problems is that such properties are matched in
> contexts where they should not, we need more information about when
> you observe the wrong behavior, i.e. in what context do you see the
> properties taken into account while you expect them to be ignored?
>
> Thanks in advance for further details,

Re: Bug: property drawers within code blocks interfere [8.2.2 (8.2.2-elpa @ /home/tod/.emacs.d/.cask/24.3.50.1/elpa/org-20131108/)]

[Bastien]

Hi Tod,

Tod Middlebrook <whatifitsnot@gmail.com> writes:

> * stuff for bug report
> #+BEGIN_SRC emacs-lisp
>   
>   (setq org-capture-templates
>         (quote
>          (
>           ("c" "Contacts" entry (file+headline "~/my-stuff/file.org" "Contacts")
>            "* %^{Name: }
>           :PROPERTIES: 
>           :EMAIL: %^{Email}
>           :PHONE: %^{Phone number}
>           :NAME: %^{Full Name}
>           :END:
>           %?
>           "))))
> #+END_SRC
> :PROPERTIES:
> :NAME: org-capture-templates
> :DEPENDS: org
> :END: 

Simply put property drawers right after the headline, or just don't
put source-code-blocks-containing-property-drawers between headline
and real property drawers.

-- 
 Bastien

Re: Bug: property drawers within code blocks interfere [8.2.2 (8.2.2-elpa @ /home/tod/.emacs.d/.cask/24.3.50.1/elpa/org-20131108/)]

[Tod Middlebrook]

Bastien, 

I apologize for the mishaps of sending from the wrong address that time
and for somehow leaving off the commentary that I thought was sent with
the code block. 

Is it a trivial fix to alter drawer and possibly other element regexps
to never match anything inside of a code block, or should I add it to
http://orgmode.org/tmp/worg/org-issues.html ?

Tod

bzg@altern.org writes:

> Hi Tod,
>
> Tod Middlebrook <whatifitsnot@gmail.com> writes:
>
>> * stuff for bug report
>> #+BEGIN_SRC emacs-lisp
>>   
>>   (setq org-capture-templates
>>         (quote
>>          (
>>           ("c" "Contacts" entry (file+headline "~/my-stuff/file.org" "Contacts")
>>            "* %^{Name: }
>>           :PROPERTIES: 
>>           :EMAIL: %^{Email}
>>           :PHONE: %^{Phone number}
>>           :NAME: %^{Full Name}
>>           :END:
>>           %?
>>           "))))
>> #+END_SRC
>> :PROPERTIES:
>> :NAME: org-capture-templates
>> :DEPENDS: org
>> :END: 
>
> Simply put property drawers right after the headline, or just don't
> put source-code-blocks-containing-property-drawers between headline
> and real property drawers.

Re: Bug: property drawers within code blocks interfere [8.2.2 (8.2.2-elpa @ /home/tod/.emacs.d/.cask/24.3.50.1/elpa/org-20131108/)]

[Bastien]

Hi Tod,

Tod Middlebrook <todmiddlebrook@gmail.com> writes:

> Is it a trivial fix to alter drawer and possibly other element regexps
> to never match anything inside of a code block, or should I add it to
> http://orgmode.org/tmp/worg/org-issues.html ?

It is not trivial, but your best chance is to describe the bug on this
mailing list so that someone is convince (1) it's not just some weird
use-case and (2) it's worth a fix.

2 cents,

-- 
 Bastien

Re: Bug: property drawers within code blocks interfere [8.2.2 (8.2.2-elpa @ /home/tod/.emacs.d/.cask/24.3.50.1/elpa/org-20131108/)]

[Aaron Ecay]

Hi Tod and Bastien,

The property drawer after the code block is a red herring: the following
file (with no real property drawer at all) misbehaves on property
setting and getting functions, with the fake properties in the code
block behaving as though they pertained to the headline (as can be
confirmed by M-: (org-entry-get nil "EMAIL")):

* stuff for bug report
#+BEGIN_SRC emacs-lisp
  (setq i-am-a-string
        "
          :PROPERTIES: 
          :EMAIL: %^{Email}
          :PHONE: %^{Phone number}
          :END:
        ")
#+END_SRC

I think this problem of unselective regex searches will also affect uses
of e.g. org-scheduled-time-regexp

This seems like an excellent use case for the parser: basically a bunch
of uses of org-*-regexp and org-re-property need to be augmented with
a check like:
(not (memq (org-element-type (org-element-at-point)) '(src-block example-block ...)))

A better alternative might be to use the parser to find the property
drawer in the first place (instead of a regex).  Either way, it seems
like the best strategy might be to fix all uses of these problem
variables at once, which is a big undertaking.

-- 
Aaron Ecay

Re: Bug: property drawers within code blocks interfere [8.2.2 (8.2.2-elpa @ /home/tod/.emacs.d/.cask/24.3.50.1/elpa/org-20131108/)]

[Bastien]

Hi Aaron and Tod,

Aaron Ecay <aaronecay@gmail.com> writes:

> This seems like an excellent use case for the parser: basically a bunch
> of uses of org-*-regexp and org-re-property need to be augmented with
> a check like:
> (not (memq (org-element-type (org-element-at-point)) '(src-block example-block ...)))
>
> A better alternative might be to use the parser to find the property
> drawer in the first place (instead of a regex).  Either way, it seems
> like the best strategy might be to fix all uses of these problem
> variables at once, which is a big undertaking.

Also don't forget the cost in terms of speed.  It's fine to fix the
behavior of Org for such cases, but those cases are rare, and could
be explicitely prevented.  If the general fix does not slow down the
parsing too much, then I'm all for it.  Nicolas might have better
insight here than me.

2 cents,

-- 
 Bastien

Re: Bug: property drawers within code blocks interfere [8.2.2 (8.2.2-elpa @ /home/tod/.emacs.d/.cask/24.3.50.1/elpa/org-20131108/)]

[Sebastien Vauban]

Hello Aaron and Bastien,

Aaron Ecay wrote:
> The property drawer after the code block is a red herring: the following
> file (with no real property drawer at all) misbehaves on property
> setting and getting functions, with the fake properties in the code
> block behaving as though they pertained to the headline (as can be
> confirmed by M-: (org-entry-get nil "EMAIL")):
>
> * stuff for bug report
> #+BEGIN_SRC emacs-lisp
>   (setq i-am-a-string
>         "
>           :PROPERTIES: 
>           :EMAIL: %^{Email}
>           :PHONE: %^{Phone number}
>           :END:
>         ")
> #+END_SRC
>
> I think this problem of unselective regex searches will also affect uses
> of e.g. org-scheduled-time-regexp

To come back on what you suspect, I must admit I still have the problem
described in the thread http://comments.gmane.org/gmane.emacs.orgmode/60959,
that is:

--8<---------------cut here---------------start------------->8---
* Some explanation of Org mode timestamps...

#+begin_src org
,SCHEDULED: <2013-11-21 Thu +1w/12d>
#+end_src

For ranges, you can try:

#+begin_src org
,<2013-11-20 Wed 15:00-17:00 +1w>.
#+end_src
--8<---------------cut here---------------end--------------->8---

both "events" will appear in my agenda (for next Wed/Thu).

Best regards,
  Seb

-- 
Sebastien Vauban

Re: Bug: property drawers within code blocks interfere [8.2.2 (8.2.2-elpa @ /home/tod/.emacs.d/.cask/24.3.50.1/elpa/org-20131108/)]

[Nicolas Goaziou]

Bastien <bzg@gnu.org> writes:

> Hi Aaron and Tod,
>
> Aaron Ecay <aaronecay@gmail.com> writes:
>
>> This seems like an excellent use case for the parser: basically a bunch
>> of uses of org-*-regexp and org-re-property need to be augmented with
>> a check like:
>> (not (memq (org-element-type (org-element-at-point)) '(src-block example-block ...)))
>>
>> A better alternative might be to use the parser to find the property
>> drawer in the first place (instead of a regex).  Either way, it seems
>> like the best strategy might be to fix all uses of these problem
>> variables at once, which is a big undertaking.
>
> Also don't forget the cost in terms of speed.  It's fine to fix the
> behavior of Org for such cases, but those cases are rare, and could
> be explicitely prevented.  If the general fix does not slow down the
> parsing too much, then I'm all for it.  Nicolas might have better
> insight here than me.

With newly introduced parser cache, the cost is amortized when using
`org-element-at-point' and `org-element-context'. With a little care,
the overhead is negligible. Therefore, these functions should be used
whenever possible.

I'm slowly working on introducing these functions in basic parts of Org
(e.g. fontification, indentation). Then I will target other functions
not using them yet. There's a lot to do. Help is welcome. IMO, at the
bare minimum, new or rewritten functions should use them.

For example, a way to correctly find the property drawer associated to
the current section is to use something like the following:

  (save-excursion
    (outline-previous-heading)
    (unless (bobp)
      (let ((limit (save-excursion (outline-next-heading) (point))))
        (catch 'exit
          (while (re-search-forward org-property-start-re limit t)
            (let ((drawer (org-element-at-point)))
              (when (eq (org-element-type drawer) 'property-drawer)
                ...
                (throw 'exit ...))))))))


Regards,

-- 
Nicolas Goaziou

Re: Bug: property drawers within code blocks interfere [8.2.2 (8.2.2-elpa @ /home/tod/.emacs.d/.cask/24.3.50.1/elpa/org-20131108/)]

[Bastien]

Nicolas Goaziou <n.goaziou@gmail.com> writes:

> I'm slowly working on introducing these functions in basic parts of Org
> (e.g. fontification, indentation). Then I will target other functions
> not using them yet. There's a lot to do. Help is welcome. IMO, at the
> bare minimum, new or rewritten functions should use them.

Well, I spent all my "time credits" with the merge of Org into Emacs,
so I already know I won't be able to help in the next few weeks, but I
hope some other core contributors will -- thanks a lot for undertaking
this changes.

-- 
 Bastien