bug in org-habits

bug in org-habits

[Mark A. Hershberger]

org-habit.el is sensitive to the ordering PROPERTIES and SCHEDULED
and expects them /in only/ the following order:

 * TODO habit name
 SCHEDULED: <2015-11-03 Tue 07:00 .+1d>
 :PROPERTIES:
 :LAST_REPEAT: [2015-11-02 Mon 07:54]
 :STYLE: habit
 :END:

Any other ordering, or doubling of PROPERTIES (e.g. STYLE in one, LAST
REPEAT in another) and the habit will show up as a simple scheduled item
and not a habit in the agenda view.

org-version: 8.3.2
emacs-version: 24.4.1

Let me know if I need to provide any other information.

Thanks!

Mark.


-- 
http://hexmode.com/

Love every man in spite of his falling into sin. Never mind the sins,
but remember that the foundation of the man is the same - the image of
God.                                      -- St. John of Kronstadt

Re: bug in org-habits

[Marco Wahl]

"Mark A. Hershberger" <mah@everybody.org> writes:

> org-habit.el is sensitive to the ordering PROPERTIES and SCHEDULED
> and expects them /in only/ the following order:
>
>  * TODO habit name
>  SCHEDULED: <2015-11-03 Tue 07:00 .+1d>
>  :PROPERTIES:
>  :LAST_REPEAT: [2015-11-02 Mon 07:54]
>  :STYLE: habit
>  :END:
>
> Any other ordering, or doubling of PROPERTIES (e.g. STYLE in one, LAST
> REPEAT in another) and the habit will show up as a simple scheduled item
> and not a habit in the agenda view.
>
> org-version: 8.3.2

Actually there has been introduced a constraint on the ordering planning
lines and property drawers in 8.3.  See http://orgmode.org/Changes.html.

This at least invalidates to use PROPERTIES before SCHEDULED afaics.


HTH
-- 
Marco Wahl
GPG: 0x49010A040A3AE6F2

Re: bug in org-habits

[Puneeth Chaganti]

On Tue, Nov 3, 2015 at 3:26 PM, Marco Wahl <marcowahlsoft@gmail.com> wrote:
>
>
> Actually there has been introduced a constraint on the ordering planning
> lines and property drawers in 8.3.  See http://orgmode.org/Changes.html.
>
> This at least invalidates to use PROPERTIES before SCHEDULED afaics.


Yes, that is correct and you can use the `org-repair-property-drawers`
utility function provided to fix your org trees.

-- 
Puneeth

Re: bug in org-habits

[John Wiegley]

>>>>> Puneeth Chaganti <punchagan@gmail.com> writes:

>> Actually there has been introduced a constraint on the ordering planning
>> lines and property drawers in 8.3. See http://orgmode.org/Changes.html.>
>> This at least invalidates to use PROPERTIES before SCHEDULED afaics.

> Yes, that is correct and you can use the `org-repair-property-drawers`
> utility function provided to fix your org trees.

I would very much like to see this constraint removed from 8.3. I have always
preferred having SCHEDULED before PROPERTIES, as this is how all my Org files
are arranged.

In fact, this one aspect of 8.3 has kept me at 8.2. I do not understand the
need for such an abitrary restriction.

John

Re: bug in org-habits

[Marco Wahl]

John Wiegley <jwiegley@gmail.com> writes:

>>>>>> Puneeth Chaganti <punchagan@gmail.com> writes:
>
>>> Actually there has been introduced a constraint on the ordering planning
>>> lines and property drawers in 8.3. See http://orgmode.org/Changes.html.>
>>> This at least invalidates to use PROPERTIES before SCHEDULED afaics.
>
>> Yes, that is correct and you can use the `org-repair-property-drawers`
>> utility function provided to fix your org trees.
>
> I would very much like to see this constraint removed from 8.3. I have always
> preferred having SCHEDULED before PROPERTIES, as this is how all my Org files
> are arranged.

Actually what you describe _is_ the expected order for the 8.3 files.

> In fact, this one aspect of 8.3 has kept me at 8.2. I do not understand the
> need for such an abitrary restriction.

So this constraint on the order should not hold you back to move on to
8.3.

I can report that in my personal org practice this constraint never got
in my way.  Further I can't imagine a use case when the user can benefit
of a SCHEDULE line below a property drawer, but this might be a bit
personal.

I can say nothing about the technical view of having that constraint.


Best regards,
-- 
Marco Wahl
GPG: 0x49010A040A3AE6F2

Re: bug in org-habits

[Stelian Iancu]

On 03/11/15 14:46, Marco Wahl wrote:
> John Wiegley <jwiegley@gmail.com> writes:
>
>>>>>>> Puneeth Chaganti <punchagan@gmail.com> writes:
>>
>>>> Actually there has been introduced a constraint on the ordering planning
>>>> lines and property drawers in 8.3. See http://orgmode.org/Changes.html.>
>>>> This at least invalidates to use PROPERTIES before SCHEDULED afaics.
>>
>>> Yes, that is correct and you can use the `org-repair-property-drawers`
>>> utility function provided to fix your org trees.
>>
>> I would very much like to see this constraint removed from 8.3. I have always
>> preferred having SCHEDULED before PROPERTIES, as this is how all my Org files
>> are arranged.
>
> Actually what you describe _is_ the expected order for the 8.3 files.

My issue is a bit different.

I have an org file with calendar appointments. I also have attachments 
to the appointments. The attachment appears in a PROPERTIES drawer.

Now if I have the timestamp (plain one) before the drawer, I cannot open 
the attachment. Pressing C-c C-a o just inserts another PROPERTIES 
drawer (with another ID) above the timestamp.

If, however, I move the timestamp after the PROPERTIES drawer, I can 
successfully open the attachment and the second drawer doesn't appear.

Also, when the second drawer appears, a new empty folder is created in 
the data directory of my org directory.

IMHO this is a bug, as at the moment I have to have the timestamp after 
the drawer.

Org version is Org-mode version 8.3.2 (release_8.3.2-253-g9b5757 @ 
/Users/si/elisp/org-mode/lisp/)

Re: bug in org-habits

[Nicolas Goaziou]

Hello,

Stelian Iancu <stelian@iancu.ch> writes:

> I have an org file with calendar appointments. I also have attachments
> to the appointments. The attachment appears in a PROPERTIES drawer.
>
> Now if I have the timestamp (plain one) before the drawer, I cannot
> open the attachment. Pressing C-c C-a o just inserts another
> PROPERTIES drawer (with another ID) above the timestamp.
>
> If, however, I move the timestamp after the PROPERTIES drawer, I can
> successfully open the attachment and the second drawer doesn't appear.
>
> Also, when the second drawer appears, a new empty folder is created in
> the data directory of my org directory.
>
> IMHO this is a bug, as at the moment I have to have the timestamp
> after the drawer.

This is not a bug. The order is HEADLINE (SCHEDULED PROPERTIES).

If you insert a timestamp between SCHEDULED and PROPERTIES, the latter
is no longer recognized as a PROPERTIES DRAWER (it becomes, in fact,
a regular drawer which happens to be named "PROPERTIES") and you lose
any information attached to it.

You can get this information back by putting the PROPERTIES drawer (the
special one) at its expected location.

Regards,

-- 
Nicolas Goaziou

Re: bug in org-habits

[John Wiegley]

>>>>> Nicolas Goaziou <mail@nicolasgoaziou.fr> writes:

> This is not a bug. The order is HEADLINE (SCHEDULED PROPERTIES).

Ah, I had misspoken. It's not SCHEDULE-before-PROPERTIES that has broken for
me, it is the inability to have PROPERTIES at the very end of the entry.

Why must the properties drawer appear before the entry's body?

John

Re: bug in org-habits

[Nicolas Goaziou]

Hello,

John Wiegley <jwiegley@gmail.com> writes:

>>>>>> Nicolas Goaziou <mail@nicolasgoaziou.fr> writes:
>
>> This is not a bug. The order is HEADLINE (SCHEDULED PROPERTIES).
>
> Ah, I had misspoken. It's not SCHEDULE-before-PROPERTIES that has broken for
> me, it is the inability to have PROPERTIES at the very end of the entry.
>
> Why must the properties drawer appear before the entry's body?

IIRC, you already asked the question. 

It is for efficiency reasons. Properties are an important feature in
Org, letting them anywhere in a potentially long entry doesn't scale
well.


Regards,

-- 
Nicolas Goaziou

Re: bug in org-habits

[John Wiegley]

>>>>> Nicolas Goaziou <mail@nicolasgoaziou.fr> writes:

> It is for efficiency reasons. Properties are an important feature in Org,
> letting them anywhere in a potentially long entry doesn't scale well.

Since it scales for my use case, can I please have a customization variable to
relax this restriction? I prefer PROPERTIES drawers at the end, and would
prefer not to trade something I want for something that doesn't affect me.

Thanks, John

Re: bug in org-habits

[Nicolas Goaziou]

John Wiegley <jwiegley@gmail.com> writes:

> Since it scales for my use case, can I please have a customization variable to
> relax this restriction?

I'd rather not have syntax too much customizable, for portability, ease
of maintenance, too. There are already too many mistakes in that area.

Also, please note that previous behaviour didn't handle some
pathological cases (e.g., property drawer in a comment block, or past an
inlinetask). Fixing that properly, i.e. parsing the section all the way
down to the drawer, would probably have had an impact in your use case.

The current behaviour is technically much more reliable.

> I prefer PROPERTIES drawers at the end, and would prefer not to trade
> something I want for something that doesn't affect me.

If you need to end your entry with a drawer, couldn't you put any of
them there? You can even have one there named "PROPERTIES".

Regards,

Re: bug in org-habits

[John Wiegley]

>>>>> Nicolas Goaziou <mail@nicolasgoaziou.fr> writes:

> I'd rather not have syntax too much customizable, for portability, ease of
> maintenance, too. There are already too many mistakes in that area.

Thanks for discussing this with me, Nicolas. I appreciate there may be
technical complexities involved. Could we special-case allow PROPERTIES to be
the *very last thing* in an entry? I don't need it to float anywhere else. I
just like it to be at the end.

> If you need to end your entry with a drawer, couldn't you put any of them
> there? You can even have one there named "PROPERTIES".

But it wouldn't be the true PROPERTIES, would it? Most of my entries look like
this (and I have many thousands of them):

** TODO Update auto insurance cards                             :Home:ATTACH:
SCHEDULED: <2016-03-11 Fri +6m>
- State "DONE"       from "TODO"       [2015-09-11 Fri 11:28]
- State "WAITING"    from "TODO"       [2015-07-22 Wed 19:48]
- State "DONE"       from "STARTED"    [2014-12-13 Sat 15:56]
- State "DONE"       from "TODO"       [2014-03-31 Mon 03:11]
- State "CANCELED"   from "TODO"       [2013-12-11 Wed 19:23]
- State "CANCELED"   from "TODO"       [2013-05-21 Tue 12:30]
- State "DONE"       from "TODO"       [2012-05-17 Thu 22:24]
- State "CANCELED"   from "TODO"       [2011-04-23 Sat 22:04]
- State "TODO"       from "DONE"       [2010-10-23 Sat 21:54]
- State "DONE"       from "TODO"       [2010-10-08 Fri 14:26]
:PROPERTIES:
:ID:       B1F3D3F6-9F39-4899-9AF8-93E019E7C6BB
:CREATED:  [2010-05-08 Sat 17:34]
:LAST_REPEAT: [2015-09-11 Fri 11:28]
:Attachments: AutoInsuranceIdCards.pdf
:END:

I suppose I've just become familiar with seeing :PROPERTIES: as the "period"
at the end of the entry, and I'll like to keep it there if we can devise a
technical answer that does not make life more difficult for you.

John

Re: bug in org-habits

[Achim Gratz]

John Wiegley writes:
> Thanks for discussing this with me, Nicolas. I appreciate there may be
> technical complexities involved. Could we special-case allow PROPERTIES to be
> the *very last thing* in an entry? I don't need it to float anywhere else. I
> just like it to be at the end.

Well, that's precisely the thing that doesn't scale and that Nicolas
wanted to avoid.  Putting the properties at the beginning of an entry
makes the search pretty much constant time and if you find something
else at the start of the entry then you know there aren't any and can go
on (this is pretty important for making sure property inheritance works
correctly, among other things).  If you could put them _anywhere_ else,
you'd have to keep searching until you either find them or you've
exhausted the span of the entry.


Regards,
Achim.
-- 
+<[Q+ Matrix-12 WAVE#46+305 Neuron microQkb Andromeda XTk Blofeld]>+

SD adaptations for KORG EX-800 and Poly-800MkII V0.9:
http://Synth.Stromeko.net/Downloads.html#KorgSDada

Re: bug in org-habits

[John Wiegley]

>>>>> Achim Gratz <Stromeko@nexgo.de> writes:

> Well, that's precisely the thing that doesn't scale and that Nicolas wanted
> to avoid. Putting the properties at the beginning of an entry makes the
> search pretty much constant time and if you find something else at the start
> of the entry then you know there aren't any and can go on (this is pretty
> important for making sure property inheritance works correctly, among other
> things). If you could put them _anywhere_ else, you'd have to keep searching
> until you either find them or you've exhausted the span of the entry.

As a user, I'm willing to pay that cost. Also, I never have other property
drawers. If it were just doing (re-search-forward "^:PROPERTIES:$"
(end-of-entry)), then it wouldn't matter where the properties drawer was, so
long as it's understood that the search may be wrong if the user has such a
string appearing elsewhere.

John

Re: bug in org-habits

[Jonathan Leech-Pepin]

On November 3, 2015 4:31:11 PM EST, Achim Gratz <Stromeko@nexgo.de> wrote:
>John Wiegley writes:
>> Thanks for discussing this with me, Nicolas. I appreciate there may
>be
>> technical complexities involved. Could we special-case allow
>PROPERTIES to be
>> the *very last thing* in an entry? I don't need it to float anywhere
>else. I
>> just like it to be at the end.
>
>Well, that's precisely the thing that doesn't scale and that Nicolas
>wanted to avoid.  Putting the properties at the beginning of an entry
>makes the search pretty much constant time and if you find something
>else at the start of the entry then you know there aren't any and can
>go
>on (this is pretty important for making sure property inheritance works
>correctly, among other things).  If you could put them _anywhere_ else,
>you'd have to keep searching until you either find them or you've
>exhausted the span of the entry.

Wouldn't last item in entry scale without issues? Find end of headline (start of next or end of buffer) and search backwards. If first element from end is a property drawer you have it, otherwise you still know there is none.

Jon

>
>Regards,
>Achim.

-- 
Sent from my Android device with K-9 Mail. Please excuse my brevity.

Re: bug in org-habits

[John Wiegley]

>>>>> Jonathan Leech-Pepin <jonathan.leechpepin@gmail.com> writes:

> Wouldn't last item in entry scale without issues? Find end of headline
> (start of next or end of buffer) and search backwards. If first element from
> end is a property drawer you have it, otherwise you still know there is
> none.

That sounds even better than what I mentioned about re-search-forward, and
O(1) time complexity as well.

John

Re: bug in org-habits

[Achim Gratz]

John Wiegley writes:
>>>>>> Jonathan Leech-Pepin <jonathan.leechpepin@gmail.com> writes:
>> Wouldn't last item in entry scale without issues? Find end of headline
>> (start of next or end of buffer) and search backwards. If first element from
>> end is a property drawer you have it, otherwise you still know there is
>> none.
>
> That sounds even better than what I mentioned about re-search-forward, and
> O(1) time complexity as well.

I don't think so.  Search for end of entry can be complex in itself and
you would never know if the properties you find by looking back aren't
belonging to an entry one level down unless you scanned the whole span
again.  Also, properties can be any size and you have to search for the
beginning of the drawer.


Regards,
Achim.
-- 
+<[Q+ Matrix-12 WAVE#46+305 Neuron microQkb Andromeda XTk Blofeld]>+

Factory and User Sound Singles for Waldorf Q+, Q and microQ:
http://Synth.Stromeko.net/Downloads.html#WaldorfSounds

Re: bug in org-habits

[John Wiegley]

>>>>> Achim Gratz <Stromeko@nexgo.de> writes:

> I don't think so. Search for end of entry can be complex in itself and you
> would never know if the properties you find by looking back aren't belonging
> to an entry one level down unless you scanned the whole span again. Also,
> properties can be any size and you have to search for the beginning of the
> drawer.

Good point.  Still, the option I'd still choose would be:

    org-allow-floating-properties

      If set to `non-nil', the :PROPERTIES: drawer may begin anywhere within
      an entry. Note that this will slow down many operations due to the
      additional scan needed, and may lead to incorrect result if similar text
      is found elsewhere within an entry, which is the recommended and default
      value for this variable is nil.

John

Re: bug in org-habits

[Nicolas Goaziou]

John Wiegley <jwiegley@gmail.com> writes:

> Thanks for discussing this with me, Nicolas. I appreciate there may be
> technical complexities involved. Could we special-case allow PROPERTIES to be
> the *very last thing* in an entry? I don't need it to float anywhere else. I
> just like it to be at the end.

I think it's a false good idea.

As a matter of fact, going to the end of an entry is not negligible,
because of inlinetasks. Also, it is not really O(1) since it depends on
the size of the entry. To get an idea, on my computer, moving past a 500
lines entry takes around 0.001s. I can imagine slower computers, or
larger entries, neither being unheard of, having a slightly noticeable
delay.

Also, supporting both locations means that users will pay the full price
in entries without a property drawer, even if they chose the fast path,
i.e., drawer at the beginning of the entry, in the first place. This
kind of defeats some of the advantages of the current state.

There is also a meaner problem. Unlike to beginning of entry, end of
entry is a moving target. Org sometimes automatically adds stuff there
(e.g., footnotes when `org-footnote-section' is nil). So do other parts
of Emacs (file local variables come to mind). Preventing insertions in
this area can be tedious, if possible at all. 

OTOH, the mechanism protecting the beginning of the entry, i.e,
`org-end-of-meta-data', is already implemented since planning lines
require it anyway.

Again, speed _was_ an issue, as testified by, e.g.,
org-use-property-inheritance's docstring. Current solution is both
robust and fast, and is well worth the high price of an incompatible
change, considering its central place in Org.

>> If you need to end your entry with a drawer, couldn't you put any of them
>> there? You can even have one there named "PROPERTIES".
>
> But it wouldn't be the true PROPERTIES, would it? Most of my entries look like
> this (and I have many thousands of them):

It wouldn't, of course. But at least you would get your "period" at the
end of the entry. It doesn't matter much if it is empty since its
contents are hidden anyway.

> ** TODO Update auto insurance cards                             :Home:ATTACH:
> SCHEDULED: <2016-03-11 Fri +6m>
> - State "DONE"       from "TODO"       [2015-09-11 Fri 11:28]
> - State "WAITING"    from "TODO"       [2015-07-22 Wed 19:48]
> - State "DONE"       from "STARTED"    [2014-12-13 Sat 15:56]
> - State "DONE"       from "TODO"       [2014-03-31 Mon 03:11]
> - State "CANCELED"   from "TODO"       [2013-12-11 Wed 19:23]
> - State "CANCELED"   from "TODO"       [2013-05-21 Tue 12:30]
> - State "DONE"       from "TODO"       [2012-05-17 Thu 22:24]
> - State "CANCELED"   from "TODO"       [2011-04-23 Sat 22:04]
> - State "TODO"       from "DONE"       [2010-10-23 Sat 21:54]
> - State "DONE"       from "TODO"       [2010-10-08 Fri 14:26]
> :PROPERTIES:
> :ID:       B1F3D3F6-9F39-4899-9AF8-93E019E7C6BB
> :CREATED:  [2010-05-08 Sat 17:34]
> :LAST_REPEAT: [2015-09-11 Fri 11:28]
> :Attachments: AutoInsuranceIdCards.pdf
> :END:

The following is not ugly either

  ** TODO Update auto insurance cards                             :Home:ATTACH:
  SCHEDULED: <2016-03-11 Fri +6m>
  :PROPERTIES:
  :ID:       B1F3D3F6-9F39-4899-9AF8-93E019E7C6BB
  :CREATED:  [2010-05-08 Sat 17:34]
  :LAST_REPEAT: [2015-09-11 Fri 11:28]
  :Attachments: AutoInsuranceIdCards.pdf
  :END:
  - State "DONE"       from "TODO"       [2015-09-11 Fri 11:28]
  - State "WAITING"    from "TODO"       [2015-07-22 Wed 19:48]
  - State "DONE"       from "STARTED"    [2014-12-13 Sat 15:56]
  - State "DONE"       from "TODO"       [2014-03-31 Mon 03:11]
  - State "CANCELED"   from "TODO"       [2013-12-11 Wed 19:23]
  - State "CANCELED"   from "TODO"       [2013-05-21 Tue 12:30]
  - State "DONE"       from "TODO"       [2012-05-17 Thu 22:24]
  - State "CANCELED"   from "TODO"       [2011-04-23 Sat 22:04]
  - State "TODO"       from "DONE"       [2010-10-23 Sat 21:54]
  - State "DONE"       from "TODO"       [2010-10-08 Fri 14:26]

even more so considering it really appears as 

  ** TODO Update auto insurance cards                             :Home:ATTACH:
  SCHEDULED: <2016-03-11 Fri +6m>
  :PROPERTIES:
  - State "DONE"       from "TODO"       [2015-09-11 Fri 11:28]
  - State "WAITING"    from "TODO"       [2015-07-22 Wed 19:48]
  - State "DONE"       from "STARTED"    [2014-12-13 Sat 15:56]
  - State "DONE"       from "TODO"       [2014-03-31 Mon 03:11]
  - State "CANCELED"   from "TODO"       [2013-12-11 Wed 19:23]
  - State "CANCELED"   from "TODO"       [2013-05-21 Tue 12:30]
  - State "DONE"       from "TODO"       [2012-05-17 Thu 22:24]
  - State "CANCELED"   from "TODO"       [2011-04-23 Sat 22:04]
  - State "TODO"       from "DONE"       [2010-10-23 Sat 21:54]
  - State "DONE"       from "TODO"       [2010-10-08 Fri 14:26]

and with a fake "THE-END" drawer

  ** TODO Update auto insurance cards                             :Home:ATTACH:
  SCHEDULED: <2016-03-11 Fri +6m>
  :PROPERTIES:
  - State "DONE"       from "TODO"       [2015-09-11 Fri 11:28]
  - State "WAITING"    from "TODO"       [2015-07-22 Wed 19:48]
  - State "DONE"       from "STARTED"    [2014-12-13 Sat 15:56]
  - State "DONE"       from "TODO"       [2014-03-31 Mon 03:11]
  - State "CANCELED"   from "TODO"       [2013-12-11 Wed 19:23]
  - State "CANCELED"   from "TODO"       [2013-05-21 Tue 12:30]
  - State "DONE"       from "TODO"       [2012-05-17 Thu 22:24]
  - State "CANCELED"   from "TODO"       [2011-04-23 Sat 22:04]
  - State "TODO"       from "DONE"       [2010-10-23 Sat 21:54]
  - State "DONE"       from "TODO"       [2010-10-08 Fri 14:26]
  :THE-END:
  
Regards,

Re: bug in org-habits

[John Wiegley]

>>>>> Nicolas Goaziou <mail@nicolasgoaziou.fr> writes:

> As a matter of fact, going to the end of an entry is not negligible, because
> of inlinetasks. Also, it is not really O(1) since it depends on the size of
> the entry. To get an idea, on my computer, moving past a 500 lines entry
> takes around 0.001s. I can imagine slower computers, or larger entries,
> neither being unheard of, having a slightly noticeable delay.

I'm having a hard time buying the performance argument, since I've been using
Org-mode for many years, and never has this been a problem. You're making me
pay a cost (enforced structure), for a value only one of us perceives.

> Also, supporting both locations means that users will pay the full price in
> entries without a property drawer, even if they chose the fast path, i.e.,
> drawer at the beginning of the entry, in the first place. This kind of
> defeats some of the advantages of the current state.

It wouldn't be "users": it would be people intentionally opting to allow
floating properties. _I_ would be paying the price, and I will pay it happily
to keep 8.2 behavior.

John

Re: bug in org-habits

[Stelian Iancu]

On 04/11/15 02:01, John Wiegley wrote:
>>>>>> Nicolas Goaziou <mail@nicolasgoaziou.fr> writes:
>> Also, supporting both locations means that users will pay the full price in
>> entries without a property drawer, even if they chose the fast path, i.e.,
>> drawer at the beginning of the entry, in the first place. This kind of
>> defeats some of the advantages of the current state.
>
> It wouldn't be "users": it would be people intentionally opting to allow
> floating properties. _I_ would be paying the price, and I will pay it happily
> to keep 8.2 behavior.
>
> John
>

I would also like to have the 8.2 behavior. If it's easier, it would be 
fine to always have the drawer either in the beginning or in the end and 
nowhere else (and have an option to chose with the default in the 
beginning).

S.

Re: bug in org-habits

[Eric S Fraga]

On Tuesday,  3 Nov 2015 at 20:01, John Wiegley wrote:

[...]

> I'm having a hard time buying the performance argument, since I've been using
> Org-mode for many years, and never has this been a problem. You're making me
> pay a cost (enforced structure), for a value only one of us perceives.

John,

others *have* found issues with performance.

In 8.2, org struggled with many of my documents, some of which are very
large.  Most, if not all, of the performance problems I was having have
disappeared since then (thank you Nicolas!).  I cannot attribute the
improvements directly to the move to a fixed structure for headline
meta-data but I can easily believe that this move has contributed due to
anecdotal experience with complex sections within my org files.

This does not mean I am against your suggestion of a variable
controlling this behaviour.  Maybe implement it and post a patch for the
community to try out?

I am perfectly happy with the current format, however, so would be
unlikely to try it.  (sorry)

-- 
: Eric S Fraga (0xFFFCF67D), Emacs 25.0.50.2, Org release_8.3.2-209-gba4d33

Re: bug in org-habits

[Bastien Guerry]

Hi John and Achim,

John Wiegley <jwiegley@gmail.com> writes:

>>>>>> Achim Gratz <Stromeko@nexgo.de> writes:
>
>> I don't think so. Search for end of entry can be complex in itself and you
>> would never know if the properties you find by looking back aren't belonging
>> to an entry one level down unless you scanned the whole span again. Also,
>> properties can be any size and you have to search for the beginning of the
>> drawer.
>
> Good point.  Still, the option I'd still choose would be:
>
>     org-allow-floating-properties
>
>       If set to `non-nil', the :PROPERTIES: drawer may begin anywhere within
>       an entry. Note that this will slow down many operations due to the
>       additional scan needed, and may lead to incorrect result if similar text
>       is found elsewhere within an entry, which is the recommended and default
>       value for this variable is nil.

I think Achim's points above are very valid, and the flexibility
offered by the option above should be very carefully examined.

I'm trying hard to put aside one day or two for org-mode next week,
I'll come back on this issue then.

Thanks,

-- 
 Bastien

Re: bug in org-habits

[John Wiegley]

>>>>> Bastien Guerry <bzg@bzg.fr> writes:

> I think Achim's points above are very valid, and the flexibility offered by
> the option above should be very carefully examined.

I spoke to Nicolas directly and he mentioned that a goal for syntax regularity
is to make it possible to reliably read and manipulate Org files outside of
Emacs.

For this I *am* willing to give up order independence of PROPERTIES. Having a
customization option would needlessly increases the number of possibilities
external processors must consider, and so I retract my request.

John

Allowing loose ordering in Org files (Was: bug in org-habits)

[John Wiegley]

>>>>> John Wiegley <johnw@gnu.org> writes:

> I spoke to Nicolas directly and he mentioned that a goal for syntax
> regularity is to make it possible to reliably read and manipulate Org files
> outside of Emacs.
> 
> For this I *am* willing to give up order independence of PROPERTIES. Having
> a customization option would needlessly increases the number of
> possibilities external processors must consider, and so I retract my
> request.

I've had time this weekend to rethink my feature request, and I realized that
even machine-friendly formatting is something I should be able to give up, to
have an Org that works better for me.

What has always made Org great (to me) is that it's a rather "light" overlay
on a plain old text file. What structure it does enforce -- say, the actual
syntax of drawers -- has always felt fairly "fluid".

Lately there seems to be a push to sacrifice some of this freedom in order to
gain efficiency and regularity. I imagine this is for the benefit of machine
parsers; but what if one doesn't use any machine parsers? Org never asked me
to give up flexibility for unknown benefits before.

It should be asked whether users want to trade formatting freedom for those
benefits. If it has been asked, I missed that discussion. So unless it's an
heavy maintenance burden to allow floating properties, for example, I don't
see why I, as a user, shouldn't be allowed to make that choice.

To those who repeat the performance argument: This is an opt-in only request.
It is not about changing the performance of default Org, or making files more
difficult to parse outside of Emacs for everyone.

John

Re: Allowing loose ordering in Org files

[Rasmus]

Hi,

I share my personal views on this below.

John Wiegley <jwiegley@gmail.com> writes:

>>>>>> John Wiegley <johnw@gnu.org> writes:
>
>> I spoke to Nicolas directly and he mentioned that a goal for syntax
>> regularity is to make it possible to reliably read and manipulate Org files
>> outside of Emacs.
>> 
>> For this I *am* willing to give up order independence of PROPERTIES. Having
>> a customization option would needlessly increases the number of
>> possibilities external processors must consider, and so I retract my
>> request.
>
> I've had time this weekend to rethink my feature request, and I realized that
> even machine-friendly formatting is something I should be able to give up, to
> have an Org that works better for me.
>
> What has always made Org great (to me) is that it's a rather "light" overlay
> on a plain old text file. What structure it does enforce -- say, the actual
> syntax of drawers -- has always felt fairly "fluid".
>
> Lately there seems to be a push to sacrifice some of this freedom in order to
> gain efficiency and regularity. I imagine this is for the benefit of machine
> parsers; but what if one doesn't use any machine parsers? Org never asked me
> to give up flexibility for unknown benefits before.

This is one concern.  Another concern is adhere to the "Org syntax", which
I guess is what you mean by regularity.

There are other implementations of Org.  For instance, Github and Gitlab
display Org files via org-ruby (AFAIK).  If the claim was, that there is a
strong desire to formalize and stick to the Org syntax, I would agree.
Furthermore, I would claim this is a good thing.

    http://orgmode.org/worg/dev/org-syntax.html#Property_Drawers

An alternative is a freer take on syntax, such as how MD has evolved,
which IMO is a bit annoying.

> It should be asked whether users want to trade formatting freedom for those
> benefits. If it has been asked, I missed that discussion. So unless it's an
> heavy maintenance burden to allow floating properties, for example, I don't
> see why I, as a user, shouldn't be allowed to make that choice.

That’s a fair point.

> To those who repeat the performance argument: This is an opt-in only request.
> It is not about changing the performance of default Org, or making files more
> difficult to parse outside of Emacs for everyone.

I disagree with your last claim.

Rasmus

-- 
It was you, Jezebel, it was you

Re: Allowing loose ordering in Org files

[John Wiegley]

>>>>> Rasmus  <rasmus@gmx.us> writes:

>> To those who repeat the performance argument: This is an opt-in only
>> request. It is not about changing the performance of default Org, or making
>> files more difficult to parse outside of Emacs for everyone.

> I disagree with your last claim.

I'm not sure I understand, can you clarify? What I meant is that I don't want
to get in the way of the maintainer's vision for "default Org". I would just
like some customization options, to keep using Org as I'm used to it.

John

Re: Allowing loose ordering in Org files

[Achim Gratz]

John Wiegley writes:
>>>>>> John Wiegley <johnw@gnu.org> writes:
>> I spoke to Nicolas directly and he mentioned that a goal for syntax
>> regularity is to make it possible to reliably read and manipulate Org files
>> outside of Emacs.

How about keeping the discussion on the list and stop Cc: and Reaply-To:
madness?

> I've had time this weekend to rethink my feature request, and I realized that
> even machine-friendly formatting is something I should be able to give up, to
> have an Org that works better for me.

The whole point of defining a formal syntax for Org is that it becomes
possible to parse Org documents with something other than Emacs and
still make sense of them.  To reap that benefit, you need to drop some
of the ad-hoc parsing that Org did in the past.

> What has always made Org great (to me) is that it's a rather "light" overlay
> on a plain old text file. What structure it does enforce -- say, the actual
> syntax of drawers -- has always felt fairly "fluid".

Still does.

> Lately there seems to be a push to sacrifice some of this freedom in order to
> gain efficiency and regularity. I imagine this is for the benefit of machine
> parsers; but what if one doesn't use any machine parsers? Org never asked me
> to give up flexibility for unknown benefits before.

Yes it did and still does, just in other places that you may or may not
care about.  For instance, it asks you to not use multi-line table
cells, or column and row spans.  It also doesn't let you use a :TBLFM:
drawer instead of that #+TBLFM: line just because it looks more neat (it
really should, BTW :-).

> It should be asked whether users want to trade formatting freedom for those
> benefits. If it has been asked, I missed that discussion. So unless it's an
> heavy maintenance burden to allow floating properties, for example, I don't
> see why I, as a user, shouldn't be allowed to make that choice.

I won't talk about "maintenance burden", but in any case it's a
technical debt.  You'd have to maintain two code paths that accomplish
the same result or at least should.

> To those who repeat the performance argument: This is an opt-in only request.
> It is not about changing the performance of default Org, or making files more
> difficult to parse outside of Emacs for everyone.

You will find that the argument really wasn't about performance, but
complexity.


Regards,
Achim.
-- 
+<[Q+ Matrix-12 WAVE#46+305 Neuron microQkb Andromeda XTk Blofeld]>+

SD adaptations for KORG EX-800 and Poly-800MkII V0.9:
http://Synth.Stromeko.net/Downloads.html#KorgSDada

Re: Allowing loose ordering in Org files

[John Wiegley]

>>>>> Achim Gratz <Stromeko@nexgo.de> writes:

> The whole point of defining a formal syntax for Org is that it becomes
> possible to parse Org documents with something other than Emacs and still
> make sense of them. To reap that benefit, you need to drop some of the
> ad-hoc parsing that Org did in the past.

Yes, I know this. The benefits of regular structure are unrelated to my
request for freedom from the constraints of such regularity.

> You will find that the argument really wasn't about performance, but
> complexity.

I can accept a complexity argument, if my request were really "a separate
code-path". I'm not sure it is. For example, my needs could be satisfied by
something as simple as:

    (defun parse-org-entry (...)
        (let ((props (funcall 'parse-org-properties-function ...)))
           ...))

`parse-org-properties-function' would point to a function that does what Org
8.3 does now. This gives me the option of porting over the 8.2 version and
keeping the old behavior. All that needs to be done is to allow this hook, no?

John

Re: Allowing loose ordering in Org files

[Rasmus]

Hi John,

John Wiegley <jwiegley@gmail.com> writes:

>>>>>> Rasmus  <rasmus@gmx.us> writes:
>
>>> To those who repeat the performance argument: This is an opt-in only
>>> request. It is not about changing the performance of default Org, or making
>>> files more difficult to parse outside of Emacs for everyone.
>
>> I disagree with your last claim.
>
> I'm not sure I understand, can you clarify? What I meant is that I don't want
> to get in the way of the maintainer's vision for "default Org".

Note that I’m *not* the maintainer and speak with no authority on the issue!

Some alternative Org interpreter must be able to find the properties
drawer since it needs to support, e.g. linking to headlines,

     * f_0 
     :PROPERTIES:
     :CUSTOM_ID: f0
     :END:
     * f_1
     See also [[#f0]]

If the placement of properties is "free", the secondary interpreter /must/
support this customization option to be able to interpret the org format.
Note, this matters for both interactive usage (being able to click/open a
reference) and for "export" (e.g. org-ruby).

> I would just like some customization options, to keep using Org as I'm
> used to it.

I sympathize, and I don’t know what is the correct behavior here.  I do
think that allowing this is obviously the right thingᵀᴹ, but it might
still be the lesser of two evils.  I don’t fell qualified to form opinions
on this.

Rasmus

-- 
Vote for Dick Taid in an election near you!

Re: Allowing loose ordering in Org files

[John Wiegley]

>>>>> Rasmus  <rasmus@gmx.us> writes:

> If the placement of properties is "free", the secondary interpreter /must/
> support this customization option to be able to interpret the org format.
> Note, this matters for both interactive usage (being able to click/open a
> reference) and for "export" (e.g. org-ruby).

If my request is fulfilled, I would expect that changing the "find properties
function" would imply that one's Org file is no longer usable by secondary
interpreters. I.e., "If you change this, you do so at your own risk".

John

Re: Allowing loose ordering in Org files

[Achim Gratz]

John Wiegley writes:
>> You will find that the argument really wasn't about performance, but
>> complexity.
>
> I can accept a complexity argument,

I meant O() complexity, not implementation complexity.

> if my request were really "a separate
> code-path". I'm not sure it is. For example, my needs could be satisfied by
> something as simple as:
>
>     (defun parse-org-entry (...)
>         (let ((props (funcall 'parse-org-properties-function ...)))
>            ...))
>
> `parse-org-properties-function' would point to a function that does what Org
> 8.3 does now. This gives me the option of porting over the 8.2 version and
> keeping the old behavior. All that needs to be done is to allow this hook, no?

If it's not about providing the alternative behaviour within Org, then
what does this allow you to do that advising parse-org-entry can't?  A
hook still has the cost of simply being there even when I don't use it.


Regards,
Achim.
-- 
+<[Q+ Matrix-12 WAVE#46+305 Neuron microQkb Andromeda XTk Blofeld]>+

SD adaptation for Waldorf rackAttack V1.04R1:
http://Synth.Stromeko.net/Downloads.html#WaldorfSDada

Re: Allowing loose ordering in Org files

[Stelian Iancu]

On 09/11/15 21:04, Achim Gratz wrote:
> John Wiegley writes:
>>> You will find that the argument really wasn't about performance, but
>>> complexity.
>>
>> I can accept a complexity argument,
>
> I meant O() complexity, not implementation complexity.
>
>> if my request were really "a separate
>> code-path". I'm not sure it is. For example, my needs could be satisfied by
>> something as simple as:
>>
>>      (defun parse-org-entry (...)
>>          (let ((props (funcall 'parse-org-properties-function ...)))
>>             ...))
>>
>> `parse-org-properties-function' would point to a function that does what Org
>> 8.3 does now. This gives me the option of porting over the 8.2 version and
>> keeping the old behavior. All that needs to be done is to allow this hook, no?
>
> If it's not about providing the alternative behaviour within Org, then
> what does this allow you to do that advising parse-org-entry can't?  A
> hook still has the cost of simply being there even when I don't use it.
>

John, if you end up writing an advice for this function, please share it 
with the list, as I would like the 8.2 behavior as well (I unfortunately 
don't know enough elisp and org internals to do such a thing).

Thanks!

S.

Re: Allowing loose ordering in Org files

[John Wiegley]

>>>>> Stelian Iancu <stelian@iancu.ch> writes:

> John, if you end up writing an advice for this function, please share it
> with the list, as I would like the 8.2 behavior as well (I unfortunately
> don't know enough elisp and org internals to do such a thing).

To Achim I would say that these are the reasons to have a hook: Because then
it gets documented as an option; it's supported within Org (rather than being
a separate thing people have to find in my .emacs on GitHub); and it shows up
when you M-x customize-group RET org RET.

John

Re: Allowing loose ordering in Org files (Was: bug in org-habits)

[Aaron Ecay]

Hi John,

2015ko azaroak 9an, John Wiegley-ek idatzi zuen:

[...]

> Lately there seems to be a push to sacrifice some of this freedom in order to
> gain efficiency and regularity. I imagine this is for the benefit of machine
> parsers; but what if one doesn't use any machine parsers? 

I don’t think it’s possible to separate things like this.  Large parts
of org use a machine parser, written in elisp.  There are (perhaps
asymptotic) plans to transition the rest of org to work based on this
parser.

Adding knobs to this parser increases the burden of those who have
to build and maintain it.  It also heightens the burden for users
(especially novices): M-x customize-group org suddenly asks you one
(or more) questions about details of the syntax that previously you
didn’t need to consider.

We have discussions about extending the syntax fairly regularly.  It
would be good to discuss what questions we might ask of those proposals
to determine whether they should go forward.  Some that I can think of
are:
1. Is there a good (user-friendly, reliable) way to accomplish the same
   goal, given the resources currently available?
2. Is there a large community of users who need this feature and/or
   would adopt it if it became available?
3. Is this something that org’s “competitors” provide easily?  (Not
   necessarily out of a spirit of competition, but rather demonstrating
   a use case.)

I don’t include difficulty of implementation on that list.  I don’t
think the developers should wag the users.  Unfortunately however, I
don’t think your proposal fares well in light of these questions.  (I
don’t mean to imply that they are authoritative; anyone could very well
propose others.  I would be happy if a consensus developed about what
the right questions are, even if there is disagreement about the answers
in this specific case.)

> Org never asked me to give up flexibility for unknown benefits before.
> 
> It should be asked whether users want to trade formatting freedom for those
> benefits. If it has been asked, I missed that discussion. So unless it's an
> heavy maintenance burden to allow floating properties, for example, I don't
> see why I, as a user, shouldn't be allowed to make that choice.

I think framing it in terms of freedom is potentially misleading.
Because org is free software, its users are maximally free to do
any of a wide variety of things, including sticking with an old version,
patching the code locally, distributing a patch/fork/set of advices,
using another program, ...

I think it’s more illuminating to think of it in terms of org as a tool:
have the changes made it more difficult for you to accomplish your goals
with org?  Has something that was previously possible become impossible?
Has something that was previously easy gotten harder?  If the answer to
one of these questions is yes, then we can think of ways to solve the
difficulties.

Of course, you’ve already received quite a bit of feedback about the
proposal from a cross-section of the community.  So what I’ve said will,
I hope, function partially as a lens through which to understand that
feedback, as well as a framework in which to continue discussion if it’s
needed.

-- 
Aaron Ecay

Re: Allowing loose ordering in Org files

[John Wiegley]

>>>>> Aaron Ecay <aaronecay@gmail.com> writes:

> Adding knobs to this parser increases the burden of those who have to build
> and maintain it.

Thank you for your reply, Aaron, I found it most illuminating.

If the answer from the maintainers is "It's more work than we want to do",
that's completely acceptable. I've been operating under the premise that it
wouldn't be difficult to add such an option (just the hook, mind you, not the
functionality behind it).

I suppose at this point it becomes a question of whether others want this as
much as I do. If it's just a handful of us, and the maintainers find the
option onerous, that's really the end of it.

> I think it’s more illuminating to think of it in terms of org as a tool:
> have the changes made it more difficult for you to accomplish your goals
> with org? Has something that was previously possible become impossible? Has
> something that was previously easy gotten harder? If the answer to one of
> these questions is yes, then we can think of ways to solve the difficulties.

There is another vector to consider, and a far more nebulous one: How does it
impact Org's "luft"? That is, the feeling of ease and comfort Org conveys in
its use.

There are many highly functional alternatives to Org that I've tried and
rejected because they lack the easy grace of Org. That grace is why I've been
able to stick with it after almost 9,000 handled tasks. Any perception of
"inertia" in a tasking system causes me to psychologically avoid it, even if I
have no rational basis for that aversion.

I sincerely hope that those with high technical motives will keep in mind the
usability of Org beyond purely technical considerations. It should say
something that a long-time user is unhappy with the way Org "feels" in 8.3.

John

Re: Allowing loose ordering in Org files

[Thomas S. Dye]

John Wiegley <jwiegley@gmail.com> writes:

>
> There is another vector to consider, and a far more nebulous one: How does it
> impact Org's "luft"? That is, the feeling of ease and comfort Org conveys in
> its use.
>
> There are many highly functional alternatives to Org that I've tried and
> rejected because they lack the easy grace of Org. That grace is why I've been
> able to stick with it after almost 9,000 handled tasks. Any perception of
> "inertia" in a tasking system causes me to psychologically avoid it, even if I
> have no rational basis for that aversion.
>
> I sincerely hope that those with high technical motives will keep in mind the
> usability of Org beyond purely technical considerations. It should say
> something that a long-time user is unhappy with the way Org "feels" in 8.3.

I admire John's many accomplishments and value his membership in the Org
mode community.

Is the hook he is requesting a difficult thing to implement?  Would it
be possible to describe the customization variable in a "Super User"
section that is clearly not for the faint at heart?

I'm not suggesting anyone should implement a hook or create a
customization variable, I'm just curious (and unable to figure out
answers on my own).

All the best,
Tom

-- 
Thomas S. Dye
http://www.tsdye.com

Re: Allowing loose ordering in Org files (Was: bug in org-habits)

[Stelian Iancu]

On 10/11/15 02:40, Aaron Ecay wrote:

> I think it’s more illuminating to think of it in terms of org as a tool:
> have the changes made it more difficult for you to accomplish your goals
> with org?  Has something that was previously possible become impossible?
> Has something that was previously easy gotten harder?  If the answer to
> one of these questions is yes, then we can think of ways to solve the
> difficulties.

For me personally this change in particular has made me confused about 
the behavior in 8.3 versus 8.2. Since I don't have a huge backlog and 
workflow with org, I can adjust to this new way if I need to. However, 
like John, I would like the possibility of customizing this behavior 
somehow.

S.

Re: Allowing loose ordering in Org files

[Nicolas Goaziou]

Hello,

Thomas S. Dye <tsd@tsdye.com> writes:

> Is the hook he is requesting a difficult thing to implement?  Would it
> be possible to describe the customization variable in a "Super User"
> section that is clearly not for the faint at heart?
>
> I'm not suggesting anyone should implement a hook or create a
> customization variable, I'm just curious (and unable to figure out
> answers on my own).

I really don't like the idea of making Org /syntax/ customizable, would
it be with the help of a hook or a variable.

Regards,

-- 
Nicolas Goaziou

Re: Allowing loose ordering in Org files

[Matt Lundin]

John Wiegley <jwiegley@gmail.com> writes:

>>>>>> Aaron Ecay <aaronecay@gmail.com> writes:
>
>> Adding knobs to this parser increases the burden of those who have to build
>> and maintain it.
>
> Thank you for your reply, Aaron, I found it most illuminating.
>
> If the answer from the maintainers is "It's more work than we want to
> do", that's completely acceptable. I've been operating under the
> premise that it wouldn't be difficult to add such an option (just the
> hook, mind you, not the functionality behind it).

After looking quickly at the code, my impression is that without
substantial refactoring, a hook/variable pointing to an alternate "find
property drawer" function is not a very clean option, since org makes
the assumption in several places that the property drawer comes
immediately after the planning info. See, for instance,
org-insert-drawer, org-end-of-meta-data, org-get-property-block.

If you were to hack something on your own, I suppose your could
rewrite/advise org-get-property-block, but I have no idea what else that
would break.

> If my request is fulfilled, I would expect that changing the "find
> properties function" would imply that one's Org file is no longer
> usable by secondary interpreters. I.e., "If you change this, you do so
> at your own risk".
[ ... quoted from another email ] 

I would vote against creating a hook or variable that comes with a
warning "change this at your own risk." I am concerned about the
precedent this would set. This seems to me what defadvice is for. Would
this really be a simpler option than posting a hack on emacswiki or
github? After all, anyone customizing the variable would still have to
grab the custom function from github, etc.

IMO, to add a customization option that might (and probably will) break
other parts of org mode threatens to add complexity to the maintenance
of org, since it might create a base of users who are relying on a
"semi-officially supported hack" rather than the official parsing logic
supported by org. Despite the disclaimer, maintainers will still be
forced to keep the possibility of this customization in mind when
rewriting functions that parse org syntax. And even if a user accepts
the full risk of the customization, it is still possible that he/she
might accidentally report a bug to the mailing list when something
breaks (not realizing it is related to the customization).

In short, if we allow for an alternative parsing logic, I think it
should be one that is officially supported/maintained.

Best,
Matt

Re: Allowing loose ordering in Org files

[Matt Lundin]

John Wiegley <jwiegley@gmail.com> writes:

> There is another vector to consider, and a far more nebulous one: How does it
> impact Org's "luft"? That is, the feeling of ease and comfort Org conveys in
> its use.

FWIW, I personally have found org both faster and much more reliable
thanks to Nicolas' heroic work to tighten up org syntax. Org files open
faster, the agenda parses faster, the exporter is orders of magnitude
more consistent, org babel blocks behave as expected, etc. And the user
interaction has far fewer glitches than I experienced before the change.
For instance, for years, org mode functions on my machine often inserted
property drawers inside of property drawers or inserted multiple
property drawers in a single entry. In my experience, the changes to the
parser have made all this much more robust and predictable.

So for me, the increasing robustness of org mode makes it feel easier,
more pleasant to use.

> There are many highly functional alternatives to Org that I've tried
> and rejected because they lack the easy grace of Org. That grace is
> why I've been able to stick with it after almost 9,000 handled tasks.
> Any perception of "inertia" in a tasking system causes me to
> psychologically avoid it, even if I have no rational basis for that
> aversion.
>
> I sincerely hope that those with high technical motives will keep in mind the
> usability of Org beyond purely technical considerations. It should say
> something that a long-time user is unhappy with the way Org "feels" in 8.3.

I'm not sure "purely technical" is fair characterization of the reasons
for the syntax changes. As I understand it, the chief reason that org
syntax needed a cleanup is because of the massive amount of
functionality that org mode acquired over the years. Ensuring all this
worked smoothly and robustly for users required a more regular,
predictable syntax. So user experience was key to the changes as well.

Best,
Matt

Re: Allowing loose ordering in Org files

[Thomas S. Dye]

Nicolas Goaziou <mail@nicolasgoaziou.fr> writes:

> Hello,
>
> Thomas S. Dye <tsd@tsdye.com> writes:
>
>> Is the hook he is requesting a difficult thing to implement?  Would it
>> be possible to describe the customization variable in a "Super User"
>> section that is clearly not for the faint at heart?
>>
>> I'm not suggesting anyone should implement a hook or create a
>> customization variable, I'm just curious (and unable to figure out
>> answers on my own).
>
> I really don't like the idea of making Org /syntax/ customizable, would
> it be with the help of a hook or a variable.

OK.  IIUC, John's options if he wants to continue using Org mode are to
run org-repair-property-drawers, craft a defadvice to change how
property drawers are identified, or use 8.2.

I've found org-lint helpful in keeping old Org mode files functional,
but my files are much less structured than John's seem to be.

All the best,
Tom

-- 
Thomas S. Dye
http://www.tsdye.com

Re: Allowing loose ordering in Org files

[Achim Gratz]

John Wiegley writes:
> If the answer from the maintainers is "It's more work than we want to do",
> that's completely acceptable. I've been operating under the premise that it
> wouldn't be difficult to add such an option (just the hook, mind you, not the
> functionality behind it).

To answer your question from another post: If we add a hook, but not the
functionality behind it, then we are going to advertise something we
don't recommend to do on grounds that a random user might very well not
comprehend.  If we do add the functionality we might be better off with
an option rather than a hook, but then we incur the debt of having to
support it both in the syntax and the implementation.  That was the
reason I asked you about simply advising some function.  It doesn't
advertise some option that then isn't implemented and if someone really
cares about that functionality we can still show (even on Worg) how to
do it.  But not in the Org manual or as an official option.

[There was a precedent to this with Org 7 where you could go in and
change what Org considered a headline.  When this was changed we've had
similar discussions and I expect this one to take the same route to be
honest.]

> There is another vector to consider, and a far more nebulous one: How does it
> impact Org's "luft"? That is, the feeling of ease and comfort Org conveys in
> its use.

If you don't use properties then it doesn't affect you at all.  If you
do, then… well, I personally simply don't care.  Just like there's
several style guides for writing C; as long as these are applied
consistently I can live with most of them and put the braces and indents
the way they prescribe.

> There are many highly functional alternatives to Org that I've tried and
> rejected because they lack the easy grace of Org. That grace is why I've been
> able to stick with it after almost 9,000 handled tasks. Any perception of
> "inertia" in a tasking system causes me to psychologically avoid it, even if I
> have no rational basis for that aversion.
>
> I sincerely hope that those with high technical motives will keep in mind the
> usability of Org beyond purely technical considerations. It should say
> something that a long-time user is unhappy with the way Org "feels" in 8.3.

So write the advise and move on?  If you weren't so heavily invested in
what you perceive as "the right style" you quite likely wouldn't care,
or would you?


Regards,
Achim.
-- 
+<[Q+ Matrix-12 WAVE#46+305 Neuron microQkb Andromeda XTk Blofeld]>+

SD adaptation for Waldorf microQ V2.22R2:
http://Synth.Stromeko.net/Downloads.html#WaldorfSDada

Re: Allowing loose ordering in Org files

[John Wiegley]

>>>>> Nicolas Goaziou <mail@nicolasgoaziou.fr> writes:

> I really don't like the idea of making Org /syntax/ customizable, would it
> be with the help of a hook or a variable.

From what I've seen so far, several users want regularity of syntax to decide
formatting, and several users want user preference to decide formatting. There
do seem to be larger costs for letting the user decide; but there are also
costs to not letting the user decide, that I feel are not being appreciated.

The reason I'm sticking on this point is because it also relates to our future
road map. If Org continues to do this -- to trade flexibility of formatting
for regularity of parsing -- it might continue to alienate some of us.

John

Re: Allowing loose ordering in Org files

[John Wiegley]

>>>>> Achim Gratz <Stromeko@nexgo.de> writes:

> If you don't use properties then it doesn't affect you at all. If you do,
> then… well, I personally simply don't care. Just like there's several style
> guides for writing C; as long as these are applied consistently I can live
> with most of them and put the braces and indents the way they prescribe.

In my regimen, every single entry has a PROPERTIES drawer, since I tag each
one with ID and CREATED, for future reference. Most items are SCHEDULED as
well. So when I open up a headline to look at the contents, I see:

    * Head
    SCHEDULED
    text
    :PROPERTIES:...

It's a trivial thing, but I'd rather not scan past two lines to start reading
my entry.

What a lot of people want is trivial, and only relevant to them, but part of
the Emacs philosophy is to give them the freedom to customize their
environment the way they want it to work, rather than decide for them what is
the "right way".

I won't argue for this anymore, if it really does incur work for you that only
benefits me and a few others. I suppose that I'm still writing these e-mails
because I want to inject some sensitivity about these matters into your future
decisions, so you don't take away such flexibility again lightly.

> So write the advise and move on? If you weren't so heavily invested in what
> you perceive as "the right style" you quite likely wouldn't care, or would
> you?

I'm invested in the spirit of the project, since I've been using it for a very
long time, and continue to recommend it to many people as an amazing tool for
self-organization. The more it becomes something I have to hack to like, the
less I feel like putting my heart into it.

I *care*, which is maybe both a blessing and a curse. It is the origin of my
creative contributions, but also the cause of these annoying threads. I'd
rather work with you guys than against you; but there are some sacred cows
that moan plaintively, when a sharp parenthesis is drawn across their throats.

Yours,
  John

Re: Allowing loose ordering in Org files

[Achim Gratz]

John Wiegley writes:
> In my regimen, every single entry has a PROPERTIES drawer, since I tag each
> one with ID and CREATED, for future reference. Most items are SCHEDULED as
> well. So when I open up a headline to look at the contents, I see:
>
>     * Head
>     SCHEDULED
>     text
>     :PROPERTIES:...
>
> It's a trivial thing, but I'd rather not scan past two lines to start reading
> my entry.

So isn't your request rather to hide the properties drawer better by
default?  You were _only_ talking about the UX in this whole thread and
that might be a lot easier to adapt while not changing the way Org
syntax is defined.

There were lots of bugs to be fixed with properties and properties
inheritance and I dabbled in that endeavour, which makes me more
cautious about wading in and changing it yet again.  Knowing where to
find the properties quickly is key to this not falling apart again,
IMHO.  Even then, due to inheritance properties are probably the most
complex thing to wrap your head around in Org aside from Babel.

[…]
> I *care*, which is maybe both a blessing and a curse. It is the origin of my
> creative contributions, but also the cause of these annoying threads. I'd
> rather work with you guys than against you; but there are some sacred cows
> that moan plaintively, when a sharp parenthesis is drawn across their throats.

As I said above, the solution may well be different than both what you
want to keep and what Org offers now.  I don't find that thread
particularly annoying, btw.


Regards,
Achim.
-- 
+<[Q+ Matrix-12 WAVE#46+305 Neuron microQkb Andromeda XTk Blofeld]>+

Waldorf MIDI Implementation & additional documentation:
http://Synth.Stromeko.net/Downloads.html#WaldorfDocs

Re: Allowing loose ordering in Org files

[Matt Lundin]

John Wiegley <jwiegley@gmail.com> writes:

>>>>>> Nicolas Goaziou <mail@nicolasgoaziou.fr> writes:
>
>> I really don't like the idea of making Org /syntax/ customizable, would it
>> be with the help of a hook or a variable.
>
> From what I've seen so far, several users want regularity of syntax to
> decide formatting, and several users want user preference to decide
> formatting. There do seem to be larger costs for letting the user
> decide; but there are also costs to not letting the user decide, that
> I feel are not being appreciated.

Could you please be more specific about what the problem is? Apart from
the location of properties, are there other forms of flexible formatting
that have been lost in recent releases? And what specifically is the
problem with a required property drawer location? Is it aesthetic?
Functional?

> The reason I'm sticking on this point is because it also relates to our future
> road map. If Org continues to do this -- to trade flexibility of formatting
> for regularity of parsing -- it might continue to alienate some of us.

I don't think this is a fair description of the direction org-mode has
been taking. I think org-mode has has unparalleled flexibility. If
anything, that flexibility has grown over the past few years. E.g., I
can enter any number of drawers, tables, lists, code blocks, links,
footnotes, etc. in a single entry. And I can export all of that data
reliable to a growing number of formats.

But when it comes to meta-data, I think it has been helpful to tighten
up the syntax. After all, org-mode has always required many types of
fixed formatting/syntax. For instance, one can't just put tags anywhere
in the body of an entry. Nor can one use arbitrary symbols to designate
headline levels. Nor can on put TODO keywords anywhere (despite frequent
requests for such functionality).

Since org-mode recognizes only one property drawer per entry, it makes
sense (for clarity, simplicity, efficiency, etc.) to require that this
special key:value metadata be "attached" to the headline (like tags,
TODOs, etc.).

Matt

Re: Allowing loose ordering in Org files

[Matt Lundin]

Matt Lundin <mdl@imapmail.org> writes:

> John Wiegley <jwiegley@gmail.com> writes:
>
>>>>>>> Nicolas Goaziou <mail@nicolasgoaziou.fr> writes:
>>
>>> I really don't like the idea of making Org /syntax/ customizable, would it
>>> be with the help of a hook or a variable.
>>
>> From what I've seen so far, several users want regularity of syntax to
>> decide formatting, and several users want user preference to decide
>> formatting. There do seem to be larger costs for letting the user
>> decide; but there are also costs to not letting the user decide, that
>> I feel are not being appreciated.
>
> Could you please be more specific about what the problem is? Apart from
> the location of properties, are there other forms of flexible formatting
> that have been lost in recent releases? And what specifically is the
> problem with a required property drawer location? Is it aesthetic?
> Functional?
>

Please disregard this bit. Just saw your reply to Achim.

Matt

Re: Allowing loose ordering in Org files

[John Wiegley]

>>>>> Achim Gratz <Stromeko@nexgo.de> writes:

> So isn't your request rather to hide the properties drawer better by
> default? You were _only_ talking about the UX in this whole thread and that
> might be a lot easier to adapt while not changing the way Org syntax is
> defined.

Good call, Achim! I became too focused on the problem, and missed the real
point.

> As I said above, the solution may well be different than both what you want
> to keep and what Org offers now. I don't find that thread particularly
> annoying, btw.

Thanks.

John

Re: Allowing loose ordering in Org files

[Karl Voit]

* John Wiegley <jwiegley@gmail.com> wrote:
>
> In my regimen, every single entry has a PROPERTIES drawer, since I tag each
> one with ID and CREATED, for future reference. 

This also holds for my Org-mode files - in general.

> Most items are SCHEDULED as well. So when I open up a headline to
> look at the contents, I see:
>
>     * Head
>     SCHEDULED
>     text
>    :PROPERTIES:...
>
> It's a trivial thing, but I'd rather not scan past two lines to start reading
> my entry.

I just wanted to add an additional notion to the discussion:

In most cases, the content of the drawers gets populated
automatically (LOGBOOK, CREATED, ...). In some cases, I manually add
properties, mainly :ID: for being able to use references to it.

Below the drawers, there is the actual content which is free text,
mixed with lists, blocks, and so forth.

In those cases, I prefer drawers being closed by default.


However, in my contacts.org file, I have entries where the actual
content is in a rather strict form like:

,----
| ** Firstname Lastname                  :FirstnameLastname:
| :PROPERTIES:
| :TYPE: person
| :TITLE:
| :EMAIL: Firstname.Lastname@example.com
| :URL: http://example.com
| :MOBILE: 0043/123456789
| :HOMEPHONE:
| :WORKPHONE:
| :PHONE:
| :COMPANY:
| :STREET: Herrengasse 42
| :POSTALCODE: 8042
| :CITY: Graz
| :COUNTRY: Österreich
| :PHOTOGRAPH: [[photo:FirstnameLastname.jpg]]
| :BORN: 1970-12-31
| :ITOLDTHEM_EMAIL: Lastname@MYDOMAIN.at
| :ITOLDTHEM_ADDRESS: home
| :ITOLDTHEM_PHONE: mobile
| :ADDRESS_CHANGE_METHOD: email
| :CREATED:  [2015-11-11 Wed 16:51]
| :END:
|
| - first contact: <2015-11-11 Wed> when meeting at id:FooConf15
`----

Here, the drawer is of particular interest to me and I'd love to
have them expanded together with the heading.

Besides, I once started an attempt to define a standard for contact
property item names in order to enable external tools to parse
contact data like [1]. Unfortunately, my focus shifted and I did not
follow my standardization attempt much further:
http://thread.gmane.org/gmane.emacs.orgmode/45347/focus=45740
http://thread.gmane.org/gmane.emacs.orgmode/47434/focus=47490
http://article.gmane.org/gmane.emacs.orgmode/57231/
[...]


Two notions: first, content of properties are of different interest
for different people in different org-mode files with different
data. It's hard to derive a general rule here.

Second, I still do think that a bit more standardization would be a
benefit for Org-mode (contact data, order of org elements, ...).
Having written a pretty dumb Org-mode parser in Python by myself for
[2] I recognized that this is not an easy job to do properly outside
of Elisp. And: being able to use Org-mode files outside of
Emacs/Elisp is also of interest for all users of Emacs/Elisp.

Just my 2 cents.


[1] https://github.com/novoid/org-contacts2vcard

[2] https://github.com/novoid/lazyblorg

-- 
mail|git|SVN|photos|postings|SMS|phonecalls|RSS|CSV|XML to Org-mode:
       > get Memacs from https://github.com/novoid/Memacs <

https://github.com/novoid/extract_pdf_annotations_to_orgmode + more on github