Debugging at least 2 regressions in org-mode master breaking ox-hugo

Debugging at least 2 regressions in org-mode master breaking ox-hugo

[Kaushal Modi]

[-- Attachment #1: Type: text/plain, Size: 1708 bytes --]

Hello,

I recently updated to the latest org-mode master and it is failing
ox-hugo[1] build and tests at 2 places.

Failure 1: org-get-outline-path has moved, and not mentioned in ORG-NEWS

Compiling ox-hugo.el now gives:

ox-hugo.el:4284:1: Warning: the function ‘org-get-outline-path’ is not
known to be defined.

I see that defun has now moved to org-refile.el. I see that
org-get-outline-path has nothing to do specific to refiling. Can that be
moved back to org.el, or may be a separate library? Otherwise, ox-hugo.el
will have to load org-refile.el too (yes, I don't use org-refile (yet), and
that's how I discovered this :))

Failure 2: Change in parsing of org babel header arguments.

This was caught by my weekly Travis CI cron jobs for ox-hugo:
https://travis-ci.org/kaushalmodi/ox-hugo/jobs/655410731#L2426

26c26
< {{< highlight emacs-lisp "hl_lines=1" >}}
---
> {{< highlight emacs-lisp "hl_lines=1 3-5" >}}

Earlier this kind of src block header:

#+begin_src emacs-lisp :hl_lines 1,3-5
...
#+end_src

got exported as

{{< highlight emacs-lisp "hl_lines=1 3-5" >}}

The regression is that now it is getting exported as

{{< highlight emacs-lisp "hl_lines=1" >}}

The values that I have after the comma in ":hl_lines 1,3-5" are getting
lost.

The relevant snippet where I parse the header arguments in ox-hugo.el is at
https://github.com/kaushalmodi/ox-hugo/blob/f8ec4aa5ad7d92f94bd8dbb814d85f980be67aea/ox-hugo.el#L2563

This behavior change in org-babel-parse-header-arguments is also not
documented in ORG-NEWS. I will now investigate what cause this regression.

...

--
Kaushal Modi

[1]: https://github.com/kaushalmodi/ox-hugo

[-- Attachment #2: Type: text/html, Size: 2663 bytes --]

Re: Debugging at least 2 regressions in org-mode master breaking ox-hugo

[Kaushal Modi]

[-- Attachment #1: Type: text/plain, Size: 1209 bytes --]

On Thu, Feb 27, 2020 at 9:00 AM Kaushal Modi <kaushal.modi@gmail.com> wrote:

> Failure 2: Change in parsing of org babel header arguments.
>
> The relevant snippet where I parse the header arguments in ox-hugo.el is
> at
> https://github.com/kaushalmodi/ox-hugo/blob/f8ec4aa5ad7d92f94bd8dbb814d85f980be67aea/ox-hugo.el#L2563
>
> This behavior change in org-babel-parse-header-arguments is also not
> documented in ORG-NEWS. I will now investigate what cause this regression.
>

The regression is caused by
https://code.orgmode.org/bzg/org-mode/commit/6b2a7cb20b357e730de151522fe4204c96615f98
or the later commit that changes `org-babel--string-to-number'.

Using this function redefinition with additional debug messages:

(defun org-babel--string-to-number (string)
  "If STRING represents a number return its value.
Otherwise return nil."
  (message "DBG: string: %S" string)
  (unless (string-match-p "\\s-" (org-trim string))
    (let ((interned-string (ignore-errors (read string))))
      (when (numberp interned-string)
        (message "DBG: interned string: %S" interned-string)
interned-string))))

I get:

DBG: string: "1,3-5"
DBG: interned string: 1

So that ",3-5" piece of information is lost.

[-- Attachment #2: Type: text/html, Size: 2073 bytes --]

Re: Debugging at least 2 regressions in org-mode master breaking ox-hugo

[Kaushal Modi]

[-- Attachment #1: Type: text/plain, Size: 1414 bytes --]

On Thu, Feb 27, 2020 at 9:13 AM Kaushal Modi <kaushal.modi@gmail.com> wrote:

> The regression is caused by
> https://code.orgmode.org/bzg/org-mode/commit/6b2a7cb20b357e730de151522fe4204c96615f98
> or the later commit that changes `org-babel--string-to-number'.
>
> Using this function redefinition with additional debug messages:
>
> (defun org-babel--string-to-number (string)
>   "If STRING represents a number return its value.
> Otherwise return nil."
>   (message "DBG: string: %S" string)
>   (unless (string-match-p "\\s-" (org-trim string))
>     (let ((interned-string (ignore-errors (read string))))
>       (when (numberp interned-string)
>         (message "DBG: interned string: %S" interned-string)
> interned-string))))
>
> I get:
>
> DBG: string: "1,3-5"
> DBG: interned string: 1
>
> So that ",3-5" piece of information is lost.
>

To be more specific, here is the call order:

org-babel-parse-header-arguments -> org-babel-read ->
org-babel--string-to-number

org-babel-read returns the string as-is if org-babel--string-to-number
returns nil.

*The regression is that earlier (org-babel--string-to-number "1,3-5") used
to return nil, but now it returns 1.*

I think that it should return a number only if 100% of the input string
represents a number. In the case of "1,3-5", it makes sense for it to still
return nil, so that org-babel-read does not throw away the ",3-5" piece of
information.

[-- Attachment #2: Type: text/html, Size: 2240 bytes --]

Re: Debugging at least 2 regressions in org-mode master breaking ox-hugo

[Adam Porter]

Kaushal Modi <kaushal.modi@gmail.com> writes:

> Failure 1: org-get-outline-path has moved, and not mentioned in ORG-NEWS
>
> Compiling ox-hugo.el now gives:
>
> ox-hugo.el:4284:1: Warning: the function ‘org-get-outline-path’ is not known to be defined.
>
> I see that defun has now moved to org-refile.el. I see that
> org-get-outline-path has nothing to do specific to refiling. Can that
> be moved back to org.el, or may be a separate library? Otherwise,
> ox-hugo.el will have to load org-refile.el too (yes, I don't use
> org-refile (yet), and that's how I discovered this :))

Yes, please move that function back.  This is going to cause breakage in
a variety of packages that use that function but do not load
org-refile.  I can hear the bug reports rumbling already...  ;)

Re: Debugging at least 2 regressions in org-mode master breaking ox-hugo

[Kaushal Modi]

[-- Attachment #1: Type: text/plain, Size: 90 bytes --]

Hello,

I'm just pinging again on this thread to bring it to attention.

Thanks.

Kaushal

[-- Attachment #2: Type: text/html, Size: 202 bytes --]

Re: Debugging at least 2 regressions in org-mode master breaking ox-hugo

[Bastien]

Hi Kaushal,

Kaushal Modi <kaushal.modi@gmail.com> writes:

> I'm just pinging again on this thread to bring it to attention.

I'm reading the list but I didn't find the time to reply to the
threads yet, I'll get back to this.

Thanks,

-- 
 Bastien

Re: Debugging at least 2 regressions in org-mode master breaking ox-hugo

[Kaushal Modi]

[-- Attachment #1: Type: text/plain, Size: 458 bytes --]

On Thu, Mar 5, 2020 at 9:04 AM Bastien <bzg@gnu.org> wrote:

> Hi Kaushal,
>
> Kaushal Modi <kaushal.modi@gmail.com> writes:
>
> > I'm just pinging again on this thread to bring it to attention.
>
> I'm reading the list but I didn't find the time to reply to the
> threads yet, I'll get back to this.
>
> Thanks,
>

Thanks, no hurry. I just wanted to make sure that these issues are read
before Org 9.4 drops, and also that regression in Org maint is noted.

[-- Attachment #2: Type: text/html, Size: 870 bytes --]

Re: Debugging at least 2 regressions in org-mode master breaking ox-hugo

[Kyle Meyer]

Adam Porter <adam@alphapapa.net> writes:

> Kaushal Modi <kaushal.modi@gmail.com> writes:
>
>> Failure 1: org-get-outline-path has moved, and not mentioned in ORG-NEWS
>>
>> Compiling ox-hugo.el now gives:
>>
>> ox-hugo.el:4284:1: Warning: the function ‘org-get-outline-path’ is not known to be defined.
>>
>> I see that defun has now moved to org-refile.el. I see that
>> org-get-outline-path has nothing to do specific to refiling. Can that
>> be moved back to org.el, or may be a separate library? Otherwise,
>> ox-hugo.el will have to load org-refile.el too (yes, I don't use
>> org-refile (yet), and that's how I discovered this :))
>
> Yes, please move that function back.  This is going to cause breakage in
> a variety of packages that use that function but do not load
> org-refile.  I can hear the bug reports rumbling already...  ;)

Despite being sympathetic to any attempt to break up org.el, I agree
that it'd be good to move the outline path functionality back.  Also,
there are a number of loading issues related to the org-refile move,
which can be seen by running `make single'.

I've prepared a commit that resolves these issues, which includes moving
org-get-outline-path and friends back to org.el, on the km/refile-fixups
branch:

  https://code.orgmode.org/bzg/org-mode/commit/18e58aa0d7fd367b3506891b633a493f402e9fee

(It's not very useful to post it as a patch here because most of it's
code movement that's better viewed with git-diff's --color-moved.)

Re: Debugging at least 2 regressions in org-mode master breaking ox-hugo

[Daniel Kraus]

Hi!

Kyle Meyer <kyle@kyleam.com> writes:

> Despite being sympathetic to any attempt to break up org.el, I agree
> that it'd be good to move the outline path functionality back.  Also,
> there are a number of loading issues related to the org-refile move,
> which can be seen by running `make single'.
>
> I've prepared a commit that resolves these issues, which includes moving
> org-get-outline-path and friends back to org.el, on the km/refile-fixups
> branch:
>
>   https://code.orgmode.org/bzg/org-mode/commit/18e58aa0d7fd367b3506891b633a493f402e9fee

Just want to say that I'm (like everyone else who uses the Emacs `native-comp` branch
with org-mode from master) are also affected by this and
would appreciate if this can be merged.

Thanks for everyone involved :)
-Daniel

Re: Debugging at least 2 regressions in org-mode master breaking ox-hugo

[Carsten Dominik]

[-- Attachment #1: Type: text/plain, Size: 1079 bytes --]

On Fri, Feb 28, 2020 at 1:04 AM Adam Porter <adam@alphapapa.net> wrote:

> Kaushal Modi <kaushal.modi@gmail.com> writes:
>
> > Failure 1: org-get-outline-path has moved, and not mentioned in ORG-NEWS
> >
> > Compiling ox-hugo.el now gives:
> >
> > ox-hugo.el:4284:1: Warning: the function ‘org-get-outline-path’ is not
> known to be defined.
> >
> > I see that defun has now moved to org-refile.el. I see that
> > org-get-outline-path has nothing to do specific to refiling. Can that
> > be moved back to org.el, or may be a separate library? Otherwise,
> > ox-hugo.el will have to load org-refile.el too (yes, I don't use
> > org-refile (yet), and that's how I discovered this :))
>
> Yes, please move that function back.  This is going to cause breakage in
> a variety of packages that use that function but do not load
> org-refile.  I can hear the bug reports rumbling already...  ;)
>

I support this. getting the outline path has more general applications that
just refiling, so tugging it away into org-refile does not make sense.

- Carsten

[-- Attachment #2: Type: text/html, Size: 1576 bytes --]

Re: Debugging at least 2 regressions in org-mode master breaking ox-hugo

[Kyle Meyer]

Daniel Kraus <daniel@kraus.my> writes:

> Just want to say that I'm (like everyone else who uses the Emacs `native-comp` branch
> with org-mode from master) are also affected by this and
> would appreciate if this can be merged.

I'll plan to bring that commit into master tomorrow.  We can
reevaluate/rework the change if needed when Bastien is back in action.

Re: Debugging at least 2 regressions in org-mode master breaking ox-hugo

[Kyle Meyer]

Kyle Meyer <kyle@kyleam.com> writes:

> I'll plan to bring that commit into master tomorrow.  We can
> reevaluate/rework the change if needed when Bastien is back in action.

Done with 3c31941139ed6de14aebee950141dabbd7c0b468.

Re: Debugging at least 2 regressions in org-mode master breaking ox-hugo

[Bastien]

Hi Kyle and everyone,

thanks for taking care of this issue.

I have taken two days off on thursday and friday to be able to
contribute to fixing remaining issues before 9.4.

Best,

-- 
 Bastien

Re: Debugging at least 2 regressions in org-mode master breaking ox-hugo

[Carsten Dominik]

[-- Attachment #1: Type: text/plain, Size: 2111 bytes --]

Hi everyone,

I just tried to archive something and hit again the issue that
org-get-outline-paths undefined.  Is there a specific holdup why this
function has not been moved back into org.el?

Thanks

Carsten


On Thu, Feb 27, 2020 at 3:02 PM Kaushal Modi <kaushal.modi@gmail.com> wrote:

> Hello,
>
> I recently updated to the latest org-mode master and it is failing
> ox-hugo[1] build and tests at 2 places.
>
> Failure 1: org-get-outline-path has moved, and not mentioned in ORG-NEWS
>
> Compiling ox-hugo.el now gives:
>
> ox-hugo.el:4284:1: Warning: the function ‘org-get-outline-path’ is not
> known to be defined.
>
> I see that defun has now moved to org-refile.el. I see that
> org-get-outline-path has nothing to do specific to refiling. Can that be
> moved back to org.el, or may be a separate library? Otherwise, ox-hugo.el
> will have to load org-refile.el too (yes, I don't use org-refile (yet), and
> that's how I discovered this :))
>
> Failure 2: Change in parsing of org babel header arguments.
>
> This was caught by my weekly Travis CI cron jobs for ox-hugo:
> https://travis-ci.org/kaushalmodi/ox-hugo/jobs/655410731#L2426
>
> 26c26
> < {{< highlight emacs-lisp "hl_lines=1" >}}
> ---
> > {{< highlight emacs-lisp "hl_lines=1 3-5" >}}
>
> Earlier this kind of src block header:
>
> #+begin_src emacs-lisp :hl_lines 1,3-5
> ...
> #+end_src
>
> got exported as
>
> {{< highlight emacs-lisp "hl_lines=1 3-5" >}}
>
> The regression is that now it is getting exported as
>
> {{< highlight emacs-lisp "hl_lines=1" >}}
>
> The values that I have after the comma in ":hl_lines 1,3-5" are getting
> lost.
>
> The relevant snippet where I parse the header arguments in ox-hugo.el is
> at
> https://github.com/kaushalmodi/ox-hugo/blob/f8ec4aa5ad7d92f94bd8dbb814d85f980be67aea/ox-hugo.el#L2563
>
> This behavior change in org-babel-parse-header-arguments is also not
> documented in ORG-NEWS. I will now investigate what cause this regression.
>
> ...
>
> --
> Kaushal Modi
>
> [1]: https://github.com/kaushalmodi/ox-hugo
>

[-- Attachment #2: Type: text/html, Size: 3316 bytes --]

Re: Debugging at least 2 regressions in org-mode master breaking ox-hugo

[Kyle Meyer]

Carsten Dominik writes:

> I just tried to archive something and hit again the issue that
> org-get-outline-paths undefined.  Is there a specific holdup why this
> function has not been moved back into org.el?

org-get-outline-path was moved back to org.el in 3c3194113 (2020-04-26).

  % git rev-parse master
  20c13221942183290dc440ca6ba91597f243b9e7
  % git grep 'defun org-get-outline-path' master
  master:lisp/org.el:(defun org-get-outline-path (&optional with-self use-cache)

Re: Debugging at least 2 regressions in org-mode master breaking ox-hugo

[Carsten Dominik]

[-- Attachment #1: Type: text/plain, Size: 649 bytes --]

Strange.  I did pull and compile, to no avail.

I will check.

Carsten

On Mon, May 11, 2020 at 4:27 PM Kyle Meyer <kyle@kyleam.com> wrote:

> Carsten Dominik writes:
>
> > I just tried to archive something and hit again the issue that
> > org-get-outline-paths undefined.  Is there a specific holdup why this
> > function has not been moved back into org.el?
>
> org-get-outline-path was moved back to org.el in 3c3194113 (2020-04-26).
>
>   % git rev-parse master
>   20c13221942183290dc440ca6ba91597f243b9e7
>   % git grep 'defun org-get-outline-path' master
>   master:lisp/org.el:(defun org-get-outline-path (&optional with-self
> use-cache)
>

[-- Attachment #2: Type: text/html, Size: 1034 bytes --]

Re: Debugging at least 2 regressions in org-mode master breaking ox-hugo

[Bastien]

Hi Kyle,

Kyle Meyer <kyle@kyleam.com> writes:

> Daniel Kraus <daniel@kraus.my> writes:
> 
>> Just want to say that I'm (like everyone else who uses the Emacs `native-comp` branch
>> with org-mode from master) are also affected by this and
>> would appreciate if this can be merged.
>
> I'll plan to bring that commit into master tomorrow.  We can
> reevaluate/rework the change if needed when Bastien is back in action.

Thanks a lot for taking care of this, and thanks Kaushal for raising
this issue.

-- 
 Bastien

Re: Debugging at least 2 regressions in org-mode master breaking ox-hugo

[Bastien]

Hi Kaushal,

sorry for the late reply, and thanks for the detailed report.

Kaushal Modi <kaushal.modi@gmail.com> writes:

> *The regression is that earlier (org-babel--string-to-number "1,3-5")
> used to return nil, but now it returns 1.*

This should be fixed now in master, as of 15a6836e4, it will be
in Org 9.4.

Best,

-- 
 Bastien

Re: Debugging at least 2 regressions in org-mode master breaking ox-hugo

[Kaushal Modi]

[-- Attachment #1: Type: text/plain, Size: 564 bytes --]

On Sun, Sep 6, 2020, 5:37 PM Bastien <bzg@gnu.org> wrote:

> Hi Kaushal,
>
> sorry for the late reply, and thanks for the detailed report.
>
> Kaushal Modi <kaushal.modi@gmail.com> writes:
>
> > *The regression is that earlier (org-babel--string-to-number "1,3-5")
> > used to return nil, but now it returns 1.*
>
> This should be fixed now in master, as of 15a6836e4, it will be
> in Org 9.4.


Thank you very much! For now, I have just overridden that Org Babel
function in ox-hugo with a package-local fix. Once the maint is fixed, I'll
remove that workaround.

[-- Attachment #2: Type: text/html, Size: 1029 bytes --]