[BUG] org-link-search fails if search string contains new lines

[BUG] org-link-search fails if search string contains new lines

[Matt Lundin]

Following links fails to locate the appropriate location in text files
if the search string in the link contains new lines.

Steps to reproduce:

/usr/bin/emacs -Q ~/config/minimal.el

where minimal.el contains...

--8<---------------cut here---------------start------------->8---
(add-to-list 'load-path "~/org-mode/lisp/")
(add-to-list 'load-path "~/org-mode/contrib/lisp/")

(setq org-agenda-files '("~/config/test.org"))

(setq org-capture-templates
      '(("n" "Note" entry
	           (file "~/config/test.org")
                   "* Test\n %a\n")))
--8<---------------cut here---------------end--------------->8---

Open a file (~/text.txt) containing....

--8<---------------cut here---------------start------------->8---
Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do
eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut enim ad
minim veniam, quis nostrud exercitation ullamco laboris nisi ut
aliquip ex ea commodo consequat. Duis aute irure dolor in
reprehenderit in voluptate velit esse cillum dolore eu fugiat nulla
pariatur. Excepteur sint occaecat cupidatat non proident, sunt in
culpa qui officia deserunt mollit anim id est laborum.
--8<---------------cut here---------------end--------------->8---

Create an active region covering the third sentence and select the
org-capture "n" template, which creates the following link:

--8<---------------cut here---------------start------------->8---
* Test
 [[file:~/test.txt::Duis%20aute%20irure%20dolor%20in%0Areprehenderit%20in%20voluptate%20velit%20esse%20cillum%20dolore%20eu%20fugiat%20nulla%0Apariatur.]]
--8<---------------cut here---------------end--------------->8---

Try to follow the link. It will open test.txt, but it will also give the
message and fail to locate the correct position in the file:

        "No match for fuzzy expression: Duis aute irure dolor in
        reprehenderit in voluptate velit esse cillum dolore eu fugiat
        nulla pariatur."

For comparison, evaluate (setq org-context-in-file-links 1) and run
capture again. The captured snippet looks like this:

--8<---------------cut here---------------start------------->8---
* Test
 [[file:~/test.txt::Duis%20aute%20irure%20dolor%20in]]
--8<---------------cut here---------------end--------------->8---

Follow the link. The fuzzy search succeeds and finds the correct
position in the text.

Matt

Re: [BUG] org-link-search fails if search string contains new lines

[Matt Lundin]

Matt Lundin <mdl@imapmail.org> writes:

>
> Create an active region covering the third sentence and select the
> org-capture "n" template, which creates the following link:
>
> * Test
>  [[file:~/test.txt::Duis%20aute%20irure%20dolor%20in%0Areprehenderit%20in%20voluptate%20velit%20esse%20cillum%20dolore%20eu%20fugiat%20nulla%0Apariatur.]]
>
> Try to follow the link. It will open test.txt, but it will also give the
> message and fail to locate the correct position in the file:
>
>         "No match for fuzzy expression: Duis aute irure dolor in
>         reprehenderit in voluptate velit esse cillum dolore eu fugiat
>         nulla pariatur."
>

The problem, I think, is the regexp construction in org-link-search.
This was introduced back in August of 2015 with commit
cfe5bc97f8b18ccbf49d0764746c7563ce8d29da.

The problematic line in org.el is 10951:

(s-multi-re (mapconcat #'regexp-quote words "[ \t]+\\(?:\n[ \t]*\\)?"))

The constructed regexp fails because it assumes a newline will be
preceded by whitespace. But often newlines are not preceded by
whitespace.

Is there a reason the following won't work? 

(s-multi-re (mapconcat #'regexp-quote words "[ \t\r\n]+"))

This was the method org-link-search used prior to the commit above. Are
we trying to avoid matching across blank lines? 

Best,
Matt

Re: [BUG] org-link-search fails if search string contains new lines

[Nicolas Goaziou]

Hello,

Matt Lundin <mdl@imapmail.org> writes:

> Matt Lundin <mdl@imapmail.org> writes:
>
>>
>> Create an active region covering the third sentence and select the
>> org-capture "n" template, which creates the following link:
>>
>> * Test
>>  [[file:~/test.txt::Duis%20aute%20irure%20dolor%20in%0Areprehenderit%20in%20voluptate%20velit%20esse%20cillum%20dolore%20eu%20fugiat%20nulla%0Apariatur.]]
>>
>> Try to follow the link. It will open test.txt, but it will also give the
>> message and fail to locate the correct position in the file:
>>
>>         "No match for fuzzy expression: Duis aute irure dolor in
>>         reprehenderit in voluptate velit esse cillum dolore eu fugiat
>>         nulla pariatur."
>>
>
> The problem, I think, is the regexp construction in org-link-search.
> This was introduced back in August of 2015 with commit
> cfe5bc97f8b18ccbf49d0764746c7563ce8d29da.
>
> The problematic line in org.el is 10951:
>
> (s-multi-re (mapconcat #'regexp-quote words "[ \t]+\\(?:\n[ \t]*\\)?"))
>
> The constructed regexp fails because it assumes a newline will be
> preceded by whitespace. But often newlines are not preceded by
> whitespace.
>
> Is there a reason the following won't work? 
>
> (s-multi-re (mapconcat #'regexp-quote words "[ \t\r\n]+"))
>
> This was the method org-link-search used prior to the commit above. Are
> we trying to avoid matching across blank lines?

Yes, we are.

Fixed, hopefully.

Regards,

-- 
Nicolas Goaziou

Re: [BUG] org-link-search fails if search string contains new lines

[Matt Lundin]

Nicolas Goaziou <mail@nicolasgoaziou.fr> writes:

>> The problem, I think, is the regexp construction in org-link-search.
>> This was introduced back in August of 2015 with commit
>> cfe5bc97f8b18ccbf49d0764746c7563ce8d29da.
>>
>> The problematic line in org.el is 10951:
>>
>> (s-multi-re (mapconcat #'regexp-quote words "[ \t]+\\(?:\n[ \t]*\\)?"))
>>
>> The constructed regexp fails because it assumes a newline will be
>> preceded by whitespace. But often newlines are not preceded by
>> whitespace.
>>
>> Is there a reason the following won't work? 
>>
>> (s-multi-re (mapconcat #'regexp-quote words "[ \t\r\n]+"))
>>
>> This was the method org-link-search used prior to the commit above. Are
>> we trying to avoid matching across blank lines?
>
> Yes, we are.
>
> Fixed, hopefully.

Thanks Nicolas. Yes, it now works with multi-line searches that do not
contain blank lines.

I think there are some instances in which one might want to search
across blank lines, such as when one captures a region that contains one
or more blank lines. E.g., let's say org-context-in-file-links is t (the
default). If one captures a section in org.el containing a blank line,
org-link-search will fail to locate the section.

Here's a sample link to try it out. (It assumes an org-mode repository
at ~/org-mode).

 [[file:~/org-mode/lisp/org.el::(defsubst%20org-uniquify%20(list)%0A%20"Non-destructively%20remove%20duplicate%20elements%20from%20LIST."%0A%20(let%20((res%20(copy-sequence%20list)))%20(delete-dups%20res)))%0A%0A(defsubst%20org-get-at-bol%20(property)%0A%20"Get%20text%20property%20PROPERTY%20at%20the%20beginning%20of%20line."%0A%20(get-text-property%20(point-at-bol)%20property))%0A]]

Thanks,
Matt

Re: [BUG] org-link-search fails if search string contains new lines

[Nicolas Goaziou]

Hello,

Matt Lundin <mdl@imapmail.org> writes:

> Thanks Nicolas. Yes, it now works with multi-line searches that do not
> contain blank lines.
>
> I think there are some instances in which one might want to search
> across blank lines, such as when one captures a region that contains one
> or more blank lines. E.g., let's say org-context-in-file-links is t (the
> default). If one captures a section in org.el containing a blank line,
> org-link-search will fail to locate the section.
>
> Here's a sample link to try it out. (It assumes an org-mode repository
> at ~/org-mode).
>
>  [[file:~/org-mode/lisp/org.el::(defsubst%20org-uniquify%20(list)%0A%20"Non-destructively%20remove%20duplicate%20elements%20from%20LIST."%0A%20(let%20((res%20(copy-sequence%20list)))%20(delete-dups%20res)))%0A%0A(defsubst%20org-get-at-bol%20(property)%0A%20"Get%20text%20property%20PROPERTY%20at%20the%20beginning%20of%20line."%0A%20(get-text-property%20(point-at-bol)%20property))%0A]]

Fixed. Thank you.

Regards,

-- 
Nicolas Goaziou                                                0x80A93738

Re: [BUG] org-link-search fails if search string contains new lines

[Skip Collins]

On Mon, Mar 27, 2017 at 7:37 AM, Nicolas Goaziou <mail@nicolasgoaziou.fr> wrote:
> Fixed, hopefully.

Ran 718 tests, 717 results as expected, 1 unexpected (2017-03-28 17:27:55-0400)

9 expected failures


1 unexpected results:

   FAILED  test-org/fuzzy-links

Re: [BUG] org-link-search fails if search string contains new lines

[Nicolas Goaziou]

Hello,

Skip Collins <skip.collins@gmail.com> writes:

> 1 unexpected results:
>
>    FAILED  test-org/fuzzy-links

This is not the case at http://www.randomsample.de:4457/waterfall

The error report would help.

Regards,

-- 
Nicolas Goaziou                                                0x80A93738

Re: [BUG] org-link-search fails if search string contains new lines

[Skip Collins]

Test test-org/fuzzy-links backtrace:
  (if (unwind-protect (setq value-10839 (let ((file (make-temp-file "o
  (let (form-description-10840) (if (unwind-protect (setq value-10839
  (let ((value-10839 (cl-gensym "ert-form-evaluation-aborted-"))) (let
  (lambda nil (let ((value-10802 (cl-gensym "ert-form-evaluation-abort
  ert--run-test-internal([cl-struct-ert--test-execution-info [cl-struc
  ert-run-test([cl-struct-ert-test test-org/fuzzy-links "Test fuzzy li
  ert-run-or-rerun-test([cl-struct-ert--stats "\\(org\\|ob\\)" [[cl-st
  ert-run-tests("\\(org\\|ob\\)" #[385 "\306\307\"\203G\211\211G\310
  ert-run-tests-batch("\\(org\\|ob\\)")
  ert-run-tests-batch-and-exit("\\(org\\|ob\\)")
  (let ((org-id-track-globally t) (org-test-selector (if org-test-sele
  org-test-run-batch-tests("\\(org\\|ob\\)")
  eval((org-test-run-batch-tests org-test-select-re))
  command-line-1(("--eval" "(setq vc-handled-backends nil org-startup-
  command-line()
  normal-top-level()
Test test-org/fuzzy-links condition:
    (ert-test-failed
     ((should
       (org-test-with-temp-text-in-file "Paragraph

line1
line2

"
(let ... ... ... ... ... ...)))
      :form
      (let
 ((file ...)
  (kill-buffer-query-functions nil)
  (inside-text ...)
  G10838)
(with-temp-file file
 (insert inside-text))
(find-file file)
(org-mode)
(setq G10838
     (progn ...))
(save-buffer)
(kill-buffer
(current-buffer))
(delete-file file)
G10838)
      :value nil))
   FAILED  656/718  test-org/fuzzy-links

Re: [BUG] org-link-search fails if search string contains new lines

[Nicolas Goaziou]

Hello,

Skip Collins <skip.collins@gmail.com> writes:

> Test test-org/fuzzy-links backtrace:
>   (if (unwind-protect (setq value-10839 (let ((file (make-temp-file "o
>   (let (form-description-10840) (if (unwind-protect (setq value-10839
>   (let ((value-10839 (cl-gensym "ert-form-evaluation-aborted-"))) (let
>   (lambda nil (let ((value-10802 (cl-gensym "ert-form-evaluation-abort
>   ert--run-test-internal([cl-struct-ert--test-execution-info [cl-struc
>   ert-run-test([cl-struct-ert-test test-org/fuzzy-links "Test fuzzy li
>   ert-run-or-rerun-test([cl-struct-ert--stats "\\(org\\|ob\\)" [[cl-st
>   ert-run-tests("\\(org\\|ob\\)" #[385 "\306\307\"\203G\211\211G\310
>   ert-run-tests-batch("\\(org\\|ob\\)")
>   ert-run-tests-batch-and-exit("\\(org\\|ob\\)")
>   (let ((org-id-track-globally t) (org-test-selector (if org-test-sele
>   org-test-run-batch-tests("\\(org\\|ob\\)")
>   eval((org-test-run-batch-tests org-test-select-re))
>   command-line-1(("--eval" "(setq vc-handled-backends nil org-startup-
>   command-line()
>   normal-top-level()
> Test test-org/fuzzy-links condition:
>     (ert-test-failed
>      ((should
>        (org-test-with-temp-text-in-file "Paragraph
>
> line1
> line2
>
> "
> (let ... ... ... ... ... ...)))
>       :form
>       (let
>  ((file ...)
>   (kill-buffer-query-functions nil)
>   (inside-text ...)
>   G10838)
> (with-temp-file file
>  (insert inside-text))
> (find-file file)
> (org-mode)
> (setq G10838
>      (progn ...))
> (save-buffer)
> (kill-buffer
> (current-buffer))
> (delete-file file)
> G10838)
>       :value nil))
>    FAILED  656/718  test-org/fuzzy-links

Thank you. Unfortunately, there is not much to do with this. Could you
do some debugging when running the faulty test? It may come from
something in your configuration.

Regards,

-- 
Nicolas Goaziou

Re: [BUG] org-link-search fails if search string contains new lines

[Skip Collins]

[-- Attachment #1: Type: text/plain, Size: 685 bytes --]

On Thu, Mar 30, 2017 at 6:50 AM, Nicolas Goaziou <mail@nicolasgoaziou.fr> wrote:
> Thank you. Unfortunately, there is not much to do with this. Could you
> do some debugging when running the faulty test? It may come from
> something in your configuration.

I think I figured it out. During the fuzzy-links test,
org-open-at-point is invoked to open a file that was created with
org-test-with-temp-text-in-file. The temp file does not have a .org
extension. As a result, my mac attempts to open the file outside of
emacs, which produces an unexpected result. The solution is to pass an
argument to org-open-at-point, which ensures that the file is opened
in emacs. A patch is attached.

[-- Attachment #2: 0001-Ensure-test-files-are-opened-in-Emacs-in-test-org-fu.patch --]
[-- Type: application/octet-stream, Size: 1125 bytes --]

From 6c3ad66280edc10a683bdfd4e730033d5f541721 Mon Sep 17 00:00:00 2001
From: Skip Collins <skip.collins@gmail.com>
Date: Fri, 7 Apr 2017 11:41:47 -0400
Subject: [PATCH] Ensure test files are opened in Emacs in test-org/fuzzy-links

---
 testing/lisp/test-org.el | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/testing/lisp/test-org.el b/testing/lisp/test-org.el
index b1d4449ef..3361a69a5 100644
--- a/testing/lisp/test-org.el
+++ b/testing/lisp/test-org.el
@@ -2292,7 +2292,7 @@ Foo Bar
        (insert (format "[[file:%s::line1 line2]]" file))
        (beginning-of-line)
        (let ((org-link-search-must-match-exact-headline nil))
-	 (org-open-at-point))
+	 (org-open-at-point 0))
        (looking-at-p "line1"))))
   (should
    (org-test-with-temp-text-in-file "Paragraph\n\nline1\n\nline2\n\n"
@@ -2301,7 +2301,7 @@ Foo Bar
        (insert (format "[[file:%s::line1 line2]]" file))
        (beginning-of-line)
        (let ((org-link-search-must-match-exact-headline nil))
-	 (org-open-at-point))
+	 (org-open-at-point 0))
        (looking-at-p "line1")))))
 
 ;;;; Link Escaping
-- 
2.12.2