Re: [RFC] Quoting property names in tag/property matches [Was: [BUG?] Matching tags: & operator no more implicit between tags and special property]

[Jens Schmidt <jschmidt4gnu@vodafonemail.de>]

[-- Attachment #1: Type: text/plain, Size: 644 bytes --]

On 2023-08-24  10:52, Jens Schmidt wrote:
> On 2023-08-24  09:32, Ihor Radchenko wrote:

>> I prefer (B). And we will need to allow escaping of the "\" itself. Like
>> \\.
> 
> OK.  Since backslash is not used yet in the rest of the regexp, (B)
> should be rather edge-case-free.  So the corresponding subre to match
> property names would look like
> 
>   \(?5:\(?:[[:alnum:]]+\|\\[^[:space:]]\)+\)
> 
> IOW, backslash quotes everything except whitespace, which by definition
> cannot be part of a property name.
> 
> Will start on this, but with tests and documentation this might take
> some time.

Here comes a first patch ... please check.

[-- Attachment #2: 0001-org-make-tags-matcher-Re-add-quoting-of-property-nam.patch --]
[-- Type: text/x-patch, Size: 8910 bytes --]

From 830750bfa10b52ac77abe8af1f2789057f9101c1 Mon Sep 17 00:00:00 2001
From: Jens Schmidt <jschmidt4gnu@vodafonemail.de>
Date: Thu, 24 Aug 2023 22:38:02 +0200
Subject: [PATCH] org-make-tags-matcher: Re-add quoting of property names

* lisp/org.el (org-make-tags-matcher):
* testing/lisp/test-org.el (test-org/map-entries): Move special cased
handling of LEVEL properties.  Add tests for other special cased
properties TODO and CATEGORY.

* lisp/org.el (org-make-tags-matcher):
* doc/org-manual.org (Matching tags and properties):
* testing/lisp/test-org.el (test-org/map-entries): Re-add and extend
quoting of property names in search strings.

Link: https://orgmode.org/list/87h6oq2nu1.fsf@gmail.com
---
 doc/org-manual.org       | 20 +++++++--------
 lisp/org.el              | 55 ++++++++++++++++++++++++----------------
 testing/lisp/test-org.el | 34 +++++++++++++++++--------
 3 files changed, 66 insertions(+), 43 deletions(-)

diff --git a/doc/org-manual.org b/doc/org-manual.org
index 17b25fef4..10a03b344 100644
--- a/doc/org-manual.org
+++ b/doc/org-manual.org
@@ -9320,21 +9320,19 @@ With the regular =<= operator, the search would handle entries without
 an =EFFORT= property as having a zero effort and would include them in
 the result as well.
 
-Currently, you can use only property names including alphanumeric
-characters, underscores, and minus characters in search strings.  In
-addition, if you want to search for a property whose name starts with
-a minus character, you have to "quote" that leading minus character
-with an explicit positive selection plus character, like this:
+You can use all characters valid in property names when matching
+properties.  However, you have to quote some characters in property
+names with backslashes when using them in search strings, namely all
+characters different from alphanumerics and underscores[fn:: If you
+quote alphanumeric characters or underscores with a backslash, that
+backslash is ignored.].  For example, to search for all entries having
+a property =-long​=and\twisted:property.name*= with value =foo=, use
+search string
 
 #+begin_example
-+-long-and-twisted-property-name-="foo"
+\-long\=and\\twisted\:property\.name\*="foo"
 #+end_example
 
-#+texinfo: @noindent
-Without that extra plus character, the minus character would be taken
-to indicate a negative selection on search term
-=long-and-twisted-property-name-​="foo"=.
-
 You can configure Org mode to use property inheritance during
 a search, but beware that this can slow down searches considerably.
 See [[*Property Inheritance]], for details.
diff --git a/lisp/org.el b/lisp/org.el
index 50df1b2d9..78f8eb2e9 100644
--- a/lisp/org.el
+++ b/lisp/org.el
@@ -11328,15 +11328,14 @@ See also `org-scan-tags'."
               "\\(?2:"
                   ;; tag regexp match
                   "{[^}]+}\\|"
-                  ;; LEVEL property match.  For sake of consistency,
-                  ;; recognize starred operators here as well.  We do
-                  ;; not need to process them below, however, since
-                  ;; the LEVEL property is always present.
-                  "LEVEL\\(?3:" opre "\\)\\*?\\(?4:[0-9]+\\)\\|"
-                  ;; regular property match
+                  ;; property match.  Try to keep this subre generic
+                  ;; and rather handle special properties like LEVEL
+                  ;; and CATEGORY further below.  This ensures that
+                  ;; the same quoting mechanics can be used for all
+                  ;; property names.
                   "\\(?:"
                       ;; property name [1]
-                      "\\(?5:[[:alnum:]_-]+\\)"
+                      "\\(?5:\\(?:[[:alnum:]_]+\\|\\\\[^[:space:]]\\)+\\)"
                       ;; operator, optionally starred
                       "\\(?6:" opre "\\)\\(?7:\\*\\)?"
                       ;; operand (regexp, double-quoted string,
@@ -11353,13 +11352,19 @@ See also `org-scan-tags'."
          (start 0)
          tagsmatch todomatch tagsmatcher todomatcher)
 
-    ;; [1] The minus characters in property names do *not* conflict
-    ;; with the exclusion operator above, since the mandatory
-    ;; following operator distinguishes these both cases.
-    ;; Accordingly, minus characters do not need any special quoting,
-    ;; even if https://orgmode.org/list/87jzv67k3p.fsf@localhost and
-    ;; commit 19b0e03f32c6032a60150fc6cb07c6f766cb3f6c suggest
-    ;; otherwise.
+    ;; [1] The history of this particular subre:
+    ;; - \\([[:alnum:]_]+\\) [pre-19b0e03]
+    ;;   Does not allow for minus characters in property names.
+    ;; - "\\(\\(?:[[:alnum:]_]+\\(?:\\\\-\\)*\\)+\\)" [19b0e03]
+    ;;   Incomplete fix of above issue, still resulting in, e.g.,
+    ;;   https://orgmode.org/list/87jzv67k3p.fsf@localhost.
+    ;; - "\\(?5:[[:alnum:]_-]+\\)" [f689eb4]
+    ;;   Allows for unquoted minus characters in property names, but
+    ;;   conflicts with searches like -TAG-PROP="VALUE".  See
+    ;;   https://orgmode.org/list/87h6oq2nu1.fsf@gmail.com.
+    ;; - current subre
+    ;;   Like second solution, but with proper unquoting and allowing
+    ;;   for all possible characters in property names to be quoted.
 
     ;; Expand group tags.
     (setq match (org-tags-expand match))
@@ -11404,22 +11409,28 @@ See also `org-scan-tags'."
 		   ;; exact tag match in [3].
 		   (tag (match-string 2 term))
 		   (regexp (eq (string-to-char tag) ?{))
-		   (levelp (match-end 4))
 		   (propp (match-end 5))
 		   (mm
 		    (cond
 		     (regexp			; [2]
                       `(with-syntax-table org-mode-tags-syntax-table
                          (org-match-any-p ,(substring tag 1 -1) tags-list)))
-		     (levelp
-		      `(,(org-op-to-function (match-string 3 term))
-			level
-			,(string-to-number (match-string 4 term))))
 		     (propp
-		      (let* (;; Convert property name to an Elisp
+		      (let* (;; Determine property name.
+                             (pn (upcase
+                                  (save-match-data
+                                    (replace-regexp-in-string
+                                     "\\\\\\(.\\)" "\\1"
+                                     (match-string 5 term)
+                                     t nil))))
+                             ;; Convert property name to an Elisp
 			     ;; accessor for that property (aka. as
-			     ;; getter value).
-			     (gv (pcase (upcase (match-string 5 term))
+			     ;; getter value).  Symbols LEVEL and TODO
+			     ;; referenced below get bound by the
+			     ;; matcher that this function returns.
+			     (gv (pcase pn
+				   ("LEVEL"
+                                    '(number-to-string level))
 				   ("CATEGORY"
 				    '(org-get-category (point)))
 				   ("TODO" 'todo)
diff --git a/testing/lisp/test-org.el b/testing/lisp/test-org.el
index e33f500a3..8355e2d77 100644
--- a/testing/lisp/test-org.el
+++ b/testing/lisp/test-org.el
@@ -2862,11 +2862,24 @@ test <point>
      (equal '(11)
 	    (org-test-with-temp-text "* Level 1\n** Level 2"
 	      (let (org-odd-levels-only) (org-map-entries #'point "LEVEL>1")))))
-    ;; Level match with (ignored) starred operator.
+    ;; Category match.
     (should
-     (equal '(11)
-	    (org-test-with-temp-text "* Level 1\n** Level 2"
-	      (let (org-odd-levels-only) (org-map-entries #'point "LEVEL>*1")))))
+     (equal '(59)
+	    (org-test-with-temp-text "
+#+CATEGORY: foo
+
+* H1
+:PROPERTIES:
+:CATEGORY: bar
+:END:
+
+* H2"
+	      (org-map-entries #'point "CATEGORY=\"foo\""))))
+    ;; Todo match.
+    (should
+     (equal '(6)
+	    (org-test-with-temp-text "* H1\n* TODO H2\n* DONE H3"
+	      (org-map-entries #'point "TODO=\"TODO\""))))
     ;; Tag match.
     (should
      (equal '(11)
@@ -2948,7 +2961,7 @@ SCHEDULED: <2014-03-04 tue.>"
 :END:
 * H3"
 	      (org-map-entries #'point "TEST!=*1"))))
-    ;; Property matches on names including minus characters.
+    ;; Property matches on names containing quoted characters.
     (org-test-with-temp-text
      "
 * H1 :BAR:
@@ -2967,11 +2980,12 @@ SCHEDULED: <2014-03-04 tue.>"
 :PROPERTIES:
 :-FOO: 2
 :END:
-* H5"
-     (should (equal '(2) (org-map-entries #'point "TEST-FOO!=*0-FOO")))
-     (should (equal '(2) (org-map-entries #'point "-FOO+TEST-FOO!=*0")))
-     (should (equal '(88) (org-map-entries #'point "+-FOO!=*0-FOO")))
-     (should (equal '(88) (org-map-entries #'point "-FOO+-FOO!=*0"))))
+* H5 :TEST:"
+     (should (equal '(2) (org-map-entries #'point "TEST\\-FOO!=*0-FOO")))
+     (should (equal '(2) (org-map-entries #'point "-FOO+TEST\\-FOO!=*0")))
+     (should (equal '(88) (org-map-entries #'point "\\-FOO!=*0-FOO")))
+     (should (equal '(88) (org-map-entries #'point "-FOO+\\-FOO!=*0")))
+     (should (equal '(88) (org-map-entries #'point "-TEST-FOO-TEST\\-FOO=1"))))
     ;; Multiple criteria.
     (should
      (equal '(23)
-- 
2.30.2