Re: org-agenda queries for absent properties

[Jens Schmidt <jschmidt4gnu@vodafonemail.de>]

[-- Attachment #1: Type: text/plain, Size: 959 bytes --]

On 2023-08-06  11:19, Jens Schmidt wrote:

> I
> will also add tests for matching property names having minuses and then
> we can see how it goes.

Went well.  With a property name regexp

   "\\(?5:[[:alnum:]_-]+\\)"

and the following Org file

--------------------------------------------------
* H1 :BAR:
:PROPERTIES:
:TEST-FOO: 1
:END:
* H2 :FOO:
:PROPERTIES:
:TEST-FOO: 2
:END:
* H3 :BAR:
:PROPERTIES:
:-FOO: 1
:END:
* H4 :FOO:
:PROPERTIES:
:-FOO: 2
:END:
* H5
--------------------------------------------------

the search expressions "TEST-FOO!=*0-FOO" and "-FOO+TEST-FOO!=*0" work
as expected, finding only the first heading.

The expression "-FOO!=*0-FOO" does *not* work, though, since the leading
minus in the property name is gobbled by the greedy inclusion/exclusion
subexp "\\(?1:[-+:]\\)?".  However, the remedy is simple: Add an
explicit plus sign: "+-FOO!=*0-FOO".

Documented that in the Org manual, added tests, looks good.  Please
check.

[-- Attachment #2: 0001-org-make-tags-matcher-Add-starred-property-operators.patch --]
[-- Type: text/x-patch, Size: 15123 bytes --]

From cb3268c8b1c69371589a9aa44a96d33e29aedbb1 Mon Sep 17 00:00:00 2001
From: Jens Schmidt <jschmidt4gnu@vodafonemail.de>
Date: Sun, 6 Aug 2023 16:38:04 +0200
Subject: [PATCH] org-make-tags-matcher: Add starred property operators, fix
 quoting

* lisp/org.el (org-make-tags-matcher): Add starred property operators.
Recognize additional operators "==", "!=", "/=".  Clean up and
document match term parsing.  Remove needless and buggy unquoting of
minus characters in property and tag names.
(org-op-to-function): Recognize additional inequality operator "/=".

* doc/org-manual.org (Matching tags and properties): Add documentation
on starred and additional operators.  Document allowed characters in
property names and handling of minus characters in property names.

* testing/lisp/test-org.el (test-org/map-entries): Add tests for
starred and additional operators.  Add tests for property names
containing minus characters.

* etc/ORG-NEWS: (~org-tags-view~ supports more property operators):
Add announcement on starred and additional operators.

Link: https://orgmode.org/list/9132e58f-d89e-f7df-bbe4-43d53a2367d2@vodafonemail.de
---
 doc/org-manual.org       |  35 +++++++++++-
 etc/ORG-NEWS             |  10 +++-
 lisp/org.el              | 120 ++++++++++++++++++++++++++++-----------
 testing/lisp/test-org.el |  64 ++++++++++++++++++++-
 4 files changed, 192 insertions(+), 37 deletions(-)

diff --git a/doc/org-manual.org b/doc/org-manual.org
index e59efc417..4252d7ac3 100644
--- a/doc/org-manual.org
+++ b/doc/org-manual.org
@@ -9246,16 +9246,18 @@ When matching properties, a number of different operators can be used
 to test the value of a property.  Here is a complex example:
 
 #+begin_example
-+work-boss+PRIORITY="A"+Coffee="unlimited"+Effort<2
++work-boss+PRIORITY="A"+Coffee="unlimited"+Effort<*2
          +With={Sarah\|Denny}+SCHEDULED>="<2008-10-11>"
 #+end_example
 
+#+cindex: operator, for property search
 #+texinfo: @noindent
 The type of comparison depends on how the comparison value is written:
 
 - If the comparison value is a plain number, a numerical comparison is
   done, and the allowed operators are =<=, ===, =>=, =<==, =>==, and
-  =<>=.
+  =<>=.  As synonym for the equality operator there is also ====, as
+  synonyms for the inequality operator there are =!== and =/==.
 
 - If the comparison value is enclosed in double-quotes, a string
   comparison is done, and the same operators are allowed.
@@ -9273,6 +9275,13 @@ The type of comparison depends on how the comparison value is written:
   is performed, with === meaning that the regexp matches the property
   value, and =<>= meaning that it does not match.
 
+- All operators may be optionally followed by an asterisk =*=, like in
+  =<*=, =!=*=, etc.  Such /starred operators/ work like their regular,
+  unstarred counterparts except that they match only headlines where
+  the tested property is actually present.  This is most useful for
+  search terms that logically exclude results, like the inequality
+  operator.
+
 So the search string in the example finds entries tagged =work= but
 not =boss=, which also have a priority value =A=, a =Coffee= property
 with the value =unlimited=, an =EFFORT= property that is numerically
@@ -9280,6 +9289,28 @@ smaller than 2, a =With= property that is matched by the regular
 expression =Sarah\|Denny=, and that are scheduled on or after October
 11, 2008.
 
+Note that the test on the =EFFORT= property uses operator =<*=, so
+that the search result will include only entries that actually have an
+=EFFORT= property defined and with numerical value smaller than 2.
+With the regular =<= operator, the search would handle entries without
+an =EFFORT= property as having a zero effort and would include them in
+the result as well.
+
+Currently, you can use only property names including alphanumeric
+characters, underscores, and minus characters in search strings.  In
+addition, if you want to search for a property whose name starts with
+a minus character, you have to "quote" that leading minus character
+with an explicit positive selection plus character, like this:
+
+#+begin_example
++-long-and-twisted-property-name-="foo"
+#+end_example
+
+#+texinfo: @noindent
+Without that extra plus character, the minus character would be taken
+to indicate a negative selection on search term
+=long-and-twisted-property-name-​="foo"=.
+
 You can configure Org mode to use property inheritance during
 a search, but beware that this can slow down searches considerably.
 See [[*Property Inheritance]], for details.
diff --git a/etc/ORG-NEWS b/etc/ORG-NEWS
index 4f16eda24..10c51e354 100644
--- a/etc/ORG-NEWS
+++ b/etc/ORG-NEWS
@@ -125,7 +125,7 @@ New functions to retrieve and set (via ~setf~) commonly used element properties:
 - =:contents-post-affiliated= :: ~org-element-post-affiliated~
 - =:contents-post-blank= :: ~org-element-post-blank~
 - =:parent= :: ~org-element-parent~
- 
+
 ***** New macro ~org-element-with-enabled-cache~
 
 The macro arranges the element cache to be active during =BODY= execution.
@@ -558,6 +558,14 @@ special repeaters ~++~ and ~.+~ are skipped.
 A capture template can target ~(here)~ which is the equivalent of
 invoking a capture template with a zero prefix.
 
+*** ~org-tags-view~ supports more property operators
+
+It supports inequality operators ~!=~ and ~/=~ in addition to the less
+common (BASIC?  Pascal?  SQL?) ~<>~.  And it supports starred versions
+of all relational operators (~<*~, ~=*~, ~!=*~, etc.) that work like
+the regular, unstarred operators but match a headline only if the
+tested property is actually present.
+
 ** New functions and changes in function arguments
 *** =TYPES= argument in ~org-element-lineage~ can now be a symbol
 
diff --git a/lisp/org.el b/lisp/org.el
index ed75f3edb..9ee3f98ba 100644
--- a/lisp/org.el
+++ b/lisp/org.el
@@ -11304,15 +11304,50 @@ See also `org-scan-tags'."
 	     "Match: "
 	     'org-tags-completion-function nil nil nil 'org-tags-history))))
 
-  (let ((match0 match)
-	(re (concat
-	     "^&?\\([-+:]\\)?\\({[^}]+}\\|LEVEL\\([<=>]\\{1,2\\}\\)"
-	     "\\([0-9]+\\)\\|\\(\\(?:[[:alnum:]_]+\\(?:\\\\-\\)*\\)+\\)"
-	     "\\([<>=]\\{1,2\\}\\)"
-	     "\\({[^}]+}\\|\"[^\"]*\"\\|-?[.0-9]+\\(?:[eE][-+]?[0-9]+\\)?\\)"
-	     "\\|" org-tag-re "\\)"))
-	(start 0)
-	tagsmatch todomatch tagsmatcher todomatcher)
+  (let* ((match0 match)
+         (opre "[<=>]=?\\|[!/]=\\|<>")
+         (re (concat
+              "^"
+              ;; implicit AND operator (OR is done by global splitting)
+              "&?"
+              ;; exclusion and inclusion (the latter being implicit)
+              "\\(?1:[-+:]\\)?"
+              ;; query term
+              "\\(?2:"
+                  ;; tag regexp match
+                  "{[^}]+}\\|"
+                  ;; LEVEL property match.  For sake of consistency,
+                  ;; recognize starred operators here as well.  We do
+                  ;; not need to process them below, however, since
+                  ;; the LEVEL property is always present.
+                  "LEVEL\\(?3:" opre "\\)\\*?\\(?4:[0-9]+\\)\\|"
+                  ;; regular property match
+                  "\\(?:"
+                      ;; property name [1]
+                      "\\(?5:[[:alnum:]_-]+\\)"
+                      ;; operator, optionally starred
+                      "\\(?6:" opre "\\)\\(?7:\\*\\)?"
+                      ;; operand (regexp, double-quoted string,
+                      ;; number)
+                      "\\(?8:"
+                          "{[^}]+}\\|"
+                          "\"[^\"]*\"\\|"
+                          "-?[.0-9]+\\(?:[eE][-+]?[0-9]+\\)?"
+                      "\\)"
+                  "\\)\\|"
+                  ;; exact tag match
+                  org-tag-re
+              "\\)"))
+         (start 0)
+         tagsmatch todomatch tagsmatcher todomatcher)
+
+    ;; [1] The minus characters in property names do *not* conflict
+    ;; with the exclusion operator above, since the mandatory
+    ;; following operator distinguishes these both cases.
+    ;; Accordingly, minus characters do not need any special quoting,
+    ;; even if https://orgmode.org/list/87jzv67k3p.fsf@localhost and
+    ;; commit 19b0e03f32c6032a60150fc6cb07c6f766cb3f6c suggest
+    ;; otherwise.
 
     ;; Expand group tags.
     (setq match (org-tags-expand match))
@@ -11352,15 +11387,16 @@ See also `org-scan-tags'."
 	    (let* ((rest (substring term (match-end 0)))
 		   (minus (and (match-end 1)
 			       (equal (match-string 1 term) "-")))
-		   (tag (save-match-data
-			  (replace-regexp-in-string
-			   "\\\\-" "-" (match-string 2 term))))
+		   ;; Bind the whole query term to `tag' and use that
+		   ;; variable for a tag regexp match in [2] or as an
+		   ;; exact tag match in [3].
+		   (tag (match-string 2 term))
 		   (regexp (eq (string-to-char tag) ?{))
 		   (levelp (match-end 4))
 		   (propp (match-end 5))
 		   (mm
 		    (cond
-		     (regexp
+		     (regexp			; [2]
                       `(with-syntax-table org-mode-tags-syntax-table
                          (org-match-any-p ,(substring tag 1 -1) tags-list)))
 		     (levelp
@@ -11368,28 +11404,46 @@ See also `org-scan-tags'."
 			level
 			,(string-to-number (match-string 4 term))))
 		     (propp
-		      (let* ((gv (pcase (upcase (match-string 5 term))
+		      (let* (;; Convert property name to an Elisp
+			     ;; accessor for that property (aka. as
+			     ;; getter value?).
+			     (gv (pcase (upcase (match-string 5 term))
 				   ("CATEGORY"
 				    '(org-get-category (point)))
 				   ("TODO" 'todo)
 				   (p `(org-cached-entry-get nil ,p))))
-			     (pv (match-string 7 term))
+			     ;; Determine operand (aka. property
+			     ;; value).
+			     (pv (match-string 8 term))
+			     ;; Determine type of operand.  Note that
+			     ;; these are not exclusive: Any TIMEP is
+			     ;; also STRP.
 			     (regexp (eq (string-to-char pv) ?{))
 			     (strp (eq (string-to-char pv) ?\"))
 			     (timep (string-match-p "^\"[[<]\\(?:[0-9]+\\|now\\|today\\|tomorrow\\|[+-][0-9]+[dmwy]\\).*[]>]\"$" pv))
+			     ;; Massage operand.  TIMEP must come
+			     ;; before STRP.
+			     (pv (cond (regexp (substring pv 1 -1))
+				       (timep  (org-matcher-time
+						(substring pv 1 -1)))
+				       (strp   (substring pv 1 -1))
+				       (t      pv)))
+			     ;; Convert operator to Elisp.
 			     (po (org-op-to-function (match-string 6 term)
-						     (if timep 'time strp))))
-			(setq pv (if (or regexp strp) (substring pv 1 -1) pv))
-			(when timep (setq pv (org-matcher-time pv)))
-			(cond ((and regexp (eq po '/=))
-			       `(not (string-match ,pv (or ,gv ""))))
-			      (regexp `(string-match ,pv (or ,gv "")))
-			      (strp `(,po (or ,gv "") ,pv))
-			      (t
-			       `(,po
-				 (string-to-number (or ,gv ""))
-				 ,(string-to-number pv))))))
-		     (t `(member ,tag tags-list)))))
+						     (if timep 'time strp)))
+			     ;; Convert whole property term to Elisp.
+			     (pt (cond ((and regexp (eq po '/=))
+					`(not (string-match ,pv (or ,gv ""))))
+				       (regexp `(string-match ,pv (or ,gv "")))
+				       (strp `(,po (or ,gv "") ,pv))
+				       (t
+					`(,po
+					  (string-to-number (or ,gv ""))
+					  ,(string-to-number pv)))))
+			     ;; Respect the star after the operand.
+			     (pt (if (match-end 7) `(and ,gv ,pt) pt)))
+			pt))
+		     (t `(member ,tag tags-list))))) ; [3]
 	      (push (if minus `(not ,mm) mm) tagsmatcher)
 	      (setq term rest)))
 	  (push `(and ,@tagsmatcher) orlist)
@@ -11520,12 +11574,12 @@ the list of tags in this group."
   "Turn an operator into the appropriate function."
   (setq op
 	(cond
-	 ((equal  op   "<"       ) '(<     org-string<  org-time<))
-	 ((equal  op   ">"       ) '(>     org-string>  org-time>))
-	 ((member op '("<=" "=<")) '(<=    org-string<= org-time<=))
-	 ((member op '(">=" "=>")) '(>=    org-string>= org-time>=))
-	 ((member op '("="  "==")) '(=     string=      org-time=))
-	 ((member op '("<>" "!=")) '(/=    org-string<> org-time<>))))
+	 ((equal  op   "<"            ) '(<     org-string<  org-time<))
+	 ((equal  op   ">"            ) '(>     org-string>  org-time>))
+	 ((member op '("<=" "=<"     )) '(<=    org-string<= org-time<=))
+	 ((member op '(">=" "=>"     )) '(>=    org-string>= org-time>=))
+	 ((member op '("="  "=="     )) '(=     string=      org-time=))
+	 ((member op '("<>" "!=" "/=")) '(/=    org-string<> org-time<>))))
   (nth (if (eq stringp 'time) 2 (if stringp 1 0)) op))
 
 (defvar org-add-colon-after-tag-completion nil)  ;; dynamically scoped param
diff --git a/testing/lisp/test-org.el b/testing/lisp/test-org.el
index 890ea6a8c..7c85da9d5 100644
--- a/testing/lisp/test-org.el
+++ b/testing/lisp/test-org.el
@@ -2833,6 +2833,11 @@ test <point>
      (equal '(11)
 	    (org-test-with-temp-text "* Level 1\n** Level 2"
 	      (let (org-odd-levels-only) (org-map-entries #'point "LEVEL>1")))))
+    ;; Level match with (ignored) starred operator.
+    (should
+     (equal '(11)
+	    (org-test-with-temp-text "* Level 1\n** Level 2"
+	      (let (org-odd-levels-only) (org-map-entries #'point "LEVEL>*1")))))
     ;; Tag match.
     (should
      (equal '(11)
@@ -2845,12 +2850,17 @@ test <point>
     (should
      (equal '(11 23)
 	    (org-test-with-temp-text "* H1 :no:\n* H2 :yes1:\n* H3 :yes2:"
-	      (org-map-entries #'point "{yes?}"))))
+	      (org-map-entries #'point "{yes.?}"))))
     ;; Priority match.
     (should
      (equal '(1)
 	    (org-test-with-temp-text "* [#A] H1\n* [#B] H2"
 	      (org-map-entries #'point "PRIORITY=\"A\""))))
+    ;; Negative priority match.
+    (should
+     (equal '(11)
+	    (org-test-with-temp-text "* [#A] H1\n* [#B] H2"
+	      (org-map-entries #'point "PRIORITY/=\"A\""))))
     ;; Date match.
     (should
      (equal '(36)
@@ -2881,6 +2891,58 @@ SCHEDULED: <2014-03-04 tue.>"
 :TEST: 2
 :END:"
 	      (org-map-entries #'point "TEST=1"))))
+    ;; Regular negative property match.
+    (should
+     (equal '(35 68)
+	    (org-test-with-temp-text "
+* H1
+:PROPERTIES:
+:TEST: 1
+:END:
+* H2
+:PROPERTIES:
+:TEST: 2
+:END:
+* H3"
+	      (org-map-entries #'point "TEST!=1"))))
+    ;; Starred negative property match.
+    (should
+     (equal '(35)
+	    (org-test-with-temp-text "
+* H1
+:PROPERTIES:
+:TEST: 1
+:END:
+* H2
+:PROPERTIES:
+:TEST: 2
+:END:
+* H3"
+	      (org-map-entries #'point "TEST!=*1"))))
+    ;; Property matches on names including minus characters.
+    (org-test-with-temp-text
+     "
+* H1 :BAR:
+:PROPERTIES:
+:TEST-FOO: 1
+:END:
+* H2 :FOO:
+:PROPERTIES:
+:TEST-FOO: 2
+:END:
+* H3 :BAR:
+:PROPERTIES:
+:-FOO: 1
+:END:
+* H4 :FOO:
+:PROPERTIES:
+:-FOO: 2
+:END:
+* H5"
+     (should (equal '(2) (org-map-entries #'point "TEST-FOO!=*0-FOO")))
+     (should (equal '(2) (org-map-entries #'point "-FOO+TEST-FOO!=*0")))
+     (should (equal '(88) (org-map-entries #'point "+-FOO!=*0-FOO")))
+     (should (equal '(88) (org-map-entries #'point "-FOO+-FOO!=*0"))))
     ;; Multiple criteria.
     (should
      (equal '(23)
-- 
2.30.2