Re: org-agenda queries for absent properties

[Jens Schmidt <jschmidt4gnu@vodafonemail.de>]

[-- Attachment #1: Type: text/plain, Size: 3181 bytes --]

On 2023-08-02  08:45, Ihor Radchenko wrote:

 > `rx' would be great.
 > But even adding comments like in your example would be an improvement.

Since the future of this code snippet seems to be uncertain I went for
comments only.

And I thought I was pretty much done when I noticed at least one major
issues in the existing code, so I decided to go with a prerelease first
plus some notes and questions.

So there will be a follow up to the attached patch, and I leave it to
you whether you give it already a review or not.  But I'd ask you for
your opinion on the following notes, where the first few should be
uncritical:

- I used "\(?NUM: ... \)" constructs to explicitly number the subres.
   Hope this is OK w.r.t. style and backward-compatibility.

- I fixed the operator-matching subre to also include `==', `!=', `/='
   but exclude `<<' and the like which currently give void-function
   errors.

- I did not fix some "a[^b]*b"-style subres to use non-greedy variants
   since these are strictly speaking not identical.  Even though newline
   characters shouldn't play a big role here ...

- I likewise did not fix the number-matching subre allowing for numbers
   like "1.2.3" to keep things short at least there.  `string-to-number'
   silently takes care of these, even if an exponent gets lost that way.

But from here it gets more intersting:

- The code uses subre "\\\\-" in property names to (supposedly) allow
   for inclusion of minus characters in property names, which (probably)
   could be confused with term negation.

- It also unquotes these minus characters for {tag regexps}:

     (tag (save-match-data
            (replace-regexp-in-string
             "\\\\-" "-" (match-string 2 term))))

   But it never unquotes them in property names.  That missing unquoting
   could be easily amended, but:

- The other issue is: Why do we need "\\\\-" for both property names and
   {tag regexps}?  This forces us to do queries like:

     {[a\\-z]}|foo\\-bar="baz"

   where in my opinion

     {[a\-z]}|foo\-bar="baz"

   should be sufficient.

- Even more, IMO one could do away completely with the minus-quoting and
   unquoting, since the overall regexp should allow for unambiguously
   matching minus characters both

   + in {tag regexps} (because of "{[^}]+}" gobbling them) and

   + in property names (because a property name must always be followed
     by some operator)

   *without* them getting confused with term negation.

   Or do I miss something here?  A cursory test with sth like

     +foo-bar="xxx"-patchday=202302

   seems to work fine.

- However, removing the unquoting of {tag regexps} would be a breaking
   change.  Even though I doubt anybody has ever used it, the more it is
   not mentioned in the documentation.

 > I had this in mind for a wile, but I am still hoping that we can
 > eventually (when it is added to Emacs) rely upon peg.el for parsing.

Given the fact that we have to discuss issues like those above, I
heartily agree.

 > https://yhetil.org/emacs-devel/875yvtbbn3.fsf@ericabrahamsen.net/

Arthouse thread: Interesting plot, surprising sidelines, not everything
comprehensible, (unfortunately) open end.

[-- Attachment #2: 0001-org-make-tags-matcher-Add-starred-property-operators.patch --]
[-- Type: text/x-patch, Size: 11088 bytes --]

From 1765e91d2f7875b321703afe34e32754a022bef4 Mon Sep 17 00:00:00 2001
From: Jens Schmidt <jschmidt4gnu@vodafonemail.de>
Date: Thu, 3 Aug 2023 22:34:56 +0200
Subject: [PATCH] org-make-tags-matcher: Add starred property operators, more
 operator synonyms

* lisp/org.el (org-make-tags-matcher): Add starred property operators.
Recognize additional operators "==", "!=", "/=".  Clean up and
document match term parsing.
(org-op-to-function): Recognize additional inequality operator "/=".

* doc/org-manual.org (Matching tags and properties):
* etc/ORG-NEWS: (~org-tags-view~ supports more property operators):
* testing/lisp/test-org.el (test-org/map-entries): Add documentation,
announcement, and tests on starred and additional operators.
---
 doc/org-manual.org       | 20 ++++++++-
 etc/ORG-NEWS             | 10 ++++-
 lisp/org.el              | 96 +++++++++++++++++++++++++++++-----------
 testing/lisp/test-org.el | 33 ++++++++++++++
 4 files changed, 130 insertions(+), 29 deletions(-)

diff --git a/doc/org-manual.org b/doc/org-manual.org
index 16fbb268f..27051fbfc 100644
--- a/doc/org-manual.org
+++ b/doc/org-manual.org
@@ -9246,16 +9246,25 @@ When matching properties, a number of different operators can be used
 to test the value of a property.  Here is a complex example:
 
 #+begin_example
-+work-boss+PRIORITY="A"+Coffee="unlimited"+Effort<2
++work-boss+PRIORITY="A"+Coffee="unlimited"+Effort<*2
          +With={Sarah\|Denny}+SCHEDULED>="<2008-10-11>"
 #+end_example
 
+#+cindex: operator, for property search
 #+texinfo: @noindent
 The type of comparison depends on how the comparison value is written:
 
 - If the comparison value is a plain number, a numerical comparison is
   done, and the allowed operators are =<=, ===, =>=, =<==, =>==, and
-  =<>=.
+  =<>=.  As synonym for the equality operator there is also ====, as
+  synonyms for the inequality operator there are =!== and =/==.
+
+- All operators may be optionally followed by an asterisk =*=, like in
+  =<*=, =!=*=, etc.  Such /starred operators/ work like their regular,
+  unstarred counterparts except that they match only headlines where
+  the tested property is actually present.  This is most useful for
+  search terms that logically exclude results, like the inequality
+  operator.
 
 - If the comparison value is enclosed in double-quotes, a string
   comparison is done, and the same operators are allowed.
@@ -9280,6 +9289,13 @@ smaller than 2, a =With= property that is matched by the regular
 expression =Sarah\|Denny=, and that are scheduled on or after October
 11, 2008.
 
+Note that the test on the =EFFORT= property uses operator =<*=, so
+that the search result will include only entries that actually have an
+=EFFORT= property defined and with numerical value smaller than 2.
+With the regular =<= operator, the search would handle entries without
+an =EFFORT= property as having a zero effort and would include them in
+the result as well.
+
 You can configure Org mode to use property inheritance during
 a search, but beware that this can slow down searches considerably.
 See [[*Property Inheritance]], for details.
diff --git a/etc/ORG-NEWS b/etc/ORG-NEWS
index 4f16eda24..10c51e354 100644
--- a/etc/ORG-NEWS
+++ b/etc/ORG-NEWS
@@ -125,7 +125,7 @@ New functions to retrieve and set (via ~setf~) commonly used element properties:
 - =:contents-post-affiliated= :: ~org-element-post-affiliated~
 - =:contents-post-blank= :: ~org-element-post-blank~
 - =:parent= :: ~org-element-parent~
- 
+
 ***** New macro ~org-element-with-enabled-cache~
 
 The macro arranges the element cache to be active during =BODY= execution.
@@ -558,6 +558,14 @@ special repeaters ~++~ and ~.+~ are skipped.
 A capture template can target ~(here)~ which is the equivalent of
 invoking a capture template with a zero prefix.
 
+*** ~org-tags-view~ supports more property operators
+
+It supports inequality operators ~!=~ and ~/=~ in addition to the less
+common (BASIC?  Pascal?  SQL?) ~<>~.  And it supports starred versions
+of all relational operators (~<*~, ~=*~, ~!=*~, etc.) that work like
+the regular, unstarred operators but match a headline only if the
+tested property is actually present.
+
 ** New functions and changes in function arguments
 *** =TYPES= argument in ~org-element-lineage~ can now be a symbol
 
diff --git a/lisp/org.el b/lisp/org.el
index 1ac912e61..a1f4c1c53 100644
--- a/lisp/org.el
+++ b/lisp/org.el
@@ -11306,11 +11306,33 @@ See also `org-scan-tags'."
 
   (let ((match0 match)
 	(re (concat
-	     "^&?\\([-+:]\\)?\\({[^}]+}\\|LEVEL\\([<=>]\\{1,2\\}\\)"
-	     "\\([0-9]+\\)\\|\\(\\(?:[[:alnum:]_]+\\(?:\\\\-\\)*\\)+\\)"
-	     "\\([<>=]\\{1,2\\}\\)"
-	     "\\({[^}]+}\\|\"[^\"]*\"\\|-?[.0-9]+\\(?:[eE][-+]?[0-9]+\\)?\\)"
-	     "\\|" org-tag-re "\\)"))
+	     "^"
+	     ;; implicit AND operator (OR is done by global splitting)
+	     "&?"
+	     ;; exclusion and inclusion (the latter being implicit)
+	     "\\(?1:[-+:]\\)?"
+	     ;; query term
+	     "\\(?2:"
+	         ;; tag regexp match
+	         "{[^}]+}\\|"
+		 ;; LEVEL property match
+		 "LEVEL\\(?3:[<=>]=?\\|[!/]=\\|<>\\)\\(?4:[0-9]+\\)\\|"
+		 ;; regular property match
+		 "\\(?:"
+		     ;; property name
+		     "\\(?5:\\(?:[[:alnum:]_]+\\(?:\\\\-\\)*\\)+\\)"
+		     ;; operator, optionally starred
+		     "\\(?6:[<=>]=?\\|[!/]=\\|<>\\)\\(?7:\\*\\)?"
+		     ;; operand
+		     "\\(?8:"
+		         "{[^}]+}\\|"
+			 "\"[^\"]*\"\\|"
+			 "-?[.0-9]+\\(?:[eE][-+]?[0-9]+\\)?"
+		     "\\)"
+		 "\\)\\|"
+		 ;; exact tag match
+		 org-tag-re
+	     "\\)"))
 	(start 0)
 	tagsmatch todomatch tagsmatcher todomatcher)
 
@@ -11352,6 +11374,11 @@ See also `org-scan-tags'."
 	    (let* ((rest (substring term (match-end 0)))
 		   (minus (and (match-end 1)
 			       (equal (match-string 1 term) "-")))
+		   ;; Bind the whole term to `tag' and use that
+		   ;; variable for a tag regexp match in (1) or as an
+		   ;; exact tag match in (2).  Unquote quoted minus
+		   ;; characters, which would be actually required
+		   ;; only for the former case.
 		   (tag (save-match-data
 			  (replace-regexp-in-string
 			   "\\\\-" "-" (match-string 2 term))))
@@ -11360,7 +11387,7 @@ See also `org-scan-tags'."
 		   (propp (match-end 5))
 		   (mm
 		    (cond
-		     (regexp
+		     (regexp			; (1)
                       `(with-syntax-table org-mode-tags-syntax-table
                          (org-match-any-p ,(substring tag 1 -1) tags-list)))
 		     (levelp
@@ -11368,28 +11395,45 @@ See also `org-scan-tags'."
 			level
 			,(string-to-number (match-string 4 term))))
 		     (propp
-		      (let* ((gv (pcase (upcase (match-string 5 term))
+		      (let* (;; Convert property name to an Elisp
+			     ;; accessor for that property.
+			     (gv (pcase (upcase (match-string 5 term))
 				   ("CATEGORY"
 				    '(org-get-category (point)))
 				   ("TODO" 'todo)
 				   (p `(org-cached-entry-get nil ,p))))
-			     (pv (match-string 7 term))
+			     ;; Determine operand (aka. property
+			     ;; value)
+			     (pv (match-string 8 term))
+			     ;; Determine type of operand.  Note that
+			     ;; these are not exclusive: Any TIMEP is
+			     ;; also STRP.
 			     (regexp (eq (string-to-char pv) ?{))
 			     (strp (eq (string-to-char pv) ?\"))
 			     (timep (string-match-p "^\"[[<]\\(?:[0-9]+\\|now\\|today\\|tomorrow\\|[+-][0-9]+[dmwy]\\).*[]>]\"$" pv))
+			     ;; Massage operand.  TIMEP must come
+			     ;; before STRP.
+			     (pv (cond (regexp (substring pv 1 -1))
+				       (timep  (org-matcher-time
+						(substring pv 1 -1)))
+				       (strp   (substring pv 1 -1))
+				       (t      pv)))
+			     ;; Convert operator to Elisp.
 			     (po (org-op-to-function (match-string 6 term)
-						     (if timep 'time strp))))
-			(setq pv (if (or regexp strp) (substring pv 1 -1) pv))
-			(when timep (setq pv (org-matcher-time pv)))
-			(cond ((and regexp (eq po '/=))
-			       `(not (string-match ,pv (or ,gv ""))))
-			      (regexp `(string-match ,pv (or ,gv "")))
-			      (strp `(,po (or ,gv "") ,pv))
-			      (t
-			       `(,po
-				 (string-to-number (or ,gv ""))
-				 ,(string-to-number pv))))))
-		     (t `(member ,tag tags-list)))))
+						     (if timep 'time strp)))
+			     ;; Convert whole term to Elisp.
+			     (pt (cond ((and regexp (eq po '/=))
+					`(not (string-match ,pv (or ,gv ""))))
+				       (regexp `(string-match ,pv (or ,gv "")))
+				       (strp `(,po (or ,gv "") ,pv))
+				       (t
+					`(,po
+					  (string-to-number (or ,gv ""))
+					  ,(string-to-number pv)))))
+			     ;; Respect the star after the operand.
+			     (pt (if (match-end 7) `(and ,gv ,pt) pt)))
+			pt))
+		     (t `(member ,tag tags-list))))) ; (2)
 	      (push (if minus `(not ,mm) mm) tagsmatcher)
 	      (setq term rest)))
 	  (push `(and ,@tagsmatcher) orlist)
@@ -11520,12 +11564,12 @@ the list of tags in this group."
   "Turn an operator into the appropriate function."
   (setq op
 	(cond
-	 ((equal  op   "<"       ) '(<     org-string<  org-time<))
-	 ((equal  op   ">"       ) '(>     org-string>  org-time>))
-	 ((member op '("<=" "=<")) '(<=    org-string<= org-time<=))
-	 ((member op '(">=" "=>")) '(>=    org-string>= org-time>=))
-	 ((member op '("="  "==")) '(=     string=      org-time=))
-	 ((member op '("<>" "!=")) '(/=    org-string<> org-time<>))))
+	 ((equal  op   "<"            ) '(<     org-string<  org-time<))
+	 ((equal  op   ">"            ) '(>     org-string>  org-time>))
+	 ((member op '("<=" "=<"     )) '(<=    org-string<= org-time<=))
+	 ((member op '(">=" "=>"     )) '(>=    org-string>= org-time>=))
+	 ((member op '("="  "=="     )) '(=     string=      org-time=))
+	 ((member op '("<>" "!=" "/=")) '(/=    org-string<> org-time<>))))
   (nth (if (eq stringp 'time) 2 (if stringp 1 0)) op))
 
 (defvar org-add-colon-after-tag-completion nil)  ;; dynamically scoped param
diff --git a/testing/lisp/test-org.el b/testing/lisp/test-org.el
index 890ea6a8c..ef52b95c7 100644
--- a/testing/lisp/test-org.el
+++ b/testing/lisp/test-org.el
@@ -2851,6 +2851,11 @@ test <point>
      (equal '(1)
 	    (org-test-with-temp-text "* [#A] H1\n* [#B] H2"
 	      (org-map-entries #'point "PRIORITY=\"A\""))))
+    ;; Negative priority match.
+    (should
+     (equal '(11)
+	    (org-test-with-temp-text "* [#A] H1\n* [#B] H2"
+	      (org-map-entries #'point "PRIORITY/=\"A\""))))
     ;; Date match.
     (should
      (equal '(36)
@@ -2881,6 +2886,34 @@ SCHEDULED: <2014-03-04 tue.>"
 :TEST: 2
 :END:"
 	      (org-map-entries #'point "TEST=1"))))
+    ;; Negative regular property match.
+    (should
+     (equal '(35 68)
+	    (org-test-with-temp-text "
+* H1
+:PROPERTIES:
+:TEST: 1
+:END:
+* H2
+:PROPERTIES:
+:TEST: 2
+:END:
+* H3"
+	      (org-map-entries #'point "TEST!=1"))))
+    ;; Starred negative regular property match.
+    (should
+     (equal '(35)
+	    (org-test-with-temp-text "
+* H1
+:PROPERTIES:
+:TEST: 1
+:END:
+* H2
+:PROPERTIES:
+:TEST: 2
+:END:
+* H3"
+	      (org-map-entries #'point "TEST!=*1"))))
     ;; Multiple criteria.
     (should
      (equal '(23)
-- 
2.30.2