* org-scan-tags
@ 2010-09-15 3:19 Ilya Shlyakhter
2010-09-15 14:36 ` org-scan-tags Ilya Shlyakhter
` (3 more replies)
0 siblings, 4 replies; 9+ messages in thread
From: Ilya Shlyakhter @ 2010-09-15 3:19 UTC (permalink / raw)
To: emacs-orgmode
In org-scan-tags, if todo-only is t, would it be possible to speed
things up by changingthe regexp go to just the lines with a TODO
keyword?
I.e. in
(let* ((re (concat "^" outline-regexp " *\\(\\<\\("
(mapconcat 'regexp-quote org-todo-keywords-1 "\\|")
(org-re
"\\>\\)\\)? *\\(.*?\\)\\(:[[:alnum:]_@:]+:\\)?[ \t]*$")))
remove the first "?" if todo-only is t. Also, regexp-opt might make
a more efficient regexp than mapconcat with regexp-quote.
Reason for request: I'm writing an extension of org for setting &
checking goals, and want to quickly find entries with headlines of the
form
GOAL ....
of which there may be relatively few in a large file. So, stepping
through all entries and then checking them for the GOAL keyword is
very inefficient.
It would be much faster if the regexp included the GOAL as a keyword.
It would be good if the parameter todo-only could be a list of
strings, and org-scan-tags would return only the headlines where the
todo keyword is from this list.
It could use regexp-opt to make an efficient regexp for this.
There also seem to be other opportunities for speeding up
org-scan-tags in this way: e.g. if the match string includes +mytag,
the regexp for the headline could include this as well.
Similarly for properties. Maybe, org-make-tags-matcher could return a
list of tags and properties that must appear in any matching entry.
It would also help if the tags matcher expression could refer to text
properties stored on the headline -- perhaps, with conditions such as
:myprop=X (i.e. same as for org properties, but property name must be
a keyword). It already does this for the 'org-category text
property.
Then one can e.g. mark entries representing unmet goals with text
properties, and then use a regular org-tags-view to browse them in a
sparsetree
or an agenda.
Thanks,
ilya
^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: org-scan-tags
2010-09-15 3:19 org-scan-tags Ilya Shlyakhter
@ 2010-09-15 14:36 ` Ilya Shlyakhter
2010-09-15 16:13 ` org-scan-tags Ilya Shlyakhter
` (2 subsequent siblings)
3 siblings, 0 replies; 9+ messages in thread
From: Ilya Shlyakhter @ 2010-09-15 14:36 UTC (permalink / raw)
To: emacs-orgmode
Another question: if org-map-continue-from is used to skip parts of
the file, could that affect
the correctness of org-scanner-tags? I.e. is any code that sets
org-map-continue-from
also responsible for updating org-scanner-tags?
thanks,
ilya
On Tue, Sep 14, 2010 at 11:19 PM, Ilya Shlyakhter <ilya_shl@alum.mit.edu> wrote:
> In org-scan-tags, if todo-only is t, would it be possible to speed
> things up by changingthe regexp go to just the lines with a TODO
> keyword?
> I.e. in
>
> (let* ((re (concat "^" outline-regexp " *\\(\\<\\("
> (mapconcat 'regexp-quote org-todo-keywords-1 "\\|")
> (org-re
> "\\>\\)\\)? *\\(.*?\\)\\(:[[:alnum:]_@:]+:\\)?[ \t]*$")))
>
> remove the first "?" if todo-only is t. Also, regexp-opt might make
> a more efficient regexp than mapconcat with regexp-quote.
>
> Reason for request: I'm writing an extension of org for setting &
> checking goals, and want to quickly find entries with headlines of the
> form
> GOAL ....
> of which there may be relatively few in a large file. So, stepping
> through all entries and then checking them for the GOAL keyword is
> very inefficient.
> It would be much faster if the regexp included the GOAL as a keyword.
>
> It would be good if the parameter todo-only could be a list of
> strings, and org-scan-tags would return only the headlines where the
> todo keyword is from this list.
> It could use regexp-opt to make an efficient regexp for this.
>
> There also seem to be other opportunities for speeding up
> org-scan-tags in this way: e.g. if the match string includes +mytag,
> the regexp for the headline could include this as well.
> Similarly for properties. Maybe, org-make-tags-matcher could return a
> list of tags and properties that must appear in any matching entry.
>
> It would also help if the tags matcher expression could refer to text
> properties stored on the headline -- perhaps, with conditions such as
> :myprop=X (i.e. same as for org properties, but property name must be
> a keyword). It already does this for the 'org-category text
> property.
> Then one can e.g. mark entries representing unmet goals with text
> properties, and then use a regular org-tags-view to browse them in a
> sparsetree
> or an agenda.
>
> Thanks,
>
> ilya
>
^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: org-scan-tags
2010-09-15 3:19 org-scan-tags Ilya Shlyakhter
2010-09-15 14:36 ` org-scan-tags Ilya Shlyakhter
@ 2010-09-15 16:13 ` Ilya Shlyakhter
2011-02-02 23:17 ` org-scan-tags Bastien
2011-02-03 5:32 ` org-scan-tags Carsten Dominik
3 siblings, 0 replies; 9+ messages in thread
From: Ilya Shlyakhter @ 2010-09-15 16:13 UTC (permalink / raw)
To: emacs-orgmode
On Tue, Sep 14, 2010 at 11:19 PM, Ilya Shlyakhter <ilya_shl@alum.mit.edu> wrote:
> There also seem to be other opportunities for speeding up
> org-scan-tags in this way: e.g. if the match string includes +mytag,
> the regexp for the headline could include this as well.
> Similarly for properties. Maybe, org-make-tags-matcher could return a
> list of tags and properties that must appear in any matching entry.
Correction: org-make-tags-matcher would also need to return the list of
_all_ tags/properties mentioned in the matcher, whose values should be
inherited.
The headline regexp would then need to match the mention of these
tags/properties
in entries, as well.
It would also be good if org-scan-tags could be told to skip selected
subtrees entirely
(selected either by another matcher or a predicate passed in).
^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: org-scan-tags
2010-09-15 3:19 org-scan-tags Ilya Shlyakhter
2010-09-15 14:36 ` org-scan-tags Ilya Shlyakhter
2010-09-15 16:13 ` org-scan-tags Ilya Shlyakhter
@ 2011-02-02 23:17 ` Bastien
2011-02-03 5:32 ` org-scan-tags Carsten Dominik
3 siblings, 0 replies; 9+ messages in thread
From: Bastien @ 2011-02-02 23:17 UTC (permalink / raw)
To: Ilya Shlyakhter; +Cc: emacs-orgmode
Hi Ilya,
Ilya Shlyakhter <ilya_shl@alum.mit.edu> writes:
> In org-scan-tags, if todo-only is t, would it be possible to speed
> things up by changingthe regexp go to just the lines with a TODO
> keyword?
> I.e. in
>
> (let* ((re (concat "^" outline-regexp " *\\(\\<\\("
> (mapconcat 'regexp-quote org-todo-keywords-1 "\\|")
> (org-re
> "\\>\\)\\)? *\\(.*?\\)\\(:[[:alnum:]_@:]+:\\)?[ \t]*$")))
>
> remove the first "?" if todo-only is t. Also, regexp-opt might make
> a more efficient regexp than mapconcat with regexp-quote.
I've optimized org-scan-tags a bit following your ideas (gaining ~12%
according to elp) -- thanks for these directions.
> It would be good if the parameter todo-only could be a list of
> strings, and org-scan-tags would return only the headlines where the
> todo keyword is from this list.
This would be confusing. Particularily, org-tags-view uses
org-scan-tags using both the todo-only argument and a matcher: so
if you make the todo-only argument aware of TODO keywords, there
might be some interference between todo-only and the matcher.
I'd rather not go that route.
Thanks,
--
Bastien
^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: org-scan-tags
2010-09-15 3:19 org-scan-tags Ilya Shlyakhter
` (2 preceding siblings ...)
2011-02-02 23:17 ` org-scan-tags Bastien
@ 2011-02-03 5:32 ` Carsten Dominik
2011-02-03 16:13 ` org-scan-tags Carsten Dominik
3 siblings, 1 reply; 9+ messages in thread
From: Carsten Dominik @ 2011-02-03 5:32 UTC (permalink / raw)
To: Ilya Shlyakhter; +Cc: emacs-orgmode
On Sep 15, 2010, at 5:19 AM, Ilya Shlyakhter wrote:
> In org-scan-tags, if todo-only is t, would it be possible to speed
> things up by changingthe regexp go to just the lines with a TODO
> keyword?
I believe this may cause a problem. The scanner needs to see at least
every parent node to be able to collect all inherited tags.
So I think that a tree like
* heading
** one :tag1:
*** TODO two :tag2:
would incorrectly miss out on :tag1:
- Carsten
> I.e. in
>
> (let* ((re (concat "^" outline-regexp " *\\(\\<\\("
> (mapconcat 'regexp-quote org-todo-keywords-1 "\
> \|")
> (org-re
> "\\>\\)\\)? *\\(.*?\\)\\(:[[:alnum:]_@:]+:\\)?
> [ \t]*$")))
>
> remove the first "?" if todo-only is t. Also, regexp-opt might make
> a more efficient regexp than mapconcat with regexp-quote.
>
> Reason for request: I'm writing an extension of org for setting &
> checking goals, and want to quickly find entries with headlines of the
> form
> GOAL ....
> of which there may be relatively few in a large file. So, stepping
> through all entries and then checking them for the GOAL keyword is
> very inefficient.
> It would be much faster if the regexp included the GOAL as a keyword.
>
> It would be good if the parameter todo-only could be a list of
> strings, and org-scan-tags would return only the headlines where the
> todo keyword is from this list.
> It could use regexp-opt to make an efficient regexp for this.
>
> There also seem to be other opportunities for speeding up
> org-scan-tags in this way: e.g. if the match string includes +mytag,
> the regexp for the headline could include this as well.
> Similarly for properties. Maybe, org-make-tags-matcher could return a
> list of tags and properties that must appear in any matching entry.
>
> It would also help if the tags matcher expression could refer to text
> properties stored on the headline -- perhaps, with conditions such as
> :myprop=X (i.e. same as for org properties, but property name must be
> a keyword). It already does this for the 'org-category text
> property.
> Then one can e.g. mark entries representing unmet goals with text
> properties, and then use a regular org-tags-view to browse them in a
> sparsetree
> or an agenda.
>
> Thanks,
>
> ilya
>
> _______________________________________________
> Emacs-orgmode mailing list
> Please use `Reply All' to send replies to the list.
> Emacs-orgmode@gnu.org
> http://lists.gnu.org/mailman/listinfo/emacs-orgmode
^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: org-scan-tags
2011-02-03 5:32 ` org-scan-tags Carsten Dominik
@ 2011-02-03 16:13 ` Carsten Dominik
2011-02-03 16:37 ` org-scan-tags Bastien
0 siblings, 1 reply; 9+ messages in thread
From: Carsten Dominik @ 2011-02-03 16:13 UTC (permalink / raw)
To: Carsten Dominik; +Cc: Ilya Shlyakhter, emacs-orgmode
On Feb 3, 2011, at 6:32 AM, Carsten Dominik wrote:
>
> On Sep 15, 2010, at 5:19 AM, Ilya Shlyakhter wrote:
>
>> In org-scan-tags, if todo-only is t, would it be possible to speed
>> things up by changingthe regexp go to just the lines with a TODO
>> keyword?
>
> I believe this may cause a problem. The scanner needs to see at least
> every parent node to be able to collect all inherited tags.
> So I think that a tree like
>
> * heading
> ** one :tag1:
> *** TODO two :tag2:
>
> would incorrectly miss out on :tag1:
OK, here is an example where it really does fail:
* heading
** one :tag1:
*** two
*** two :tag2:
*** TODO two :tag2:
*** two :tag2:
Fold up the tree, then do
C-c / m +tag1/! RET
This should find the "TODO two", but it does not, because the
new regexp moves right past the "one" line and so tag1 is
overlooked.
- Carsten
>
> - Carsten
>
>> I.e. in
>>
>> (let* ((re (concat "^" outline-regexp " *\\(\\<\\("
>> (mapconcat 'regexp-quote org-todo-keywords-1 "\
>> \|")
>> (org-re
>> "\\>\\)\\)? *\\(.*?\\)\\(:[[:alnum:]_@:]+:\\)?
>> [ \t]*$")))
>>
>> remove the first "?" if todo-only is t. Also, regexp-opt might make
>> a more efficient regexp than mapconcat with regexp-quote.
>>
>> Reason for request: I'm writing an extension of org for setting &
>> checking goals, and want to quickly find entries with headlines of
>> the
>> form
>> GOAL ....
>> of which there may be relatively few in a large file. So, stepping
>> through all entries and then checking them for the GOAL keyword is
>> very inefficient.
>> It would be much faster if the regexp included the GOAL as a keyword.
>>
>> It would be good if the parameter todo-only could be a list of
>> strings, and org-scan-tags would return only the headlines where the
>> todo keyword is from this list.
>> It could use regexp-opt to make an efficient regexp for this.
>>
>> There also seem to be other opportunities for speeding up
>> org-scan-tags in this way: e.g. if the match string includes +mytag,
>> the regexp for the headline could include this as well.
>> Similarly for properties. Maybe, org-make-tags-matcher could
>> return a
>> list of tags and properties that must appear in any matching entry.
>>
>> It would also help if the tags matcher expression could refer to text
>> properties stored on the headline -- perhaps, with conditions such as
>> :myprop=X (i.e. same as for org properties, but property name must
>> be
>> a keyword). It already does this for the 'org-category text
>> property.
>> Then one can e.g. mark entries representing unmet goals with text
>> properties, and then use a regular org-tags-view to browse them in a
>> sparsetree
>> or an agenda.
>>
>> Thanks,
>>
>> ilya
>>
>> _______________________________________________
>> Emacs-orgmode mailing list
>> Please use `Reply All' to send replies to the list.
>> Emacs-orgmode@gnu.org
>> http://lists.gnu.org/mailman/listinfo/emacs-orgmode
>
^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: org-scan-tags
2011-02-03 16:13 ` org-scan-tags Carsten Dominik
@ 2011-02-03 16:37 ` Bastien
2011-02-03 16:47 ` org-scan-tags Carsten Dominik
2011-02-05 0:36 ` org-scan-tags Ilya Shlyakhter
0 siblings, 2 replies; 9+ messages in thread
From: Bastien @ 2011-02-03 16:37 UTC (permalink / raw)
To: Carsten Dominik; +Cc: Ilya Shlyakhter, emacs-orgmode
Carsten Dominik <carsten.dominik@gmail.com> writes:
> OK, here is an example where it really does fail:
>
>
> * heading
> ** one :tag1:
> *** two
> *** two :tag2:
> *** TODO two :tag2:
> *** two :tag2:
>
>
> Fold up the tree, then do
>
> C-c / m +tag1/! RET
>
> This should find the "TODO two", but it does not, because the
> new regexp moves right past the "one" line and so tag1 is
> overlooked.
Right, thanks for the detailed example. I reverted the commit,
it should be fine again.
--
Bastien
^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: org-scan-tags
2011-02-03 16:37 ` org-scan-tags Bastien
@ 2011-02-03 16:47 ` Carsten Dominik
2011-02-05 0:36 ` org-scan-tags Ilya Shlyakhter
1 sibling, 0 replies; 9+ messages in thread
From: Carsten Dominik @ 2011-02-03 16:47 UTC (permalink / raw)
To: Bastien; +Cc: emacs-orgmode Mode
Thanks for the fast reaction, Bastioen!
- Carsten
On Feb 3, 2011, at 5:37 PM, Bastien wrote:
> Carsten Dominik <carsten.dominik@gmail.com> writes:
>
>> OK, here is an example where it really does fail:
>>
>>
>> * heading
>> ** one :tag1:
>> *** two
>> *** two :tag2:
>> *** TODO two :tag2:
>> *** two :tag2:
>>
>>
>> Fold up the tree, then do
>>
>> C-c / m +tag1/! RET
>>
>> This should find the "TODO two", but it does not, because the
>> new regexp moves right past the "one" line and so tag1 is
>> overlooked.
>
> Right, thanks for the detailed example. I reverted the commit,
> it should be fine again.
>
> --
> Bastien
^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: org-scan-tags
2011-02-03 16:37 ` org-scan-tags Bastien
2011-02-03 16:47 ` org-scan-tags Carsten Dominik
@ 2011-02-05 0:36 ` Ilya Shlyakhter
1 sibling, 0 replies; 9+ messages in thread
From: Ilya Shlyakhter @ 2011-02-05 0:36 UTC (permalink / raw)
To: Bastien; +Cc: emacs-orgmode, Carsten Dominik
Thanks for catching this, Carsten!
This could perhaps be fixed by doing a full lookup of the tags up the
hierarchy, rather than relying on the cached tags.
This is more expensive, but if fewer entries actually have to be
looked at (because the search only stops at TODO entries),
it might be faster overall.
One general way to speed up searches would be to move as much work as
possible into Emacs' built-in regexp matcher.
When parsing a search expression, right now it is parsed into an elisp
form that is evaluated at each entry and says
whether the entry matches. Each clause of a search expression could
instead be parsed into an elisp form _and_ a regexp,
such that matching the regexp would be a necessary (but not
sufficient) condition for the entry to match.
E.g. if looking for entries with property PROP equal to 1, you could
construct a regexp that would match only that.
Some things aren't expressible in regexp language so they'd still have
to be checked in lisp. And tag lookups could not use the
cache. But if most of the filtering is done by Emacs' regexp
matcher, and only a bit of lisp filtering on top of that,
overall searches might be faster.
On Thu, Feb 3, 2011 at 11:37 AM, Bastien <bastien.guerry@wikimedia.fr> wrote:
> Carsten Dominik <carsten.dominik@gmail.com> writes:
>
>> OK, here is an example where it really does fail:
>>
>>
>> * heading
>> ** one :tag1:
>> *** two
>> *** two :tag2:
>> *** TODO two :tag2:
>> *** two :tag2:
>>
>>
>> Fold up the tree, then do
>>
>> C-c / m +tag1/! RET
>>
>> This should find the "TODO two", but it does not, because the
>> new regexp moves right past the "one" line and so tag1 is
>> overlooked.
>
> Right, thanks for the detailed example. I reverted the commit,
> it should be fine again.
>
> --
> Bastien
>
^ permalink raw reply [flat|nested] 9+ messages in thread
end of thread, other threads:[~2011-02-05 0:36 UTC | newest]
Thread overview: 9+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2010-09-15 3:19 org-scan-tags Ilya Shlyakhter
2010-09-15 14:36 ` org-scan-tags Ilya Shlyakhter
2010-09-15 16:13 ` org-scan-tags Ilya Shlyakhter
2011-02-02 23:17 ` org-scan-tags Bastien
2011-02-03 5:32 ` org-scan-tags Carsten Dominik
2011-02-03 16:13 ` org-scan-tags Carsten Dominik
2011-02-03 16:37 ` org-scan-tags Bastien
2011-02-03 16:47 ` org-scan-tags Carsten Dominik
2011-02-05 0:36 ` org-scan-tags Ilya Shlyakhter
Code repositories for project(s) associated with this public inbox
https://git.savannah.gnu.org/cgit/emacs/org-mode.git
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).