[-- Attachment #1: Type: text/plain, Size: 4191 bytes --] * [RFC] DOCT: Declarative Org Capture Templates I've been working on an alternative syntax for org-capture-templates. The result is a package called "DOCT" (declarative org capture templates): https://github.com/progfolio/doct A brief list of what I believe to be improvements over the current syntax/implementation: - DOCT validates templates before runtime execution. For exmaple, you have a template with an entry type of `'entry' and you forget the leading star in the template string. Days later you go to use that template. It's borked. You have a number of options: - forget about whatever you wanted to capture and press on with your original task - manually take a note about what you originally wanted to capture and another note to fix the broken template - drop what you're doing and fix everything None of these are ideal and all of them result in distraction. DOCT performs a number of checks ahead of time when possible to prevent these types of errors. - DOCT makes the parent/child relationship between templates explicit. `org-capture-templates` is a flat list. The relationship between templates is hardcoded in each template's "keys" value. If you want to change the key for a top-level menu, you must then change it in each descendant's keys. DOCT uses nested plists and implements property inheritance. - DOCT manages per-template hooks/contexts. Currently if you want to have a hook run for a particular template, you have to filter against `org-capture-plist' to check for the appropriate :key value. As stated above, this is fragile and you have to update that key value in numerous places if it ever changes. The same goes for `org-capture-templates-contexts`. DOCT allows specifying per-template contexts and hooks with the rest of the template's configuration. - DOCT makes adding custom metadata to templates easy. A common pattern for attaching data to a template is to add to `org-capture-plist'. This pollutes `org-capture-plist' more than necessary. DOCT adds custom data to `org-capture-plist' under a single :doct keyword and allows users to access that data in the template string with familiar %-escape syntax. This example is one I use daily for taking notes in programming projects: #+begin_src emacs-lisp (doct `("Note" :keys "n" :file ,(defun my/project-note-file () (let ((file (expand-file-name "notes.org" (when (functionp 'projectile-project-root) (projectile-project-root))))) (with-current-buffer (find-file-noselect file) (org-mode) ;;set to utf-8 because we may be visiting raw file (setq buffer-file-coding-system 'utf-8-unix) (when-let ((headline (doct-get :headline))) (unless (org-find-exact-headline-in-buffer headline) (goto-char (point-max)) (insert "* " headline) (org-set-tags (downcase headline)))) (unless (file-exists-p file) (write-file file)) file))) :template (lambda () (concat "* %{todo-state} " (when (y-or-n-p "link?") "%A\n") "%?")) :todo-state "TODO" :children (("bug" :keys "b" :headline "Bugs") ("documentation" :keys "d" :headline "Documentation") ("enhancement" :keys "e" :headline "Enhancements" :todo-state "IDEA") ("feature" :keys "f" :headline "Features" :todo-state "IDEA") ("optimization" :keys "o" :headline "Optimizations") ("miscellaneous" :keys "m" :headline "Miscellaneous") ("security" :keys "s" :headline "Security")))) #+end_src Each template inherits its parent's file finding function,template string, and a default :todo-state. The template string access the child's :todo-state keyword with the "%{KEYWORD}" syntax in the template string. I would be happy to work on getting these features into Org if there is any interest. Any feedback is welcome. Thanks, nv. [-- Attachment #2: Type: text/html, Size: 7074 bytes --]
Hello, No Wayman <iarchivedmywholelife@gmail.com> writes: > * [RFC] DOCT: Declarative Org Capture Templates Thank you for your work. I have some comments. Disclaimer: I am only using very basic capture templates. So, I cannot comment realistically on the new syntax you suggest. In particular, the example you give is way too complex for me to understand the benefits of your syntax. I suggest to start out with showing simple use-cases. Anyway, I hope more advanced capture template users can chime in and comment your design. My beginner questions are the following : - Is it compatible with the current syntax? If it isn't, is there a way to ease transition to the new syntax? - Is it simple to use on simple use-cases? - Is it more capable than the current syntax, i.e., does it handle situations that are not possible, even in a convoluted way, currently? > - DOCT validates templates before runtime execution. > > For exmaple, you have a template with an entry type of `'entry' > and you forget the leading star in the template string. > Days later you go to use that template. It's borked. This is different from introducing a new syntax for capture templates. Actually, `org-insert-place-item' and `org-capture-place-table-line' both try to fix misshaped templates already. OTOH `org-capture-place-entry' merely calls `org-capture-verify-tree' on the template, i.e., it barfs if the template is ill-defined. It is a discrepancy we could fix independently on your new syntax. I invite you to propose a patch for `org-capture-place-entry' so it does a better job at fixing the initial template, if needed. I'll gladly apply it. Regards, -- Nicolas Goaziou
Nicolas Goaziou <mail@nicolasgoaziou.fr> writes: > Hello, > > No Wayman <iarchivedmywholelife@gmail.com> writes: > >> * [RFC] DOCT: Declarative Org Capture Templates > > Thank you for your work. Thank you for taking the time to respond. > Disclaimer: I am only using very basic capture templates. So, I > cannot > comment realistically on the new syntax you suggest. In > particular, the > example you give is way too complex for me to understand the > benefits of > your syntax. I suggest to start out with showing simple > use-cases. Apologies, it's hard to strike a balance between showing something practical and over-writing. > - Is it compatible with the current syntax? If it isn't, is > there a way > to ease transition to the new syntax? My package translates DOCT's syntax into the current syntax. I have written a separate package that does basic translation from the current syntax to DOCT's as well. It is not optimal yet, but does work for simple cases. There are a few features of DOCT (property inheritance, management of hooks/contexts) that make it more difficult than just a syntax swap. I could come up with something more fully featured, but in my experience thus far, it does not take long to translate from one syntax to the other manually. > - Is it simple to use on simple use-cases? I would say so. There is a single function that does the translation. For example: (doct '("Example" :keys "e" :file "")) Returns: (("e" "Example" entry (file "") nil :doct (:doct-name "Example" :keys "e" :file ""))) Part of my frustration of writing templates was always having to look up the structure of the template list. Keys, description, type, target (with its variations) and template-string (with its variations) is a lot to remember. Whereas, with: (doct '("Example" :keys "e" :file "")) I need only remember that the description comes first. The keywords are more self-describing as well. There's an inherent complexity to the flexibility that org-capture offers, but this makes templates easier to write/read. > - Is it more capable than the current syntax, i.e., does it > handle > situations that are not possible, even in a convoluted way, > currently? >> - DOCT validates templates before runtime execution. >> >> For exmaple, you have a template with an entry type of `'entry' >> and you forget the leading star in the template string. >> Days later you go to use that template. It's borked. > > This is different from introducing a new syntax for capture > templates. > > Actually, `org-insert-place-item' and > `org-capture-place-table-line' > both try to fix misshaped templates already. OTOH > `org-capture-place-entry' merely calls `org-capture-verify-tree' > on the > template, i.e., it barfs if the template is ill-defined. It is > a discrepancy we could fix independently on your new syntax. > > I invite you to propose a patch for `org-capture-place-entry' so > it does > a better job at fixing the initial template, if needed. I'll > gladly > apply it. `org-capture-place-entry' is run after `org-capture' is invoked, so while I agree a patch could improve the error, the user still hits that error when they are using their capture template (defeating the point of org-capture letting you take a quick note without losing your current context). DOCT checks while converting declarations to templates, so the error is thrown before org-capture is used (almost like linting for templates). Aside from that, most of what DOCT does is sugar for common use patterns I've observed in others' org-capture-templates. For example, adding per-declaration hooks: Without DOCT: ;;rest of template configured elsewhere... ;;make sure you update this if you ever change the key for this template! (defun my-org-template-hook () (when (string= (org-capture-get :key t) "t") (message "Template \"t\" selected."))) (add-hook 'org-capture-mode-hook 'my-org-template-hook) With DOCT: (doct `("Example" :keys "t" :file "" :hook (lambda () (message "Template %s selected." (doct-get :keys))))) DOCT ensures that function is only run for that template without having the user manually filter against `org-capture-plist's :key. It also allows the user to do this inline with the rest of the declaration vs spreading it out.
I am pinging this thread again because I believe that doct syntax is
much easier to write compared to current defaults. It should be added to
Org core.
Also, if anyone agrees with my arguments below, do not stay silent and
drop a "+1" email below. Otherwise, this whole thing will be stalled.
There is no point discussing technical aspects, if there is no interest.
Citing again an example illustrating the difference between a slightly
non-trivial capture template using current vs. doct syntax:
No Wayman <iarchivedmywholelife@gmail.com> writes:
> Without DOCT:
>
> ;;rest of template configured elsewhere...
> ;;make sure you update this if you ever change the key for this
> template!
> (defun my-org-template-hook ()
> (when (string= (org-capture-get :key t) "t")
> (message "Template \"t\" selected.")))
>
> (add-hook 'org-capture-mode-hook 'my-org-template-hook)
>
> With DOCT:
>
> (doct `("Example" :keys "t" :file ""
> :hook (lambda () (message "Template %s selected."
> (doct-get :keys)))))
>
> DOCT ensures that function is only run for that template without
> having the user manually filter against `org-capture-plist's :key.
> It also allows the user to do this inline with the rest of the
> declaration vs spreading it out.
I believe that the right way to introduce this syntax is creating an
easy-to-use macro like (org-declare-capture-template ...), which takes
care about fiddling with org-capture-templates variable and its format.
The current format of a number of Org customisations is cumbersome. For
example, org-capture-templates and org-agenda-custom-commands often
become long list of lists with each element that must have a very
specific position. Every single time I need to write a new template, I
have to carefully consult the docstring; and every single time I an
reviewing my templates, I have to consult the docstring again simply to
understand the meaning of the first, second, ..., nth element of the
template. Having CL-style :keyword value plists is much more readable.
More generally, it would be useful to provide a doct-like functionality
for org-agenda-custom-commands and similar variables (like
org-agenda-custom-commands, or say, also org-latex-classes).
Best,
Ihor
Hello,
Ihor Radchenko <yantar92@gmail.com> writes:
> I am pinging this thread again because I believe that doct syntax is
> much easier to write compared to current defaults. It should be added to
> Org core.
>
> Also, if anyone agrees with my arguments below, do not stay silent and
> drop a "+1" email below. Otherwise, this whole thing will be stalled.
> There is no point discussing technical aspects, if there is no
> interest.
I think there should be a direct mapping between Customize interface and
values. Adding this macro as a band-aid to simply configuration is not,
IMO, a solution.
If capture templates values are too complicated, what about simplifying
them, and possibly use this macro as a temporary solution to help
transition?
Regards,
--
Nicolas Goaziou
[-- Attachment #1: Type: text/plain, Size: 2939 bytes --] +1 as interested I saw this project a while ago https://github.com/progfolio/doct <https://github.com/progfolio/doct> I was concerned about long term support and if I would be left with templates I would have to rewrite again, plus I was too busy at the time to adopt it and it stays in my list of TODOs that are not scheduled. That being said, I use org capture daily, but to the points you raise, I avoid changing or adding to to my capture templates. > On Mar 20, 2022, at 3:19 AM, Ihor Radchenko <yantar92@gmail.com> wrote: > > I am pinging this thread again because I believe that doct syntax is > much easier to write compared to current defaults. It should be added to > Org core. > > Also, if anyone agrees with my arguments below, do not stay silent and > drop a "+1" email below. Otherwise, this whole thing will be stalled. > There is no point discussing technical aspects, if there is no interest. > > Citing again an example illustrating the difference between a slightly > non-trivial capture template using current vs. doct syntax: > > No Wayman <iarchivedmywholelife@gmail.com> writes: > >> Without DOCT: >> >> ;;rest of template configured elsewhere... >> ;;make sure you update this if you ever change the key for this >> template! >> (defun my-org-template-hook () >> (when (string= (org-capture-get :key t) "t") >> (message "Template \"t\" selected."))) >> >> (add-hook 'org-capture-mode-hook 'my-org-template-hook) >> >> With DOCT: >> >> (doct `("Example" :keys "t" :file "" >> :hook (lambda () (message "Template %s selected." >> (doct-get :keys))))) >> >> DOCT ensures that function is only run for that template without >> having the user manually filter against `org-capture-plist's :key. >> It also allows the user to do this inline with the rest of the >> declaration vs spreading it out. > > I believe that the right way to introduce this syntax is creating an > easy-to-use macro like (org-declare-capture-template ...), which takes > care about fiddling with org-capture-templates variable and its format. > > The current format of a number of Org customisations is cumbersome. For > example, org-capture-templates and org-agenda-custom-commands often > become long list of lists with each element that must have a very > specific position. Every single time I need to write a new template, I > have to carefully consult the docstring; and every single time I an > reviewing my templates, I have to consult the docstring again simply to > understand the meaning of the first, second, ..., nth element of the > template. Having CL-style :keyword value plists is much more readable. > > More generally, it would be useful to provide a doct-like functionality > for org-agenda-custom-commands and similar variables (like > org-agenda-custom-commands, or say, also org-latex-classes). > > Best, > Ihor > [-- Attachment #2: Type: text/html, Size: 4222 bytes --]
Mark Barton <mbarton98@gmail.com> writes: > I was concerned about long term support and if I would be left with templates I would have to rewrite again, plus I was too busy at the time to adopt it and it stays in my list of TODOs that are not scheduled. To clarify, I do not think that we need to switch to doct format of the templates. The old format must be kept for backwards-compatibility. > That being said, I use org capture daily, but to the points you raise, I avoid changing or adding to to my capture templates. There is not much problem with current format if you already have working capture templates. They should keep working even if doct gets merged. The problem with current format reveals itself once you need to tweak the existing templates or need to write new templates. doct makes things much easier and much more readable. Best, Ihor
Nicolas Goaziou <mail@nicolasgoaziou.fr> writes: > I think there should be a direct mapping between Customize interface and > values. Adding this macro as a band-aid to simply configuration is not, > IMO, a solution. I think that current customize interface for org-capture-template is perfectly fine. doct aids users who set org-capture-templates programatically. doct also does not break the mapping between customize and the values - it converts the doct syntax into the default org-capture-template format + adds a number of shorthands for capture-related hooks + to capture contexts + provides easier definition of similar templates (via inheritance). For example the below doct definition provides two similar templates without a need to duplicate them: (defcustom org-capture-ref-capture-template `( :group "org-capture-ref template" :type entry ,@org-capture-ref-capture-target :fetch-bibtex (lambda () (org-capture-ref-process-capture)) ; this must run first :link-type (lambda () (org-capture-ref-get-bibtex-field :type)) :extra (lambda () (if (org-capture-ref-get-bibtex-field :journal) (s-join "\n" '("- [ ] [[elisp:(browse-url (url-encode-url (format \"https://sci-hub.se/%s\" (org-entry-get nil \"DOI\"))))][downlaod and attach pdf]]" "- [ ] [[elisp:org-attach-open][read paper capturing interesting references]]" "- [ ] [[elisp:(browse-url (url-encode-url (format \"https://www.semanticscholar.org/search?q=%s\" (org-entry-get nil \"TITLE\"))))][check citing articles]]" "- [ ] [[elisp:(browse-url (url-encode-url (format \"https://www.connectedpapers.com/search?q=%s\" (org-entry-get nil \"TITLE\"))))][check related articles]]" "- [ ] check if bibtex entry has missing fields" "- [ ] Consider subscribing to new citations")) "")) :org-entry (lambda () (org-capture-ref-get-org-entry)) :template ("%{fetch-bibtex}* TODO %?%{space}%{org-entry}" "%{extra}") :children (("Interactive org-capture-ref template" :keys ,(car org-capture-ref-capture-keys) :space " ") ("Silent org-capture-ref template" :keys ,(cadr org-capture-ref-capture-keys) :space "" :immediate-finish t))) "Default capture template. The template is a template defined using `doct' syntax. See docstring of `doct' for details." :type 'list :group 'org-capture-ref) > If capture templates values are too complicated, what about simplifying > them, and possibly use this macro as a temporary solution to help > transition? Could you elaborate what you are referring to? doct is a simplification of otherwise more complex templates. Without doct, users may need to write addition Elisp to define complex templates. With doct, a number of things can be done in a more compact form. A bright example is defining a capture hook that will only be activated for a specific template. doct allows to do it seamlessly, while vanilla syntax will require user to set the hook manually and check the current active template via elisp in a non-obvious ways that require knowing org-capture internals. Best, Ihor
On 20 March 2022 18:19, Ihor Radchenko <yantar92@gmail.com> wrote:
> Also, if anyone agrees with my arguments below, do not stay silent and
> drop a "+1" email below. Otherwise, this whole thing will be stalled.
> There is no point discussing technical aspects, if there is no interest.
It isn't really something that bothers me, or causes my any sort of
problems, given that I have the same set of capture templates for close
to 3 years now, and have never felt the need to change it.
That being said, getting to those capture templates was a pain indeed,
having to, as you alluded to in another e-mail, constantly revisit the
docstring for the precise position of each element, and more often than
not getting them wrong. So I'm all up for it!
Cheers,
--
João Pedro de A. Paula
IT undergraduate at Universidade Federal do Rio Grande do Norte (UFRN)
Hello, Ihor Radchenko <yantar92@gmail.com> writes: > Nicolas Goaziou <mail@nicolasgoaziou.fr> writes: > >> I think there should be a direct mapping between Customize interface and >> values. Adding this macro as a band-aid to simply configuration is not, >> IMO, a solution. > > I think that current customize interface for org-capture-template is > perfectly fine. doct aids users who set org-capture-templates > programatically. I am merely pointing out that the programatically defined value is then no longer compatible with Customize interface. It may be important, or not. >> If capture templates values are too complicated, what about simplifying >> them, and possibly use this macro as a temporary solution to help >> transition? > > Could you elaborate what you are referring to? doct is a simplification > of otherwise more complex templates. Without doct, users may need to > write addition Elisp to define complex templates. With doct, a number of > things can be done in a more compact form. The fact that we need a tool like "doct" to write templates in a compact form may be a sign that the data type is not good enough. Actually it sounds like a failure somehow. Should templates definition be more compact out of the box? Could the data type be made more powerful to permit more complex templates without relying on doct? IOW, can a tool such a doct be made transparent to the user? Regards, -- Nicolas Goaziou
Nicolas Goaziou <mail@nicolasgoaziou.fr> writes: > > Ihor Radchenko <yantar92@gmail.com> writes: > >> Nicolas Goaziou <mail@nicolasgoaziou.fr> writes: >> >>> I think there should be a direct mapping between Customize interface and >>> values. Adding this macro as a band-aid to simply configuration is not, >>> IMO, a solution. >> >> I think that current customize interface for org-capture-template is >> perfectly fine. doct aids users who set org-capture-templates >> programatically. > > I am merely pointing out that the programatically defined value is then > no longer compatible with Customize interface. It may be important, or > not. > This could be an issue for users who use both customise and programatically modify their configuration. <snip> > The fact that we need a tool like "doct" to write templates in a compact > form may be a sign that the data type is not good enough. Actually it > sounds like a failure somehow. > > Should templates definition be more compact out of the box? Could the > data type be made more powerful to permit more complex templates without > relying on doct? IOW, can a tool such a doct be made transparent to the > user? > Yes, I would agree. Such tools are generally a 'code smell'. Underlying data structures should be transparent and not require helper functions to assist the user in creating or updating them. While I can see how doct can help with some use cases, I'm a little concerned about the mixing of both setting of the org-capture-templatges variable and adding/manipulating hooks together in one function. It feels like two separate concerns being meshed into one function/macro for convenience and attempting to work around limitations in the existing system. As you say, doct might be useful as a transition aid, but perhaps the correct fix is to make it largely unnecessary (which I would assume would also provide the additional functionality it offers to users of customise and not only to those who configure their setup programatically). From a personal perspective, I've never needed the additional functionality offered by doct and while writing a capture template does require lots of referencing of documentation, I suspect that even with doct, I would still need to do that as I simply don't write new capture templates often enough to remember all the options (even if they are named rather than positional arguments).
Nicolas Goaziou <mail@nicolasgoaziou.fr> writes: > The fact that we need a tool like "doct" to write templates in a > compact > form may be a sign that the data type is not good enough. > Actually it > sounds like a failure somehow. Agreed. That's why I wrote doct in the first place. As others have expressed, there's far too many elements required to ask the user to recall what position they fall in. A plist is better in that the keys describe their values and can be specified in any order. This also makes it easier for people to share templates, because they can actually tell what the template is intended to do by looking at it. Another thing that is confusing about "capture templates" is that the words "template" and "entry" are overloaded in the documentation. From the docstring of org-capture-templates: > Templates for the creation of new entries. > Each entry is a list with the following items: So org-capture-templates is a "list of entries which create new entries via templates"? Confusing for me now, let alone the first time I tried to write my own templates. doct also does more to check the validity of the template's form when it is defined. I prioritized this because the worst time to figure out your template is malformed is when you go to capture something in the middle of another, unrelated task. > Should templates definition be more compact out of the box? > Could the > data type be made more powerful to permit more complex templates > without > relying on doct? IOW, can a tool such a doct be made transparent > to the > user? My initial idea for integrating doct with Org was to just provide a user option, e.g. org-capture-use-doct-syntax. Then the only thing needed would be a conditional in org-capture-upgrade-templates which would delegate to doct when said user option is non-nil. That would allow those who are interested in using the feature to opt-in without breaking existing templates. Then as support for older syntax is dropped, (if ever? it looks like `org-capture-upgrade-templates' dates back to early 2017, and I think it makes sense to retain such a path) those features could be moved into org-capture. I'm open to bringing doct's features into Org mode, but I'd prefer it not to be spread out over another two years.
+1 -- Jens Reimer
No Wayman <iarchivedmywholelife@gmail.com> writes:
> I'm open to bringing doct's features into Org mode, but I'd prefer
> it not to be spread out over another two years.
Proper integration of doct into org-capture.el would indeed be ideal. I
am happy that you are ok with such option.
Practically, we can do it step-by-step in separate patches:
1. Introduce alist format for org-capture-templates. No new
functionality yet, just change the existing supported template
options to :keyword value pairs. Also, make sure that we keep
backwards compatibility.
2. Extend the alist with new options, like ${keyword} expansion, hooks,
contexts, children, etc Maybe in a sequence of patches.
3. Provide validation of capture templates.
WDYT?
Best,
Ihor
Ihor Radchenko <yantar92@gmail.com> writes: > 2. Extend the alist with new options, like ${keyword} expansion, hooks, > contexts, children, etc Maybe in a sequence of patches. For reference, someone proposed a patch to extend org-capture-templates and support per-template hooks: https://list.orgmode.org/orgmode/20220206140551.1263130-1-herr.valentin.mann@gmail.com/ Best, Ihor
[-- Attachment #1: Type: text/plain, Size: 1172 bytes --] On Sat, Mar 26, 2022 at 4:31 AM Ihor Radchenko <yantar92@gmail.com> wrote: > No Wayman <iarchivedmywholelife@gmail.com> writes: > > > I'm open to bringing doct's features into Org mode, but I'd prefer > > it not to be spread out over another two years. > > Proper integration of doct into org-capture.el would indeed be ideal. I > am happy that you are ok with such option. > > Practically, we can do it step-by-step in separate patches: > 1. Introduce alist format for org-capture-templates. No new > functionality yet, just change the existing supported template > options to :keyword value pairs. Also, make sure that we keep > backwards compatibility. > 2. Extend the alist with new options, like ${keyword} expansion, hooks, > contexts, children, etc Maybe in a sequence of patches. > 3. Provide validation of capture templates. > > WDYT? > > Best, > Ihor > I'll just say that I would be an enthusiastic adopter of the new syntax (just one data point). I agree wholeheartedly that named properties in plists are much easier to remember and adjust than positional arguments, especially when there are lots of the latter. So, I hope this move sforward. [-- Attachment #2: Type: text/html, Size: 1678 bytes --]