Re: [Idea] Org Collections

[Tim Cross <theophilusx@gmail.com>]

In general, I like the idea. While you are correct that most of what you
refer to can be achieved using existing org functionality, I like the
clear separation 'collections' would offer - it is like a namespace for
a collection of org documents. I like the idea of being able to isolate
customization/configuration on a per collection basis - it would
simplify some of my existing customisation plus I could experiment with
new/alternative setups in a manner which is less likely to adversely
impact my existing setup. I also suspect agenda generation and searching
could become more efficient if we can easily isolate such activities to
a collection (would still want 'global' options though). 

I would start by keeping it as simple as possible and I would restrict
configuration options (at least initially) to make things cleaner to
begin with. The .dir-locals approach has some benefits, but I've lost
count of the amount of time wasted trying to track down some
unwanted/incorrect setting only to realise it was from a .dir-locals I'd
forgotten about.

I do work in a number of different contexts - different employers,
contracts, etc. Currently, tags and properties are very useful in such
contexts. However, complete separation of these concerns into
collections sounds very appealing. Setup for a new contract/project
would be very simple and managing the different requirements would be
much easier (for example, these days, I often have to produce documents
with the correct format, logo, colour scheme etc). I have a growing set
of document style definitions that would be easier to manage on a per
collection basis rather than as a global setting).

In some of my work, I have restrictions on where data can be
stored. For example, I might be required to store all documents, data
etc relating to one contract only on a storage device owned by the
contractor. At the same time, I don't want any of my personal or
unrelated contract data stored on the corporate storage server. This can
be a little challenging when it comes to things like capture or
attachments. However, if I could have collections with completely
different 'roots' and have all my capture templates and attachment
actions apply relative to that root, then life might be easier.

Its a +1 from me. 

Gustav Wikström <gustav@whil.se> writes:

> Hi list and all honored readers!
>
> I have an idea. One that I've mentioned before in side notes. And I want to emphasize that this still only is an idea. But I want to present this idea to you. As a way to gather feedback and get input. Maybe the idea is stupid!? Maybe you think it's already solved? Or maybe it's not, and lots of you resonate with it as well. In any case, please let me know what you think on the piece below!
>
>                     ________________________________
>
>                      ORG MODE: COLLECTIONS/PROJECTS
>
>                             Gustav Wikström
>                     ________________________________
>
>
> Table of Contents
> _________________
>
> 1. Motivation
> 2. Idea
> 3. Benefit
> .. 1. For the user
> .. 2. For the developer
> 4. Example use cases
> .. 1. Separate actions from reference
> .. 2. Work / Personal separation
> .. 3. Separated book library
> .. 4. More?
> 5. Risks and challenges
> .. 1. Which configuration to use?
> .. 2. Should project config allow local variables?
> ..... 1. How to initialize the local variables?
> .. 3. Conflict with other customizations
> .. 4. Files that belong to multiple collections
> .. 5. Dynamic lists of files and folders for a collection?
> 6. Alternatives
> 7. References
>
>
> 1 Motivation
> ============
>
>   Org mode is more than a major mode for emacs buffers. That has been
>   clear for quite some time. Org mode can operate on sets of files.
>   Consolidate TODO's and calendar information into custom views. Publish
>   sets of files. To do this Org mode assumes that the user has a
>   configuration for each of those features. Each feature is responsible
>   for maintaining its own context. And almost all of that context has
>   to be set globally. So even though Org mode has commands and features
>   that operate on sets of files and folders it has not yet developed
>   that in a congruent, extensible and composable way. Thus, for the
>   sanity of our users and developers I think it's time to ... introduce
>   another concept! One that hopefully can simplify things both for users
>   and developers.
>
>
> 2 Idea
> ======
>
>   I propose to introduce `Collection' as a concept in the realm of Org
>   mode. [1]
>
>   An Org mode collection is defined as the combination of:
>   1. A short name and description
>   2. A collection of Org mode documents
>   3. A collection of files and/or folders called attachments and
>      attachment-locations for the project
>   4. A collection of configurations for the given project
>
>   Globally available collections are defined in a list,
>   `org-collections'. Org mode should include a safe parameter that can
>   be set as a folder customization to the local active project,
>   `org-collections-active'. The default should be to the first entry in
>   `org-collections' unless customized. This local parameter would be
>   used to instruct Emacs and Org mode on which collection is active.
>   Only one collection at a time can be active.
>
>   Org agenda should use `org-collections-active' as default for the
>   collection of Org mode documents to operate on. Org agenda should get
>   a new command to switch between active projects.
>
>   I'm thinking that there could be a special Emacs major mode for the
>   collection as well, called "Org collections mode". Not sure exactly
>   what to display and how to represent the project there... But
>   certainly some kind of list of included documents and attachments.
>   When in that mode there should possibly be single key
>   keyboard-shortcuts to the most important features that operate on the
>   collection. And switch between them.
>
>
> 3 Benefit
> =========
>
> 3.1 For the user
> ~~~~~~~~~~~~~~~~
>
>   A user would gain mainly two benefits as I can see right now:
>   1. The ability to clearly define (multiple) collections of files that
>      belong together across org mode, with unique configurations.
>   2. Less global configuration state to manage and worry about!
>
>   The second point might not look like much but is sooo important! Most
>   programmers know that global state should be avoided. Putting things
>   in a context most of the time makes things better. And if we can
>   configure Org mode connected to a context it makes it much more useful
>   for those who use Org mode for multiple purposes.
>
>   The first point is equally important in my opinion. Today one must
>   configure Org mode per feature. If you want to configure publishing
>   you do that globally. If you want to configure the agenda, you have to
>   do that globally as well. If you want to define a location for
>   attachments, do it globally! What about custom TODO-keywords? Do it
>   globally! Track ID-locations? Define a location globally!
>
>   All above adds cognitive load to the user and makes it difficult to
>   maintain the configuration as the use of Org mode grows (as it should
>   ;) ). You have to define the context for each and every feature for it
>   to know what to operate on. I claim that both the human psyche and the
>   system itself will have a much more easy time if it could configure
>   these features together, in a given context!
>
>
> 3.2 For the developer
> ~~~~~~~~~~~~~~~~~~~~~
>
>   I claim there will be benefits for developers as well. Today there
>   exists many packages that extend Org mode functionality. Many work
>   with the idea of collections. Some that come to mind:
>   - Org brain (<https://github.com/Kungsgeten/org-brain>)
>   - Org ql (<https://github.com/alphapapa/org-ql>)
>   - Ox hugo (<https://ox-hugo.scripter.co/>)
>
>   I think that with the addition of the `collections' concept into Org
>   mode, package developers get a concept they can easily attach to. Yes,
>   you can easily define your own package-specific concept for that as
>   well. But then the user loses out in having to configure another
>   feature. And yes, today you as a developer can say that Org agenda
>   will be my collection to operate on. But this is a big limitation
>   since it limits what your package effectively can only work to a
>   single list of files.
>
>   Having a collections concept means you as a developer have another
>   base on which you can extend. No need to define your own concept if
>   `Org-agenda-files' isn't enough; make it work together with
>   `org-collections' instead. Org mode users will be happy because what
>   they have already defined as important for them can be reused for new
>   things with ease.
>
>   Developing features inside Org mode itself hopefully also can benefit
>   from this concept. I'm sure there are many people out there with cool
>   ideas on how to extend and work with Org documents. And I'm equally
>   sure that the value of developing many of those features will be
>   bigger if they could naturally attach to an Org collections
>   definition!
>
>
> 4 Example use cases
> ===================
>
> 4.1 Separate actions from reference
> ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
>
>   One practice promoted by GTD is to separate actionable items from
>   reference information. While that practice can be overcome by search
>   etc. some might still value a clear separation.
>
>   Want to look up something related to my general references? Search the
>   Org collection related to reference-information! Maybe set up custom
>   views and uses of TODO keywords for reference information for special
>   agenda views.
>
>   Want to only display not yet finished tasks? Switch to the Org
>   collection for actionable items and browse away.
>
>
> 4.2 Work / Personal separation
> ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
>
>   The heading says it all. Some like to separate work and personal stuff
>   out from each other. What more clear way to do that than can there be
>   than to separate them into their own Org collections? That way you
>   potentially could let your work-related workflow (I.e. TODO-keywords)
>   be different than the personal workflow. Without having to think about
>   a global configuration that has to allow for both.
>
>
> 4.3 Separated book library
> ~~~~~~~~~~~~~~~~~~~~~~~~~~
>
>   Org mode can be used as a media manager of sort. Just define your
>   conventions for the Org collection using TODO-keywords, categories and
>   properties. Attach the e-books you have as attachments in an
>   attachment-scheme special for your book library. Configure export of
>   the library using maybe a custom HTML/CSS-visual and publish it
>   somewhere for yourself to look at when on your phone. And do this
>   without having to think of how changing all these things will affect
>   the global state of Org mode, potentially messing up your other uses
>   for task management or other notes and libraries you're trying to
>   manage!
>
>   Note that one can still have a holistic view on all Org mode documents
>   as well, if important. It only requires a definition of a collection
>   as the collection of all other collections!
>
>
> 4.4 More?
> ~~~~~~~~~
>
>   Please add more ideas when you think of them!
>
>
> 5 Risks and challenges
> ======================
>
> 5.1 Which configuration to use?
> ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
>
>   When I'm visiting a file that belongs to a collection, how should
>   Emacs resolve configurations for that file?
>
>   There may be configurations in the following places:
>   - Global in `emacs-custom.el' or `.emacs.d/init.el'
>   - Directory local variables in the tree
>   - File local variables
>   - Local variables for the project definition in which the file
>     belongs?
>
>   Should visiting a file always have to scan the collections list to see
>   if the file belongs to any of them, in order to load customizations?
>   Hmm... Maybe!? Or - maybe not if Emacs can rely on the fact that the
>   user cares to set the local variable `org-collections-active' (or
>   whatever it should be called)? In that case, just evaluate the
>   settings for that project without doing any scan.
>
>
> 5.2 Should project config allow local variables?
> ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
>
>   Should the collections definition allow customization of variables
>   that apply for Org mode features? Hmm... Maybe!? One thing that comes
>   to mind is that a project should be able to define a custom attachment
>   directory... How else would the attachment-feature know what
>   attachment directory to use for files in that collection?
>
>   Another option could ofc. be that each feature would have to add
>   support for looking into the collection definition and override the
>   local variable. But that will add development effort and complexity to
>   each feature. Not suggested.
>
>
> 5.2.1 How to initialize the local variables?
> --------------------------------------------
>
>   When visiting a file that belongs to a collection, should Emacs at
>   that point initialize the collection-configuration for that
>   collection? Ideally some kind of collection-resolution would be made.
>   Otherwise users will get strange behaviors when the think they are in
>   one project but Org mode hasn't changed the local variables to match
>   it. On the other hand, it doesn't sound very performant to have to
>   check collection-belonging every time an Org mode file is visited!
>
>   Possibly solve this with a variable that can be localized -
>   `org-collections-active'?
>
>
> 5.3 Conflict with other customizations
> ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
>
>   Maybe I've defined an attachment directory as a directory local
>   variables in a folder, for all subfolders and files to inherit. Should
>   collection-customizations override that? Or should the directory local
>   variables take precedence?
>
>   Maybe could be solved by letting the (advanced) user choose using a
>   customization itself, something like
>   `org-collections-precede-local-variables' ? Need a intuitive default
>   though. Most sane default is probably to let local variables take
>   precedence. Those are created by the user anyways, so she should be
>   aware.
>
>   The more I think of it, there shouldn't be a customization for this at
>   all. I think local definitions always should override the collection
>   definition.
>
>
> 5.4 Files that belong to multiple collections
> ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
>
>   What if I'm being a clever user and define multiple collections for
>   the same files (I.e. overlap in the Venn-diagram of files grouped by
>   collections). Which collection is "active" when I'm visiting the file?
>
>   This depends on if Emacs should evaluate the collection-settings for
>   each file visit or not. If they are evaluated for each file visit then
>   the first matching project in the list of collections should apply for
>   that file. If a cache is created that lists file and collection
>   relationships then each file should relate to a list of collections
>   where the first collection in that list should apply.
>
>   If Emacs can rely on `org-collections-active' being set, then the
>   collection referenced there should be used.
>
>
> 5.5 Dynamic lists of files and folders for a collection?
> ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
>
>   Should the list of files allow for folders with recursion and patterns
>   should it be required to provide a fixed defined list of files?
>
>   Preferably the same way as `org-agenda-files' work today. Maybe some
>   kind of caching-mechanism is needed though, for commands that might
>   have to look for file, collection relations. A cache adds potential
>   pain for the user though. If a file is added to a folder in a
>   collection and a "collection-command" is run then the new file might
>   not show up in the results anyway... So the user will be affected by
>   caching and will have to know about it. Not good...
>
>
> 6 Alternatives
> ==============
>
>   Doing research for this feature made me realize that much of what I'm
>   proposing already exist! In another form though, as [directory
>   variables]. That requires customizations to be defined as safe though.
>   And today some of the things I would consider to define a collection
>   aren't safe. For example `org-agenda-files', `org-todo-keywords',
>   `org-publish-project-alist'.
>
>   Some issues with relying on directory variables (Assuming they also
>   are made safe):
>   - When invoking Org agenda I will have to first visit a file inside a
>     specific folder to get the agenda for the correct project
>   - ....
>
>
> [directory variables] <info:emacs#Directory variables>
>
>
> 7 References
> ============
>
>   I've mentioned this idea the Org mode mailing list previously, but
>   only as short side notes to other topics:
>   - <https://lists.gnu.org/archive/html/emacs-orgmode/2018-11/msg00211.html>
>   - <https://lists.gnu.org/archive/html/emacs-orgmode/2019-09/msg00010.html>
>
>   Note that I've talked about it as "project". I think that name still
>   could be considered instead of "collection". Collection is more
>   general and less overloaded in terms of productivity software. And it
>   shifts the focus away from task management a bit, which I think can be
>   a good thing. Because while Org mode may often start to be used as a
>   task/project manager software, it's useful in a much wider context
>   than that!
>
>
>
> Footnotes
> _________
>
> [1] I've previously written about this as "Projects". While Project
> was my initial name for this feature I think collection may be a
> better option. For the sake of this text both options work just fine.
> The idea is the same.


-- 
Tim Cross