That seems hierarchical, which is ok (as in org-mode itself) but how about implementing a more general graph mechanism, which could be used to do this but more flexibly?

On Sat, 14 Dec 2019, 21:04 Gustav Wikström, <gustav@whil.se> wrote:
Hi list and all honored readers!

I have an idea. One that I've mentioned before in side notes. And I want to emphasize that this still only is an idea. But I want to present this idea to you. As a way to gather feedback and get input. Maybe the idea is stupid!? Maybe you think it's already solved? Or maybe it's not, and lots of you resonate with it as well. In any case, please let me know what you think on the piece below!

                    ________________________________

                     ORG MODE: COLLECTIONS/PROJECTS

                            Gustav Wikström
                    ________________________________


Table of Contents
_________________

1. Motivation
2. Idea
3. Benefit
.. 1. For the user
.. 2. For the developer
4. Example use cases
.. 1. Separate actions from reference
.. 2. Work / Personal separation
.. 3. Separated book library
.. 4. More?
5. Risks and challenges
.. 1. Which configuration to use?
.. 2. Should project config allow local variables?
..... 1. How to initialize the local variables?
.. 3. Conflict with other customizations
.. 4. Files that belong to multiple collections
.. 5. Dynamic lists of files and folders for a collection?
6. Alternatives
7. References


1 Motivation
============

  Org mode is more than a major mode for emacs buffers. That has been
  clear for quite some time. Org mode can operate on sets of files.
  Consolidate TODO's and calendar information into custom views. Publish
  sets of files. To do this Org mode assumes that the user has a
  configuration for each of those features. Each feature is responsible
  for maintaining its own context. And almost all of that context has
  to be set globally. So even though Org mode has commands and features
  that operate on sets of files and folders it has not yet developed
  that in a congruent, extensible and composable way. Thus, for the
  sanity of our users and developers I think it's time to ... introduce
  another concept! One that hopefully can simplify things both for users
  and developers.


2 Idea
======

  I propose to introduce `Collection' as a concept in the realm of Org
  mode. [1]

  An Org mode collection is defined as the combination of:
  1. A short name and description
  2. A collection of Org mode documents
  3. A collection of files and/or folders called attachments and
     attachment-locations for the project
  4. A collection of configurations for the given project

  Globally available collections are defined in a list,
  `org-collections'. Org mode should include a safe parameter that can
  be set as a folder customization to the local active project,
  `org-collections-active'. The default should be to the first entry in
  `org-collections' unless customized. This local parameter would be
  used to instruct Emacs and Org mode on which collection is active.
  Only one collection at a time can be active.

  Org agenda should use `org-collections-active' as default for the
  collection of Org mode documents to operate on. Org agenda should get
  a new command to switch between active projects.

  I'm thinking that there could be a special Emacs major mode for the
  collection as well, called "Org collections mode". Not sure exactly
  what to display and how to represent the project there... But
  certainly some kind of list of included documents and attachments.
  When in that mode there should possibly be single key
  keyboard-shortcuts to the most important features that operate on the
  collection. And switch between them.


3 Benefit
=========

3.1 For the user
~~~~~~~~~~~~~~~~

  A user would gain mainly two benefits as I can see right now:
  1. The ability to clearly define (multiple) collections of files that
     belong together across org mode, with unique configurations.
  2. Less global configuration state to manage and worry about!

  The second point might not look like much but is sooo important! Most
  programmers know that global state should be avoided. Putting things
  in a context most of the time makes things better. And if we can
  configure Org mode connected to a context it makes it much more useful
  for those who use Org mode for multiple purposes.

  The first point is equally important in my opinion. Today one must
  configure Org mode per feature. If you want to configure publishing
  you do that globally. If you want to configure the agenda, you have to
  do that globally as well. If you want to define a location for
  attachments, do it globally! What about custom TODO-keywords? Do it
  globally! Track ID-locations? Define a location globally!

  All above adds cognitive load to the user and makes it difficult to
  maintain the configuration as the use of Org mode grows (as it should
  ;) ). You have to define the context for each and every feature for it
  to know what to operate on. I claim that both the human psyche and the
  system itself will have a much more easy time if it could configure
  these features together, in a given context!


3.2 For the developer
~~~~~~~~~~~~~~~~~~~~~

  I claim there will be benefits for developers as well. Today there
  exists many packages that extend Org mode functionality. Many work
  with the idea of collections. Some that come to mind:
  - Org brain (<https://github.com/Kungsgeten/org-brain>)
  - Org ql (<https://github.com/alphapapa/org-ql>)
  - Ox hugo (<https://ox-hugo.scripter.co/>)

  I think that with the addition of the `collections' concept into Org
  mode, package developers get a concept they can easily attach to. Yes,
  you can easily define your own package-specific concept for that as
  well. But then the user loses out in having to configure another
  feature. And yes, today you as a developer can say that Org agenda
  will be my collection to operate on. But this is a big limitation
  since it limits what your package effectively can only work to a
  single list of files.

  Having a collections concept means you as a developer have another
  base on which you can extend. No need to define your own concept if
  `Org-agenda-files' isn't enough; make it work together with
  `org-collections' instead. Org mode users will be happy because what
  they have already defined as important for them can be reused for new
  things with ease.

  Developing features inside Org mode itself hopefully also can benefit
  from this concept. I'm sure there are many people out there with cool
  ideas on how to extend and work with Org documents. And I'm equally
  sure that the value of developing many of those features will be
  bigger if they could naturally attach to an Org collections
  definition!


4 Example use cases
===================

4.1 Separate actions from reference
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

  One practice promoted by GTD is to separate actionable items from
  reference information. While that practice can be overcome by search
  etc. some might still value a clear separation.

  Want to look up something related to my general references? Search the
  Org collection related to reference-information! Maybe set up custom
  views and uses of TODO keywords for reference information for special
  agenda views.

  Want to only display not yet finished tasks? Switch to the Org
  collection for actionable items and browse away.


4.2 Work / Personal separation
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

  The heading says it all. Some like to separate work and personal stuff
  out from each other. What more clear way to do that than can there be
  than to separate them into their own Org collections? That way you
  potentially could let your work-related workflow (I.e. TODO-keywords)
  be different than the personal workflow. Without having to think about
  a global configuration that has to allow for both.


4.3 Separated book library
~~~~~~~~~~~~~~~~~~~~~~~~~~

  Org mode can be used as a media manager of sort. Just define your
  conventions for the Org collection using TODO-keywords, categories and
  properties. Attach the e-books you have as attachments in an
  attachment-scheme special for your book library. Configure export of
  the library using maybe a custom HTML/CSS-visual and publish it
  somewhere for yourself to look at when on your phone. And do this
  without having to think of how changing all these things will affect
  the global state of Org mode, potentially messing up your other uses
  for task management or other notes and libraries you're trying to
  manage!

  Note that one can still have a holistic view on all Org mode documents
  as well, if important. It only requires a definition of a collection
  as the collection of all other collections!


4.4 More?
~~~~~~~~~

  Please add more ideas when you think of them!


5 Risks and challenges
======================

5.1 Which configuration to use?
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

  When I'm visiting a file that belongs to a collection, how should
  Emacs resolve configurations for that file?

  There may be configurations in the following places:
  - Global in `emacs-custom.el' or `.emacs.d/init.el'
  - Directory local variables in the tree
  - File local variables
  - Local variables for the project definition in which the file
    belongs?

  Should visiting a file always have to scan the collections list to see
  if the file belongs to any of them, in order to load customizations?
  Hmm... Maybe!? Or - maybe not if Emacs can rely on the fact that the
  user cares to set the local variable `org-collections-active' (or
  whatever it should be called)? In that case, just evaluate the
  settings for that project without doing any scan.


5.2 Should project config allow local variables?
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

  Should the collections definition allow customization of variables
  that apply for Org mode features? Hmm... Maybe!? One thing that comes
  to mind is that a project should be able to define a custom attachment
  directory... How else would the attachment-feature know what
  attachment directory to use for files in that collection?

  Another option could ofc. be that each feature would have to add
  support for looking into the collection definition and override the
  local variable. But that will add development effort and complexity to
  each feature. Not suggested.


5.2.1 How to initialize the local variables?
--------------------------------------------

  When visiting a file that belongs to a collection, should Emacs at
  that point initialize the collection-configuration for that
  collection? Ideally some kind of collection-resolution would be made.
  Otherwise users will get strange behaviors when the think they are in
  one project but Org mode hasn't changed the local variables to match
  it. On the other hand, it doesn't sound very performant to have to
  check collection-belonging every time an Org mode file is visited!

  Possibly solve this with a variable that can be localized -
  `org-collections-active'?


5.3 Conflict with other customizations
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

  Maybe I've defined an attachment directory as a directory local
  variables in a folder, for all subfolders and files to inherit. Should
  collection-customizations override that? Or should the directory local
  variables take precedence?

  Maybe could be solved by letting the (advanced) user choose using a
  customization itself, something like
  `org-collections-precede-local-variables' ? Need a intuitive default
  though. Most sane default is probably to let local variables take
  precedence. Those are created by the user anyways, so she should be
  aware.

  The more I think of it, there shouldn't be a customization for this at
  all. I think local definitions always should override the collection
  definition.


5.4 Files that belong to multiple collections
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

  What if I'm being a clever user and define multiple collections for
  the same files (I.e. overlap in the Venn-diagram of files grouped by
  collections). Which collection is "active" when I'm visiting the file?

  This depends on if Emacs should evaluate the collection-settings for
  each file visit or not. If they are evaluated for each file visit then
  the first matching project in the list of collections should apply for
  that file. If a cache is created that lists file and collection
  relationships then each file should relate to a list of collections
  where the first collection in that list should apply.

  If Emacs can rely on `org-collections-active' being set, then the
  collection referenced there should be used.


5.5 Dynamic lists of files and folders for a collection?
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

  Should the list of files allow for folders with recursion and patterns
  should it be required to provide a fixed defined list of files?

  Preferably the same way as `org-agenda-files' work today. Maybe some
  kind of caching-mechanism is needed though, for commands that might
  have to look for file, collection relations. A cache adds potential
  pain for the user though. If a file is added to a folder in a
  collection and a "collection-command" is run then the new file might
  not show up in the results anyway... So the user will be affected by
  caching and will have to know about it. Not good...


6 Alternatives
==============

  Doing research for this feature made me realize that much of what I'm
  proposing already exist! In another form though, as [directory
  variables]. That requires customizations to be defined as safe though.
  And today some of the things I would consider to define a collection
  aren't safe. For example `org-agenda-files', `org-todo-keywords',
  `org-publish-project-alist'.

  Some issues with relying on directory variables (Assuming they also
  are made safe):
  - When invoking Org agenda I will have to first visit a file inside a
    specific folder to get the agenda for the correct project
  - ....


[directory variables] <info:emacs#Directory variables>


7 References
============

  I've mentioned this idea the Org mode mailing list previously, but
  only as short side notes to other topics:
  - <https://lists.gnu.org/archive/html/emacs-orgmode/2018-11/msg00211.html>
  - <https://lists.gnu.org/archive/html/emacs-orgmode/2019-09/msg00010.html>

  Note that I've talked about it as "project". I think that name still
  could be considered instead of "collection". Collection is more
  general and less overloaded in terms of productivity software. And it
  shifts the focus away from task management a bit, which I think can be
  a good thing. Because while Org mode may often start to be used as a
  task/project manager software, it's useful in a much wider context
  than that!



Footnotes
_________

[1] I've previously written about this as "Projects". While Project
was my initial name for this feature I think collection may be a
better option. For the sake of this text both options work just fine.
The idea is the same.