From mboxrd@z Thu Jan 1 00:00:00 1970 From: Tim Cross Subject: Re: [Idea] Org Collections Date: Sun, 15 Dec 2019 09:25:56 +1100 Message-ID: <87r216ilxn.fsf@gmail.com> References: Mime-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable Return-path: Received: from eggs.gnu.org ([2001:470:142:3::10]:56070) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1igFrY-0001XP-5z for emacs-orgmode@gnu.org; Sat, 14 Dec 2019 17:26:15 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1igFrV-0002av-C5 for emacs-orgmode@gnu.org; Sat, 14 Dec 2019 17:26:12 -0500 Received: from mail-pf1-x431.google.com ([2607:f8b0:4864:20::431]:35968) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1igFrU-0002ZW-Io for emacs-orgmode@gnu.org; Sat, 14 Dec 2019 17:26:09 -0500 Received: by mail-pf1-x431.google.com with SMTP id x184so3468975pfb.3 for ; Sat, 14 Dec 2019 14:26:04 -0800 (PST) Received: from tim-desktop (2001-44b8-31f2-bb00-6ece-be6a-2aa8-e698.static.ipv6.internode.on.net. [2001:44b8:31f2:bb00:6ece:be6a:2aa8:e698]) by smtp.gmail.com with ESMTPSA id d14sm14208610pjz.12.2019.12.14.14.26.00 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sat, 14 Dec 2019 14:26:01 -0800 (PST) In-reply-to: List-Id: "General discussions about Org-mode." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: emacs-orgmode-bounces+geo-emacs-orgmode=m.gmane.org@gnu.org Sender: "Emacs-orgmode" To: emacs-orgmode@gnu.org In general, I like the idea. While you are correct that most of what you refer to can be achieved using existing org functionality, I like the clear separation 'collections' would offer - it is like a namespace for a collection of org documents. I like the idea of being able to isolate customization/configuration on a per collection basis - it would simplify some of my existing customisation plus I could experiment with new/alternative setups in a manner which is less likely to adversely impact my existing setup. I also suspect agenda generation and searching could become more efficient if we can easily isolate such activities to a collection (would still want 'global' options though).=20 I would start by keeping it as simple as possible and I would restrict configuration options (at least initially) to make things cleaner to begin with. The .dir-locals approach has some benefits, but I've lost count of the amount of time wasted trying to track down some unwanted/incorrect setting only to realise it was from a .dir-locals I'd forgotten about. I do work in a number of different contexts - different employers, contracts, etc. Currently, tags and properties are very useful in such contexts. However, complete separation of these concerns into collections sounds very appealing. Setup for a new contract/project would be very simple and managing the different requirements would be much easier (for example, these days, I often have to produce documents with the correct format, logo, colour scheme etc). I have a growing set of document style definitions that would be easier to manage on a per collection basis rather than as a global setting). In some of my work, I have restrictions on where data can be stored. For example, I might be required to store all documents, data etc relating to one contract only on a storage device owned by the contractor. At the same time, I don't want any of my personal or unrelated contract data stored on the corporate storage server. This can be a little challenging when it comes to things like capture or attachments. However, if I could have collections with completely different 'roots' and have all my capture templates and attachment actions apply relative to that root, then life might be easier. Its a +1 from me.=20 Gustav Wikstr=C3=B6m writes: > Hi list and all honored readers! > > I have an idea. One that I've mentioned before in side notes. And I want = to emphasize that this still only is an idea. But I want to present this id= ea to you. As a way to gather feedback and get input. Maybe the idea is stu= pid!? Maybe you think it's already solved? Or maybe it's not, and lots of y= ou resonate with it as well. In any case, please let me know what you think= on the piece below! > > ________________________________ > > ORG MODE: COLLECTIONS/PROJECTS > > Gustav Wikstr=C3=B6m > ________________________________ > > > Table of Contents > _________________ > > 1. Motivation > 2. Idea > 3. Benefit > .. 1. For the user > .. 2. For the developer > 4. Example use cases > .. 1. Separate actions from reference > .. 2. Work / Personal separation > .. 3. Separated book library > .. 4. More? > 5. Risks and challenges > .. 1. Which configuration to use? > .. 2. Should project config allow local variables? > ..... 1. How to initialize the local variables? > .. 3. Conflict with other customizations > .. 4. Files that belong to multiple collections > .. 5. Dynamic lists of files and folders for a collection? > 6. Alternatives > 7. References > > > 1 Motivation > =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D > > Org mode is more than a major mode for emacs buffers. That has been > clear for quite some time. Org mode can operate on sets of files. > Consolidate TODO's and calendar information into custom views. Publish > sets of files. To do this Org mode assumes that the user has a > configuration for each of those features. Each feature is responsible > for maintaining its own context. And almost all of that context has > to be set globally. So even though Org mode has commands and features > that operate on sets of files and folders it has not yet developed > that in a congruent, extensible and composable way. Thus, for the > sanity of our users and developers I think it's time to ... introduce > another concept! One that hopefully can simplify things both for users > and developers. > > > 2 Idea > =3D=3D=3D=3D=3D=3D > > I propose to introduce `Collection' as a concept in the realm of Org > mode. [1] > > An Org mode collection is defined as the combination of: > 1. A short name and description > 2. A collection of Org mode documents > 3. A collection of files and/or folders called attachments and > attachment-locations for the project > 4. A collection of configurations for the given project > > Globally available collections are defined in a list, > `org-collections'. Org mode should include a safe parameter that can > be set as a folder customization to the local active project, > `org-collections-active'. The default should be to the first entry in > `org-collections' unless customized. This local parameter would be > used to instruct Emacs and Org mode on which collection is active. > Only one collection at a time can be active. > > Org agenda should use `org-collections-active' as default for the > collection of Org mode documents to operate on. Org agenda should get > a new command to switch between active projects. > > I'm thinking that there could be a special Emacs major mode for the > collection as well, called "Org collections mode". Not sure exactly > what to display and how to represent the project there... But > certainly some kind of list of included documents and attachments. > When in that mode there should possibly be single key > keyboard-shortcuts to the most important features that operate on the > collection. And switch between them. > > > 3 Benefit > =3D=3D=3D=3D=3D=3D=3D=3D=3D > > 3.1 For the user > ~~~~~~~~~~~~~~~~ > > A user would gain mainly two benefits as I can see right now: > 1. The ability to clearly define (multiple) collections of files that > belong together across org mode, with unique configurations. > 2. Less global configuration state to manage and worry about! > > The second point might not look like much but is sooo important! Most > programmers know that global state should be avoided. Putting things > in a context most of the time makes things better. And if we can > configure Org mode connected to a context it makes it much more useful > for those who use Org mode for multiple purposes. > > The first point is equally important in my opinion. Today one must > configure Org mode per feature. If you want to configure publishing > you do that globally. If you want to configure the agenda, you have to > do that globally as well. If you want to define a location for > attachments, do it globally! What about custom TODO-keywords? Do it > globally! Track ID-locations? Define a location globally! > > All above adds cognitive load to the user and makes it difficult to > maintain the configuration as the use of Org mode grows (as it should > ;) ). You have to define the context for each and every feature for it > to know what to operate on. I claim that both the human psyche and the > system itself will have a much more easy time if it could configure > these features together, in a given context! > > > 3.2 For the developer > ~~~~~~~~~~~~~~~~~~~~~ > > I claim there will be benefits for developers as well. Today there > exists many packages that extend Org mode functionality. Many work > with the idea of collections. Some that come to mind: > - Org brain () > - Org ql () > - Ox hugo () > > I think that with the addition of the `collections' concept into Org > mode, package developers get a concept they can easily attach to. Yes, > you can easily define your own package-specific concept for that as > well. But then the user loses out in having to configure another > feature. And yes, today you as a developer can say that Org agenda > will be my collection to operate on. But this is a big limitation > since it limits what your package effectively can only work to a > single list of files. > > Having a collections concept means you as a developer have another > base on which you can extend. No need to define your own concept if > `Org-agenda-files' isn't enough; make it work together with > `org-collections' instead. Org mode users will be happy because what > they have already defined as important for them can be reused for new > things with ease. > > Developing features inside Org mode itself hopefully also can benefit > from this concept. I'm sure there are many people out there with cool > ideas on how to extend and work with Org documents. And I'm equally > sure that the value of developing many of those features will be > bigger if they could naturally attach to an Org collections > definition! > > > 4 Example use cases > =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D > > 4.1 Separate actions from reference > ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ > > One practice promoted by GTD is to separate actionable items from > reference information. While that practice can be overcome by search > etc. some might still value a clear separation. > > Want to look up something related to my general references? Search the > Org collection related to reference-information! Maybe set up custom > views and uses of TODO keywords for reference information for special > agenda views. > > Want to only display not yet finished tasks? Switch to the Org > collection for actionable items and browse away. > > > 4.2 Work / Personal separation > ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ > > The heading says it all. Some like to separate work and personal stuff > out from each other. What more clear way to do that than can there be > than to separate them into their own Org collections? That way you > potentially could let your work-related workflow (I.e. TODO-keywords) > be different than the personal workflow. Without having to think about > a global configuration that has to allow for both. > > > 4.3 Separated book library > ~~~~~~~~~~~~~~~~~~~~~~~~~~ > > Org mode can be used as a media manager of sort. Just define your > conventions for the Org collection using TODO-keywords, categories and > properties. Attach the e-books you have as attachments in an > attachment-scheme special for your book library. Configure export of > the library using maybe a custom HTML/CSS-visual and publish it > somewhere for yourself to look at when on your phone. And do this > without having to think of how changing all these things will affect > the global state of Org mode, potentially messing up your other uses > for task management or other notes and libraries you're trying to > manage! > > Note that one can still have a holistic view on all Org mode documents > as well, if important. It only requires a definition of a collection > as the collection of all other collections! > > > 4.4 More? > ~~~~~~~~~ > > Please add more ideas when you think of them! > > > 5 Risks and challenges > =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D > > 5.1 Which configuration to use? > ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ > > When I'm visiting a file that belongs to a collection, how should > Emacs resolve configurations for that file? > > There may be configurations in the following places: > - Global in `emacs-custom.el' or `.emacs.d/init.el' > - Directory local variables in the tree > - File local variables > - Local variables for the project definition in which the file > belongs? > > Should visiting a file always have to scan the collections list to see > if the file belongs to any of them, in order to load customizations? > Hmm... Maybe!? Or - maybe not if Emacs can rely on the fact that the > user cares to set the local variable `org-collections-active' (or > whatever it should be called)? In that case, just evaluate the > settings for that project without doing any scan. > > > 5.2 Should project config allow local variables? > ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ > > Should the collections definition allow customization of variables > that apply for Org mode features? Hmm... Maybe!? One thing that comes > to mind is that a project should be able to define a custom attachment > directory... How else would the attachment-feature know what > attachment directory to use for files in that collection? > > Another option could ofc. be that each feature would have to add > support for looking into the collection definition and override the > local variable. But that will add development effort and complexity to > each feature. Not suggested. > > > 5.2.1 How to initialize the local variables? > -------------------------------------------- > > When visiting a file that belongs to a collection, should Emacs at > that point initialize the collection-configuration for that > collection? Ideally some kind of collection-resolution would be made. > Otherwise users will get strange behaviors when the think they are in > one project but Org mode hasn't changed the local variables to match > it. On the other hand, it doesn't sound very performant to have to > check collection-belonging every time an Org mode file is visited! > > Possibly solve this with a variable that can be localized - > `org-collections-active'? > > > 5.3 Conflict with other customizations > ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ > > Maybe I've defined an attachment directory as a directory local > variables in a folder, for all subfolders and files to inherit. Should > collection-customizations override that? Or should the directory local > variables take precedence? > > Maybe could be solved by letting the (advanced) user choose using a > customization itself, something like > `org-collections-precede-local-variables' ? Need a intuitive default > though. Most sane default is probably to let local variables take > precedence. Those are created by the user anyways, so she should be > aware. > > The more I think of it, there shouldn't be a customization for this at > all. I think local definitions always should override the collection > definition. > > > 5.4 Files that belong to multiple collections > ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ > > What if I'm being a clever user and define multiple collections for > the same files (I.e. overlap in the Venn-diagram of files grouped by > collections). Which collection is "active" when I'm visiting the file? > > This depends on if Emacs should evaluate the collection-settings for > each file visit or not. If they are evaluated for each file visit then > the first matching project in the list of collections should apply for > that file. If a cache is created that lists file and collection > relationships then each file should relate to a list of collections > where the first collection in that list should apply. > > If Emacs can rely on `org-collections-active' being set, then the > collection referenced there should be used. > > > 5.5 Dynamic lists of files and folders for a collection? > ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ > > Should the list of files allow for folders with recursion and patterns > should it be required to provide a fixed defined list of files? > > Preferably the same way as `org-agenda-files' work today. Maybe some > kind of caching-mechanism is needed though, for commands that might > have to look for file, collection relations. A cache adds potential > pain for the user though. If a file is added to a folder in a > collection and a "collection-command" is run then the new file might > not show up in the results anyway... So the user will be affected by > caching and will have to know about it. Not good... > > > 6 Alternatives > =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D > > Doing research for this feature made me realize that much of what I'm > proposing already exist! In another form though, as [directory > variables]. That requires customizations to be defined as safe though. > And today some of the things I would consider to define a collection > aren't safe. For example `org-agenda-files', `org-todo-keywords', > `org-publish-project-alist'. > > Some issues with relying on directory variables (Assuming they also > are made safe): > - When invoking Org agenda I will have to first visit a file inside a > specific folder to get the agenda for the correct project > - .... > > > [directory variables] > > > 7 References > =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D > > I've mentioned this idea the Org mode mailing list previously, but > only as short side notes to other topics: > - > - > > Note that I've talked about it as "project". I think that name still > could be considered instead of "collection". Collection is more > general and less overloaded in terms of productivity software. And it > shifts the focus away from task management a bit, which I think can be > a good thing. Because while Org mode may often start to be used as a > task/project manager software, it's useful in a much wider context > than that! > > > > Footnotes > _________ > > [1] I've previously written about this as "Projects". While Project > was my initial name for this feature I think collection may be a > better option. For the sake of this text both options work just fine. > The idea is the same. --=20 Tim Cross