From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mp2 ([2001:41d0:2:4a6f::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by ms11 with LMTPS id D/2/ALhuv1/oPwAA0tVLHw (envelope-from ) for ; Thu, 26 Nov 2020 09:00:40 +0000 Received: from aspmx1.migadu.com ([2001:41d0:2:4a6f::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by mp2 with LMTPS id KOezN7duv18XFgAAB5/wlQ (envelope-from ) for ; Thu, 26 Nov 2020 09:00:39 +0000 Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by aspmx1.migadu.com (Postfix) with ESMTPS id 1D02F940253 for ; Thu, 26 Nov 2020 09:00:39 +0000 (UTC) Received: from localhost ([::1]:42686 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1kiD8o-0003ix-2x for larch@yhetil.org; Thu, 26 Nov 2020 04:00:38 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]:48280) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1kiD88-0003eZ-95 for emacs-orgmode@gnu.org; Thu, 26 Nov 2020 03:59:56 -0500 Received: from static.rcdrun.com ([95.85.24.50]:40263) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1kiD84-0002RP-Kn for emacs-orgmode@gnu.org; Thu, 26 Nov 2020 03:59:56 -0500 Received: from localhost ([::ffff:41.202.241.56]) (AUTH: PLAIN admin, TLS: TLS1.2,256bits,ECDHE_RSA_AES_256_GCM_SHA384) by static.rcdrun.com with ESMTPSA id 00000000002C0007.000000005FBF6E87.00002A57; Thu, 26 Nov 2020 08:59:50 +0000 Date: Thu, 26 Nov 2020 11:57:51 +0300 From: Jean Louis To: Ihor Radchenko Subject: Re: Is Org really so simple? Message-ID: References: <878sauhhv1.fsf@web.de> <875z5ygwwr.fsf@web.de> <87r1olfvh4.fsf@web.de> <878sascg5j.fsf@localhost> <877dq8bysg.fsf@localhost> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Disposition: inline Content-Transfer-Encoding: 8bit In-Reply-To: <877dq8bysg.fsf@localhost> User-Agent: Mutt/2.0 (3d08634) (2020-11-07) Received-SPF: pass client-ip=95.85.24.50; envelope-from=bugs@gnu.support; helo=static.rcdrun.com X-Spam_score_int: -18 X-Spam_score: -1.9 X-Spam_bar: - X-Spam_report: (-1.9 / 5.0 requ) BAYES_00=-1.9, SPF_HELO_PASS=-0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: emacs-orgmode@gnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: "General discussions about Org-mode." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: "Dr. Arne Babenhauserheide" , Texas Cyberthal , "emacs-orgmode@gnu.org" Errors-To: emacs-orgmode-bounces+larch=yhetil.org@gnu.org Sender: "Emacs-orgmode" X-Migadu-Flow: inc X-Scanner: ns3122888.ip-94-23-21.eu Authentication-Results: aspmx1.migadu.com; dkim=none; dmarc=none; spf=pass (aspmx1.migadu.com: domain of emacs-orgmode-bounces@gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=emacs-orgmode-bounces@gnu.org X-Spam-Score: 1.49 X-TUID: PdagKJdKX5uS * Ihor Radchenko [2020-11-26 06:09]: > Texas Cyberthal : > > By philosophy, I mean the dev consensus on the correct way to do > > things, and coded configuration and usability biases. > > According to my experience with org-mode development (I am not talking > about third-party packages here), it is discouraged to change org-mode > towards hiding metadata in org files or store *unique* data (that cannot > be derived from the contents of the org files) related to org-mode > externally (not in org files). It is not official statement, but rather > my impression so far. Personally, and from the idea of having structured referencable information I have derived my choice to move small pieces of information to database management as that way I can keep unique IDs without thinking and without having things sparse on the computer. I am working on a meta level, editing pieces of information that are yet to become Org file. We use Org files for indexing and great chunk of code has been written to help users index meta data, agenda does that for example. It tries to make a main index. But it does not cope well with complex user cases. In my opinion Org could be more useful would there be more improvements to collection of meta data from all of the system and making it more available or exposed and accessible to users. Multi-occur option in Agenda is nice and serves the same purpose more or less and other options. To explain "more exposed", that would mean that there could be easier to access listing options that user can access the index in more liberated way. The dialogue for agenda is somehow rigid, it gives options to do this or that, one has to search or enter queries. But that is where it stops. When I say more liberated way, that is analogous to SQL query language and I do not say users should learn something complicated. Rather the query that is made by the user should be remembered as such and be made available as a command, macro or similar that can be saved for later. Example is when I search for TODO entries with special TODO kwd, then I may search for 3-4 keywords or 7-10 keywords during one day. Then there would be repetition to search again for those keywords. There is history value access in the minibuffer but that does not remember searches for the next Emacs session. Having rememberable customized searches for users would be useful. Other issue with the Agenda window is that I cannot move from the minibuffer to other window, as if I wish to enter TODO I may wish in the same time to open some fiels and look for references while in the same time I am working with the Agenda options. It is hard for me to understand why it was made that way, could it be because of read-char approach? > > And people try to do exactly that, people are developing Org in the > > manner to relate objects in Org file to anything else. And they do > > hard work as they do it manually. Relational database speeds up and > > does not tell user to manually hyperlink various relations. > > Could you provide some more examples? I do not see how relational > database is different from creating hyperlinks in org. Either way, the > user needs to file an object/headline somewhere into org file, which is > inherently manual. That is exactly how I am working personally and the software will be available when polished. It is based on same principles of having everything hyperlinked, stored, accessible, indexable, it is dynamic knowledge repository which I use as for meta Org-like editing of items. It can transform everything or just one subtree to Org files on the fly which can be saved or delegated to others. The explanation itself is in relational databases concept. We are using Org mode and we already try doing, we are trying hard to make relations stick together. Compared to SQL databases I see that as hard work, effort to patch the unpatchable. Maybe this URL could serve as good example to explain concept of relational databases in simple manner: https://www.dummies.com/programming/sql/knowing-just-enough-about-relational-databases/ Or more serious one from IBM (which probably does not explain concepts well): https://www.ibm.com/cloud/learn/relational-databases If table in the database is related (defined or internally connected by its meaning and values) to other tables, then user is spared of writing repetitive tasks. Practical example would be: - if text file is internally related to Joe Doe, then by clicking on the text file such as Org file, I could automatically get various hyperlinks to anything related to Joe Doe: Joe's emails list, Joe's SMS list, Joe's contact information, I could click and initiate a call with my mobile phone and just write notes without thinking on a phone number, I could click in Org file and send SMS to Joe that will be saved in computer without thinking on Joe's phone number, I could see relatives of Joe and find his sister and again have all the hyperlinks and relations to various other pieces of information related to Joe. - and if I am in an Org file that has relation to other objects I would not need to construct hyperlinks by hand, they would be or could be constructed semi-automatically because the relation is already known. Quoting me: > > And people try to do exactly that, people are developing Org in the > > manner to relate objects in Org file to anything else. One example of way too much work is making unique IDs. Either user decides to make it by invoking M-x, or saves the file and uses hook, or does not invoke it. So it is in the Org but subset of users does need Org ID numbers. When I work with the database and wish to edit something equivalent to a headline than the unique ID is generated automatically and I do not need to think of it. It is made by database design. Excerpt from hlinks table on my side: CREATE TABLE hlinks ( hlinks_id SERIAL NOT NULL PRIMARY KEY, .... Instead of customizing Emacs to insert IDs, customizing Org, customizing specific files, going over specific files to find which has unique ID which one does not have, having duplicates all over the place, having pretty ugly and not meaningfull unique ID always shown on screen and more or less to a degree disturbing, instead of having them easily editable and changeable and error prone, instead of many actions related to that unique ID, I have the above one line that does it for me as database does the work. I need not think any more and since I created that table I did not think until today about the unique IDs because they are very reliable, stable, forever, database is thinking for me, computer is taking job from me, and I am not working for my computer. Now if unique ID is related to one of following other tables of subjects, objects: - last USER who ever modified the heading, note, task, URL, anything related to that ID, I can find that last user. Hyperlink could be automatically generated button that points to on the fly generated Org buffer showing me such modification. - I can find ANY USER who ever modified anything related to that unique ID and I can find specific time of modification in past, with hyperlink that I do not make by hand, that would show me all users and from which list I could hyperlink to the first example above. - There is DATE in the hlinks table, so it is possible to have date as hyperlink button from where I can click to find other same date related entries, or I could invoke list to find nearby dates. I do not create hyperlinks more than once in a function: (defun hyperscope-insert-buttons (tags) (let ((tags (split-string tags))) (insert " Tags: ") (dolist (tag tags) (hyperscope-insert-button tag) (insert " ")))) (defun hyperscope-insert-button (tag) (insert-text-button tag 'action `(lambda (b) (hyperscope nil nil nil ,tag)) 'mouse-action `(lambda (b) (hyperscope nil nil nil ,tag)))) That is my button. I never create those hyperlinks again. This relates only to the tags, but hyperlinks can be for any other type of relation. If the tag is "Dewi" the hyperlink is automatically shown in the header of the entry. It allows me to quickly see all entries for "Dewi" tag and come back to it. - the heading of meta Org can have a status defined by user anyhow and I do not mean TODO/DONE but could be anything that changes later actions of the user, or automatically generates hyperlinks. - possible EXPIRATION entry is there, it could be automatically hyperlinked to other expired items - CURATOR entry is there, which is multi user based. User writing in the database is not necessarily curator of the entry. All items to curator's lists can be shown by hyperlinking automatically. - HYPERLINK TYPE could be Message-ID to open up specific email message, then I could get a hyperlink to open all Message-IDs by that attribute - MIME TYPE is there as if it is text/org then database knows it will invoke for me the Org mode. There will be no need for local variables, although it could be implemented. If there are various mime types the hyperlink can be there for user to list them all. - HYPERLINK NAME is hyperlinked itself and can be hyperlinked and shown from one section to other just as Org mode. Org mode in fact can be used to display those pieces of information. Discussion here is only there to give ideas to me and others to lessen repetitive actions and help us in creating more useful features. - HYPERLINK itself is where user goes. On my side Hyperlinks are centralized and this is very handy as I do not repeat myself to edit and change numerous files when creating hyperlinks in such files. Person does not need a PostgreSQL database to create these kind of universal hyperlinks, it just needs to know one centralized file. It can be defcustom option in Emacs. But it could be editable list. Or it could be simple text file with unique IDs at front and description. Example of such file would be: 1 GNU Website, free software for you @@@ https://www.gnu.org 2 A new skimmer uses WebSockets and a fake credit card @@@ https://blogs.akamai.com/2020/11/a-new-skimmer-uses-websockets-and-a-fake-credit-card-form-to-steal-sensitive-data.html It is trivial to have a function that is called my-link which would then parse the file, find the ID number at front and have the user jump to the link. User could be offered file with links to "store" link or transfer straight to Org file. Org file would have the link something like: [[(my-link 1)][GNU Website, free software for you]] If the underlying hyperlink changes for some reason such hyperlink need not be changed in Org files. What if hyperlink points to specific PDF file on the system? Why should I hard code in the sparse disconnected Org file a hyperlink to specific file on the file system? Maybe this file changes its position, then my hyperlink in Org file will not work, what if same hyperlink has been used 10, 20 or 50 times in other Org files? When using centralized approach for hyperlinks you forget about editing the Org files, you just edit the centralized list of such hyperlinks. That is what I am doing now. Yesterday I have implemented new feature based on what we were speaking about filing files and I have now made redundant long time filing system. Database has sets, set is like a parent of a subtree or subtree name. I have connected sets by their unique IDs to the specific directory on the file system. Directory is based on the ID number. Maybe you noticed that I file files related to specific person to their directory on the hard disk, but I never think of the directory position. In Dired I am invoking s-f and I am asked for person's name, I choose the person and files are filed there by year/month/date or otherwise or I can choose the date. Then we discussed how some people access files by using searches or indexed databases, searching by terms, queries, tags, and there is also way to access files by navigating file system what most people do. And then I mentioned conceptual access. This last one accommodates better my mind and I was already used to file files by people, organizations and subject. >From there now I just think of a file and file it under specific subject. If I think of specific subject, I can access the file. If I think of related subjects I can still find somehow the file. For example I think of LISP something, nothing specific, press key, type LISP maybe there are 63 entries, but one only is upper case I am on the LISP entry displayed in Emacs within a second. This entry is a set or parent of subtree of other sets. I could now quickly jump into the LISP directory: The entry (ID numbers need not be shown): 35147 LISP Set → Now I could jump from that entry into Dired: /home/data1/protected/hyperscope/3/5/1/4/7: And the directory ID remains always the same /3/5/1/4/7 regardless if I modify somewhat the entry. And if I enter the Set or subtree related to LISP then I could get other entries such as CLISP, Emacs Lisp, Newlisp, Picolisp and so on. If I file something in those directories, I could invoke indexing option to curate some of those sub-directories within /home/data1/protected/hyperscope/3/5/1/4/7 While this looks complicate when reading, practically it is easier to file files under specific subjects or categories this way, as this is the workflow: - open Dired - think where you wish to file - press key to choose what you thought like "LISP" - press enter No need to think of the directory as directory has been choosen automatically. Next time I need to jump to directory: - think where to jump - press key to find the meaning, and ENTER Back to hyperlinks in Org file. If I am writing an entry in meta Org-like manner then hyperlinks are generated automatically that go to related files, that goes to related other subjects. > Could you provide some more examples? I do not see how relational > database is different from creating hyperlinks in org. I am already using this system but is not easy to explain. Meta data on my side is not hidden but is also not shown. I can edit it as easy as Org probably easier. c a - I am editing author's name in free text. If author is in the database I would choose author. Author's name can be hyperlinked to find other entries under same author. No need to type it by hand. T - I am editing tags freely, I can enter: emacs lisp todo review; and those words would become tags. After ENTER I can immediately see hyperlinks to those tags. Then if I am to edit some special status or type of the node I get also automatic hyperlinks. > Either way, the > user needs to file an object/headline somewhere into org file, which is > inherently manual. Yes, and no. Yes because it is true, and no because Org has been already developed to help the user and useful enhancements or not limited. Concepts I have explained above need not be used with a database. Concept is of relations and that user would not need to repeat many actions if concepts are pulled together and integrated. Concept of using centralized list would alone spare so much typing for the user. How it would be if some headlines are repetitive? Maybe centralized list of headlines to be choosen from and quickly inserted would help speed up the process. Deadline would maybe remain the same or other properties. No need for tedious searching for headling, duplicate, or copy and yank the headline again. Choose from the central list and build on that. Headlines are related to things like people, company, organizations, and those descriptions of companies, people, etc. majority of Org users probably keep in the Org file itself. I just think it could be so. But it is bunch of related information, including files. If headline is related to central list of IDs that relate to something then user could automatically invoke or get simply visualized hyperlinks, or by click decide to insert such. Central list would have entries like these: 1 Joe Doe @@ ~/files/JoeDoe @@ +1 234 567-8900 @@ joedoe@example.com @@ etc. Then simple function can construct various hyperlinks for the headline that has specific property. Let us say I wish to insert new headline related to Joe Doe in such imaginary enhancement: - press key - choose Joe Doe by maybe completion functions (this prepares property) - headline is prepared and I just write it while property remains there Prepared headling could look like this with X being my cursor: * X :CID-1: Now I write the headline: * Joe to purchase ticket in Kahama and travel on Saturday to Dar es Salaam :CID-1: If the task is assigned to Communication Officer or senior who has to advise Joe how to do it, then on the next key press I could get various hyperlinks because there is CID-1 that relates to Joe Doe: * Joe to purchase ticket in Kahama and travel on Saturday to Dar es Salaam :CID-1: [Joe's files] [Initiate call] [SMS to Joe] [Send this task to Joe] [Plain email to Joe] Or the heading can automatically become following * Joe to purchase ticket in Kahama and travel on Saturday to Dar es Salaam :CID-1: - Joe Doe phone number: +1 234 567-890 - Joe Doe's files: ~/files/JoeDoe - Joe Doe's email: joedoe@example.com - Family members: [Sister] [Brother] [Father] - Other Joe Doe's tasks: [All tasks] [TODO Tasks] - People introduced: [173 people introduced] - Location: [Show on the map] - Transactions: [Show table of transactions] Or I could just enter some of those hyperlinks specifically. But maybe when I am choosing heading with the relation CID-1 property I do wish to have some of those links, or I could prepare the template or customize how those hyperlinks would look like. > > I see hard work by many people who try to enhance Org as hierarchical > > knowledge of data because people want to have feature X, but group of > > those enhancements in reality belong to relational databases and not > > to text files. > > It is an interesting point. I would be happy if some existing tools > could be reused instead of re-inventing the wheel for org. Do you have > concrete examples where it can be useful? If you have, I encourage you > to bring up a feature request to discuss this possibility with org-mode > devs. By principle concepts and relational automation does belong into core of the Org mode as Org as of today does follow principles of Doug Engelbart. Doug Engelbart Institute - Boosting mankind's capability for coping with complex, urgent problems https://www.dougengelbart.org/ Draft OHS-Project Plan https://web.archive.org/web/20070704185333/http://www.bootstrap.org/augdocs/bi-2120.html TECHNOLOGY TEMPLATE PROJECT OHS Framework https://www.dougengelbart.org/content/view/110/460/ About Dynamic Knowledge Repositories (DKR) https://www.dougengelbart.org/content/view/190/163/ Or maybe you wish to see the Mother of all Demos. Highlights of the 1968 "Mother of All Demos" https://www.dougengelbart.org/content/view/276/000/ As there you will see the outline mode hyperlinked to everything being hyperlinked from anything else, which principles are partially implemented on the same website, you can see how finely grained one can reference a paragraph on that website. Maybe in beginning the Org mode was just enhancement of the Outline mode which I find great by the way for its simplicity. But today Org mode has evolved into sparse or disconnected way of doing things. Developers try their best. In my opinion from somebody who uses relational tables I do not find that approach nice, I find it very tedious and time spending activity. I would like to see Org file being finely grained so that each item of the list can be referenced, each paragraph to be uniquely referenced or sets of paragraphs. There is a package that I guess, converts Org file to OPML which is an acronym for Outline Processor Markup Language, as Org is outline processor that tries doing it in plain text. It misses all the meta data so tendencies by people go back to more structured format. Because specific paragraph referencing is now not easily possible I am using database approach of editing chunks of the Org file where then any piece of Org can become referential. Maybe the fundamental feature in its basics already exists if the whole Org file could be exported as LISP data structure. Then user could say "chunk paragraphs" to get references to single paragraphs. Or feature could run over the Org file and create unique IDs or anchors for each paragraph which could then be referenced in their exports. If Org is exported to HTML then referencing from other documents to specific paragraph becomes possible. While this is my wish, it is not necessarily wish for Org. GNU Hyperbole https://www.gnu.org/s/hyperbole The GNU Hyperbole Koutline mode predates Org mode and allows finely grained referencing. But it is not really plain text. For example there are no headings or paragraphs as everything is a structured node. This may not be so handy. Mixture of Koutliner and Org mode is what I like to build myself. If I work on meta Org-kind-of level then I am editing nodes and for those nodes I am deciding what they are, maybe node is a Set or Heading, maybe it is WWW hyperlink, maybe it is note, maybe single paragraph to be processed as such and formatted as such in various outputs, maybe it is set of paragraphs, and maybe it is one chunk of Org file or any other type of file, could be even image, video, anything. Then when there are chunks defined I can structure the output or move nodes up and down with easy. To create Org file from meta Org-like editing is speedy and not noticable. If one set of chunks of such Org file have been used in other set or vice versa, I can use reuse those nodes without duplicating the node. If specific table of transaction is referenced from multiple files users of Org will want to include it in those files. That is also what minimizes replication, I do not need to duplicate the body or meta data or contents of elements but I just need to say what is the series by semi-visual ordering of those elements by their priority. Heading A can be one node, edited by user one time. It need not be duplicated for other file, rather only a pointer to Heading A can be included and the Heading then appears in a new Org file. This type of link here is fine for plain text, really makes sense: ** TODO Some heading here It just does not scale and does not have relations. As there are no automatically designated relations then I have to assign it myself: - to which person this task is assigned? Normally I send tasks to person assigned - which person is related to this task? Do I really need to write the name? Then I have no reference or relation back to the person. I would like to have but do not. My tasks are "For staff Joe to go to location ABC and negotiate with company XYZ". If task is related to company which can be just on second to relate it, and assigned to staff Joe, which is another second or few to find Jow, and location ABC which is another few seconds and related to company XYZ which is another few seconds then all what I do are selections. No hyperlink writing on the low level. The task that is assigned to Joe, then can be sent to Joe with all the relevant and related information such as geographic location of the location ABC, license number related, region or district where it is located, then all contact information related to Joe and contact information related to company XYZ, including people names related to company XYZ and their contacts. My time is saved. Joe has got full information and can fullfil the task. This may include the scheduled date and deadlines as well and it could also include list of related tasks, without me thinking. To say TODO, PENDING, add property this or that by hand, is repetition that I am not inclined any more to perform, so I am transitioning everything into meta level. Using few keys I can export it to Org and use further Org facilities to export into PDF or ASCII, Unicode or similar. Org started as enhancement of the outline-mode with focus on plain text. But it is not any more plain text, it is full of properties that may not be readable to people who did not write them or who are not used to it. Properties can also confuse the person and performance of task can be ruined. So it develops more and more into activity that does need relational database. If such relational database is implemented in the Emacs EIEIO or maybe as more simpler LISP data or various lists of hashes saved in files, or is maybe PostgreSQL or maybe GDBM or MySQL database, NoSQL, that really does not matter. What matters is that headings are related to various lists of other things and users do not have relational facilities. They write them tedious by hand and developers try to help by hacking something that has been already invented. You said let us not reinvent the wheel but Org mode is wheel reinvented regardless if it did not use Engelbart's principles it became quite similar to Augment as outliner software. Doug Engelbart Institute - About NLS / Augment - 1968 https://www.dougengelbart.org/content/view/155/87/