Re: Is Org really so simple?

[Jean Louis <bugs@gnu.support>]

* Ihor Radchenko <yantar92@gmail.com> [2020-11-26 06:09]:
>  Texas Cyberthal <texas.cyberthal@gmail.com>:
> > By philosophy, I mean the dev consensus on the correct way to do
> > things, and coded configuration and usability biases.
> 
> According to my experience with org-mode development (I am not talking
> about third-party packages here), it is discouraged to change org-mode
> towards hiding metadata in org files or store *unique* data (that cannot
> be derived from the contents of the org files) related to org-mode
> externally (not in org files). It is not official statement, but rather
> my impression so far.

Personally, and from the idea of having structured referencable
information I have derived my choice to move small pieces of
information to database management as that way I can keep unique IDs
without thinking and without having things sparse on the computer. I
am working on a meta level, editing pieces of information that are yet
to become Org file.

We use Org files for indexing and great chunk of code has been written
to help users index meta data, agenda does that for example. It tries
to make a main index. But it does not cope well with complex user
cases.

In my opinion Org could be more useful would there be more
improvements to collection of meta data from all of the system and
making it more available or exposed and accessible to
users. Multi-occur option in Agenda is nice and serves the same
purpose more or less and other options.

To explain "more exposed", that would mean that there could be easier
to access listing options that user can access the index in more
liberated way. The dialogue for agenda is somehow rigid, it gives
options to do this or that, one has to search or enter queries. But
that is where it stops.

When I say more liberated way, that is analogous to SQL query
language and I do not say users should learn something complicated.

Rather the query that is made by the user should be remembered as such
and be made available as a command, macro or similar that can be saved
for later.

Example is when I search for TODO entries with special TODO kwd, then
I may search for 3-4 keywords or 7-10 keywords during one
day. Then there would be repetition to search again for those
keywords. There is history value access in the minibuffer but that
does not remember searches for the next Emacs session. Having
rememberable customized searches for users would be useful.

Other issue with the Agenda window is that I cannot move from the
minibuffer to other window, as if I wish to enter TODO I may wish in
the same time to open some fiels and look for references while in the
same time I am working with the Agenda options. It is hard for me to
understand why it was made that way, could it be because of read-char
approach? 

> > And people try to do exactly that, people are developing Org in the
> > manner to relate objects in Org file to anything else. And they do
> > hard work as they do it manually. Relational database speeds up and
> > does not tell user to manually hyperlink various relations.
> 
> Could you provide some more examples? I do not see how relational
> database is different from creating hyperlinks in org. Either way, the
> user needs to file an object/headline somewhere into org file, which is
> inherently manual.

That is exactly how I am working personally and the software will be
available when polished. It is based on same principles of having
everything hyperlinked, stored, accessible, indexable, it is dynamic
knowledge repository which I use as for meta Org-like editing of
items. It can transform everything or just one subtree to Org files on
the fly which can be saved or delegated to others.

The explanation itself is in relational databases concept. We are
using Org mode and we already try doing, we are trying hard to make
relations stick together. Compared to SQL databases I see that as hard
work, effort to patch the unpatchable.

Maybe this URL could serve as good example to explain concept of
relational databases in simple manner:
https://www.dummies.com/programming/sql/knowing-just-enough-about-relational-databases/

Or more serious one from IBM (which probably does not explain concepts
well):
https://www.ibm.com/cloud/learn/relational-databases

If table in the database is related (defined or internally connected
by its meaning and values) to other tables, then user is spared of
writing repetitive tasks.

Practical example would be:

- if text file is internally related to Joe Doe, then by clicking on
  the text file such as Org file, I could automatically get various
  hyperlinks to anything related to Joe Doe: Joe's emails list, Joe's
  SMS list, Joe's contact information, I could click and initiate a
  call with my mobile phone and just write notes without thinking on a
  phone number, I could click in Org file and send SMS to Joe that
  will be saved in computer without thinking on Joe's phone number, I
  could see relatives of Joe and find his sister and again have all
  the hyperlinks and relations to various other pieces of information
  related to Joe.

- and if I am in an Org file that has relation to other objects I
  would not need to construct hyperlinks by hand, they would be or
  could be constructed semi-automatically because the relation is
  already known.

Quoting me:

> > And people try to do exactly that, people are developing Org in the
> > manner to relate objects in Org file to anything else.

One example of way too much work is making unique IDs. Either user
decides to make it by invoking M-x, or saves the file and uses hook,
or does not invoke it. So it is in the Org but subset of users does
need Org ID numbers.

When I work with the database and wish to edit something equivalent to
a headline than the unique ID is generated automatically and I do not
need to think of it. It is made by database design.

Excerpt from hlinks table on my side:

CREATE TABLE hlinks (
hlinks_id SERIAL NOT NULL PRIMARY KEY,
....

Instead of customizing Emacs to insert IDs, customizing Org,
customizing specific files, going over specific files to find which
has unique ID which one does not have, having duplicates all over the
place, having pretty ugly and not meaningfull unique ID always shown
on screen and more or less to a degree disturbing, instead of having
them easily editable and changeable and error prone, instead of many
actions related to that unique ID, I have the above one line that does
it for me as database does the work. I need not think any more and
since I created that table I did not think until today about the
unique IDs because they are very reliable, stable, forever, database
is thinking for me, computer is taking job from me, and I am not
working for my computer.

Now if unique ID is related to one of following other tables of
subjects, objects:

- last USER who ever modified the heading, note, task, URL, anything
  related to that ID, I can find that last user. Hyperlink could be
  automatically generated button that points to on the fly generated
  Org buffer showing me such modification.

- I can find ANY USER who ever modified anything related to that
  unique ID and I can find specific time of modification in past, with
  hyperlink that I do not make by hand, that would show me all users
  and from which list I could hyperlink to the first example above.

- There is DATE in the hlinks table, so it is possible to have date as
  hyperlink button from where I can click to find other same date
  related entries, or I could invoke list to find nearby dates.

I do not create hyperlinks more than once in a function:

  (defun hyperscope-insert-buttons (tags)
  (let ((tags (split-string tags)))
    (insert "   Tags: ")
    (dolist (tag tags)
      (hyperscope-insert-button tag)
      (insert " "))))

  (defun hyperscope-insert-button (tag)
  (insert-text-button tag
		      'action
		      `(lambda (b) (hyperscope nil nil nil ,tag))
		      'mouse-action
		      `(lambda (b) (hyperscope nil nil nil ,tag))))

  That is my button. I never create those hyperlinks again. This
  relates only to the tags, but hyperlinks can be for any other type
  of relation.

  If the tag is "Dewi" the hyperlink is automatically shown in the
  header of the entry. It allows me to quickly see all entries for
  "Dewi" tag and come back to it.

- the heading of meta Org can have a status defined by user anyhow and
  I do not mean TODO/DONE but could be anything that changes later
  actions of the user, or automatically generates hyperlinks.

- possible EXPIRATION entry is there, it could be automatically
  hyperlinked to other expired items

- CURATOR entry is there, which is multi user based. User writing in
  the database is not necessarily curator of the entry. All items to
  curator's lists can be shown by hyperlinking automatically.

- HYPERLINK TYPE could be Message-ID to open up specific email
  message, then I could get a hyperlink to open all Message-IDs by
  that attribute

- MIME TYPE is there as if it is text/org then database knows it will
  invoke for me the Org mode. There will be no need for local
  variables, although it could be implemented. If there are various
  mime types the hyperlink can be there for user to list them all.

- HYPERLINK NAME is hyperlinked itself and can be hyperlinked and
  shown from one section to other just as Org mode. Org mode in fact
  can be used to display those pieces of information.

  Discussion here is only there to give ideas to me and others to
  lessen repetitive actions and help us in creating more useful
  features.

- HYPERLINK itself is where user goes. On my side Hyperlinks are
  centralized and this is very handy as I do not repeat myself to edit
  and change numerous files when creating hyperlinks in such
  files. Person does not need a PostgreSQL database to create these
  kind of universal hyperlinks, it just needs to know one centralized
  file. It can be defcustom option in Emacs. But it could be editable
  list. Or it could be simple text file with unique IDs at front and
  description.

  Example of such file would be:

  1 GNU Website, free software for you @@@ https://www.gnu.org
  2 A new skimmer uses WebSockets and a fake credit card @@@ https://blogs.akamai.com/2020/11/a-new-skimmer-uses-websockets-and-a-fake-credit-card-form-to-steal-sensitive-data.html

It is trivial to have a function that is called my-link which would
then parse the file, find the ID number at front and have the user
jump to the link. User could be offered file with links to "store"
link or transfer straight to Org file.

Org file would have the link something like:

[[(my-link 1)][GNU Website, free software for you]]

If the underlying hyperlink changes for some reason such hyperlink
need not be changed in Org files.

What if hyperlink points to specific PDF file on the system? Why
should I hard code in the sparse disconnected Org file a hyperlink to
specific file on the file system? Maybe this file changes its
position, then my hyperlink in Org file will not work, what if same
hyperlink has been used 10, 20 or 50 times in other Org files?

When using centralized approach for hyperlinks you forget about
editing the Org files, you just edit the centralized list of such
hyperlinks. That is what I am doing now.

Yesterday I have implemented new feature based on what we were
speaking about filing files and I have now made redundant long time
filing system. Database has sets, set is like a parent of a subtree or
subtree name. I have connected sets by their unique IDs to the
specific directory on the file system. Directory is based on the ID
number. Maybe you noticed that I file files related to specific person
to their directory on the hard disk, but I never think of the
directory position. In Dired I am invoking s-f and I am asked for
person's name, I choose the person and files are filed there by
year/month/date or otherwise or I can choose the date.

Then we discussed how some people access files by using searches or
indexed databases, searching by terms, queries, tags, and there is
also way to access files by navigating file system what most people
do. And then I mentioned conceptual access. This last one accommodates
better my mind and I was already used to file files by people,
organizations and subject.

From there now I just think of a file and file it under specific
subject.

If I think of specific subject, I can access the file. If I think of
related subjects I can still find somehow the file.

For example I think of LISP something, nothing specific, press key,
type LISP maybe there are 63 entries, but one only is upper case I am
on the LISP entry displayed in Emacs within a second. This entry is a
set or parent of subtree of other sets. I could now quickly jump into
the LISP directory:

The entry (ID numbers need not be shown):

    35147 LISP                                                                   Set →

Now I could jump from that entry into Dired:

  /home/data1/protected/hyperscope/3/5/1/4/7:
  
And the directory ID remains always the same /3/5/1/4/7 regardless if
I modify somewhat the entry. And if I enter the Set or subtree related
to LISP then I could get other entries such as CLISP, Emacs Lisp,
Newlisp, Picolisp and so on.

If I file something in those directories, I could invoke indexing
option to curate some of those sub-directories within
/home/data1/protected/hyperscope/3/5/1/4/7

While this looks complicate when reading, practically it is easier to
file files under specific subjects or categories this way, as this is
the workflow:

- open Dired
- think where you wish to file
- press key to choose what you thought like "LISP"
- press enter

No need to think of the directory as directory has been choosen
automatically.

Next time I need to jump to directory:

- think where to jump
- press key to find the meaning, and ENTER

Back to hyperlinks in Org file. If I am writing an entry in meta
Org-like manner then hyperlinks are generated automatically that go to
related files, that goes to related other subjects.

> Could you provide some more examples? I do not see how relational
> database is different from creating hyperlinks in org.

I am already using this system but is not easy to explain. Meta data
on my side is not hidden but is also not shown. I can edit it as easy
as Org probably easier.

c a - I am editing author's name in free text. If author is in the
      database I would choose author. Author's name can be hyperlinked to
      find other entries under same author. No need to type it by hand.

T - I am editing tags freely, I can enter: emacs lisp todo review; and
    those words would become tags. After ENTER I can immediately see
    hyperlinks to those tags.

Then if I am to edit some special status or type of the node I get
also automatic hyperlinks.

>  Either way, the
> user needs to file an object/headline somewhere into org file, which is
> inherently manual.

Yes, and no. Yes because it is true, and no because Org has been
already developed to help the user and useful enhancements or not
limited.

Concepts I have explained above need not be used with a database.
Concept is of relations and that user would not need to repeat many
actions if concepts are pulled together and integrated. 

Concept of using centralized list would alone spare so much typing for
the user.

How it would be if some headlines are repetitive? Maybe centralized
list of headlines to be choosen from and quickly inserted would help
speed up the process. Deadline would maybe remain the same or other
properties. No need for tedious searching for headling, duplicate, or
copy and yank the headline again. Choose from the central list and
build on that.

Headlines are related to things like people, company, organizations,
and those descriptions of companies, people, etc. majority of Org
users probably keep in the Org file itself. I just think it could be
so. But it is bunch of related information, including files.

If headline is related to central list of IDs that relate to something
then user could automatically invoke or get simply visualized
hyperlinks, or by click decide to insert such.

Central list would have entries like these:

1 Joe Doe @@ ~/files/JoeDoe @@ +1 234 567-8900 @@ joedoe@example.com @@ etc.

Then simple function can construct various hyperlinks for the headline
that has specific property.

Let us say I wish to insert new headline related to Joe Doe in such
imaginary enhancement:

- press key
- choose Joe Doe by maybe completion functions (this prepares property)
- headline is prepared and I just write it while property remains there

Prepared headling could look like this with X being my cursor:

* X                                 :CID-1:

Now I write the headline:

* Joe to purchase ticket in Kahama and travel on Saturday to Dar es Salaam  :CID-1:

If the task is assigned to Communication Officer or senior who has to
advise Joe how to do it, then on the next key press I could get
various hyperlinks because there is CID-1 that relates to Joe Doe:

* Joe to purchase ticket in Kahama and travel on Saturday to Dar es Salaam  :CID-1:

  [Joe's files] [Initiate call] [SMS to Joe] [Send this task to Joe] [Plain email to Joe]

Or the heading can automatically become following

* Joe to purchase ticket in Kahama and travel on Saturday to Dar es Salaam  :CID-1:

-  Joe Doe phone number: +1 234 567-890
-       Joe Doe's files: ~/files/JoeDoe
-       Joe Doe's email: joedoe@example.com
-        Family members: [Sister] [Brother] [Father]
- Other Joe Doe's tasks: [All tasks] [TODO Tasks]
-     People introduced: [173 people introduced]
-              Location: [Show on the map]
-          Transactions: [Show table of transactions]

Or I could just enter some of those hyperlinks specifically. But maybe
when I am choosing heading with the relation CID-1 property I do wish
to have some of those links, or I could prepare the template or
customize how those hyperlinks would look like.

> > I see hard work by many people who try to enhance Org as hierarchical
> > knowledge of data because people want to have feature X, but group of
> > those enhancements in reality belong to relational databases and not
> > to text files.
> 
> It is an interesting point. I would be happy if some existing tools
> could be reused instead of re-inventing the wheel for org. Do you have
> concrete examples where it can be useful? If you have, I encourage you
> to bring up a feature request to discuss this possibility with org-mode
> devs.

By principle concepts and relational automation does belong into core
of the Org mode as Org as of today does follow principles of Doug
Engelbart.

Doug Engelbart Institute - Boosting mankind's capability for coping with complex, urgent problems
https://www.dougengelbart.org/

Draft OHS-Project Plan
https://web.archive.org/web/20070704185333/http://www.bootstrap.org/augdocs/bi-2120.html

TECHNOLOGY TEMPLATE PROJECT OHS Framework 
https://www.dougengelbart.org/content/view/110/460/

About Dynamic Knowledge Repositories (DKR)
https://www.dougengelbart.org/content/view/190/163/

Or maybe you wish to see the Mother of all Demos.

Highlights of the 1968 "Mother of All Demos"
https://www.dougengelbart.org/content/view/276/000/

As there you will see the outline mode hyperlinked to everything being
hyperlinked from anything else, which principles are partially
implemented on the same website, you can see how finely grained one
can reference a paragraph on that website.

Maybe in beginning the Org mode was just enhancement of the Outline
mode which I find great by the way for its simplicity. But today Org
mode has evolved into sparse or disconnected way of doing
things. Developers try their best. In my opinion from somebody who
uses relational tables I do not find that approach nice, I find it
very tedious and time spending activity.

I would like to see Org file being finely grained so that each item of
the list can be referenced, each paragraph to be uniquely referenced
or sets of paragraphs. There is a package that I guess, converts Org
file to OPML which is an acronym for Outline Processor Markup
Language, as Org is outline processor that tries doing it in plain
text. It misses all the meta data so tendencies by people go back to
more structured format.

Because specific paragraph referencing is now not easily possible I am
using database approach of editing chunks of the Org file where then
any piece of Org can become referential.

Maybe the fundamental feature in its basics already exists if the
whole Org file could be exported as LISP data structure. Then user
could say "chunk paragraphs" to get references to single
paragraphs. Or feature could run over the Org file and create unique
IDs or anchors for each paragraph which could then be referenced in
their exports. If Org is exported to HTML then referencing from other
documents to specific paragraph becomes possible. While this is my
wish, it is not necessarily wish for Org.

GNU Hyperbole
https://www.gnu.org/s/hyperbole

The GNU Hyperbole Koutline mode predates Org mode and allows finely
grained referencing. But it is not really plain text. For example
there are no headings or paragraphs as everything is a structured
node. This may not be so handy. Mixture of Koutliner and Org mode is
what I like to build myself. If I work on meta Org-kind-of level then
I am editing nodes and for those nodes I am deciding what they are,
maybe node is a Set or Heading, maybe it is WWW hyperlink, maybe it is
note, maybe single paragraph to be processed as such and formatted as
such in various outputs, maybe it is set of paragraphs, and maybe it
is one chunk of Org file or any other type of file, could be even
image, video, anything. Then when there are chunks defined I can
structure the output or move nodes up and down with easy.

To create Org file from meta Org-like editing is speedy and not
noticable. If one set of chunks of such Org file have been used in
other set or vice versa, I can use reuse those nodes without
duplicating the node. If specific table of transaction is referenced
from multiple files users of Org will want to include it in those
files. That is also what minimizes replication, I do not need to
duplicate the body or meta data or contents of elements but I just
need to say what is the series by semi-visual ordering of those
elements by their priority.

Heading A can be one node, edited by user one time. It need not be
duplicated for other file, rather only a pointer to Heading A can be
included and the Heading then appears in a new Org file.

This type of link here is fine for plain text, really makes sense:

** TODO Some heading here

It just does not scale and does not have relations. As there are no
automatically designated relations then I have to assign it myself:

- to which person this task is assigned? Normally I send tasks to
  person assigned

- which person is related to this task? Do I really need to write the
  name? Then I have no reference or relation back to the person. I
  would like to have but do not. My tasks are "For staff Joe to go to
  location ABC and negotiate with company XYZ".

  If task is related to company which can be just on second to relate
  it, and assigned to staff Joe, which is another second or few to
  find Jow, and location ABC which is another few seconds and related
  to company XYZ which is another few seconds then all what I do are
  selections. No hyperlink writing on the low level.

  The task that is assigned to Joe, then can be sent to Joe with all
  the relevant and related information such as geographic location of
  the location ABC, license number related, region or district where
  it is located, then all contact information related to Joe and
  contact information related to company XYZ, including people names
  related to company XYZ and their contacts. My time is saved. Joe has
  got full information and can fullfil the task. This may include the
  scheduled date and deadlines as well and it could also include list
  of related tasks, without me thinking. 

To say TODO, PENDING, add property this or that by hand, is repetition
that I am not inclined any more to perform, so I am transitioning
everything into meta level. Using few keys I can export it to Org and
use further Org facilities to export into PDF or ASCII, Unicode or
similar.

Org started as enhancement of the outline-mode with focus on plain
text. But it is not any more plain text, it is full of properties that
may not be readable to people who did not write them or who are not
used to it. Properties can also confuse the person and performance of
task can be ruined. So it develops more and more into activity that
does need relational database. If such relational database is
implemented in the Emacs EIEIO or maybe as more simpler LISP data or
various lists of hashes saved in files, or is maybe PostgreSQL or
maybe GDBM or MySQL database, NoSQL, that really does not matter.

What matters is that headings are related to various lists of other
things and users do not have relational facilities. They write them
tedious by hand and developers try to help by hacking something that
has been already invented. You said let us not reinvent the wheel but
Org mode is wheel reinvented regardless if it did not use Engelbart's
principles it became quite similar to Augment as outliner software.

Doug Engelbart Institute - About NLS / Augment - 1968
https://www.dougengelbart.org/content/view/155/87/