Re: contact management in emacs

[Jean Louis <bugs@gnu.support>]

* John Kitchin <jkitchin@andrew.cmu.edu> [2021-03-08 23:06]:
> This question is going to go a little off the original topic. I wonder
> how far you can push your model of replacing the org-file with a
> database.

Those functions which I use most often I can fully replace with the
database work. The visual stuff is handled by tabulated-list-mode, so
I cannot currently see the contents of a simple heading unless I press
a key. But I am sure that database based Org style editing is possible
in general.

What I mean is that one could have an Org file displayed but
internally all of its structures could be translated on the fly into
the database. This would allow collaboration with many users and
finely based version control of each elements of the structure.

I think that automatic update and version control of each contents of
a heading could be saved to database, or it could be saved to database
by using key or by switching from one heading to other heading.

If user switches to other heading, would that other heading be edited
by other user in the same time, that other heading could be
automatically updated.

I was making simple functions to move Org headings into the database,
and functions to move database entries into Org files on the fly.

IMHO, what Org does is either intentionally or not intentionally, or
mixed-wise related to Doug Engelbart's work. 

TECHNOLOGY TEMPLATE PROJECT OHS Framework 
https://www.dougengelbart.org/content/view/110/460/

> This question is going to go a little off the original topic. I wonder
> how far you can push your model of replacing the org-file with a
> database.

After that introduction, and I hope you understand what I wrote (as I
am not sure if I have expressed myself well enough), then I can answer
how far it may be pushed. I have defined in my software Hyperscope
that each node can have different type. All nodes have its ID
numbers. So now I could say that nodes have various finely grained
types such as:

- Subtree (This could be considered either heading or title in Org
  editing), it contains other nodes. 

- headings or subtrees within subtree are similar to sub-headings in
  Org editing

- I can define Org Babel-like nodes, for example now I have SQL
  output, I can enter SQL SELECT and get the report. That type of node
  definition is alternative to Org Babel. I did not work much on it,
  but it is not hard to define. As soon as the node type is defined,
  inside of the node type there could be defined, even inside of the
  database, on how to run or execute that code block. Right now I am
  hardcoding that.

- then I can go finely grained, I could say: this node is
  paragraph. But I need not do that.

- then I could go more finely grained, I could say this node is a list
  of items, but I could also say this node is one item within a list.

- I could say this is one line as node, or this is one sentence as
  node.

- I could say this is one word within a sentence.

That means I can defined "Elementary objects" as defined here:
https://www.dougengelbart.org/content/view/110/460/#2a1a

Objects are basic content packets of an arbitrary, user and developer
extensible nature. Types of elementary objects could contain text,
graphics, equations, tables, spreadsheets, canned-images, video,
sound, code elements, etc.

Now you can imagine that those paragraphs could be reused in various
Org files, reusing such paragraphs becomes trivial. There is no copy
and paste any more, just one time definition where some paragraph
shall be inserted:

- edit subtree of nodes
- decide where to insert some other existing node
- insert it and forget about it

This then allows editing of the subtree (Org equivalent) node A or
node B, and each time that paragraph is edited it is also changed in
the other subtree. Copying subtrees so that they do not synchronize is
naturally possible.

Finely grained definition of nodes allows for finely grained
referencing to those nodes what is currently not possible with Org
mode. I have researched that option.

Review this page: https://www.dougengelbart.org/content/view/110/460/
and you will see how each paragraph is finely referenced on that
page. Specific referencing enhances educational capabilities in
specialization on specific subject. It saves times, efforts, also a
lot of money.

Imagine group of 20 miners and their 3 administrators who are supposed
to work on a lead mine, they did finish their general education, but
now they are faced with specialized subject of "lead mining". Having a
very precise set of references to lead mining saves money, as
education of miners and administrators is paid in their salaries, if
they are reading irrelevant instructions or if they are offered whole
books to read and find references themselves, that would spend 20
times plus 3 times more money for their time to gain that special
knowledge. Having a finely grained set of references that has to be
read by those specific people specializes those people to the specific
subject of knowledge quickly and efficiently, they can get only
information that relates to galena as mineral, melting, refining,
collecting of galena and processing of it.

Fine referencing thus helps in research and education and other
subjects.

To index bodies of knowledge is not so hard with computerized
information. Once index is made, such index contains the subject, for
example "galena". By using larger index, one can then form and curate
a subtree or heading of nodes and use that heading for education to
specialize miners in a specific subject.

Indexing and converting to Org file is possible, then if one does not
work with database, one can still select those headings that relate to
specific subject with Emacs functions and then copy and paste those
specific headings into a new file.

But what if any node is changed either in the new file or the main
index? They are not automatically synchronized. Database backed
editing allows automatic synchronization. Or nodes may be made
separate by duplicating them.

It also allows instead of creating a new subtree, to create just a
result based on query and exclude/include some other lines and create
new Org file with those entries for later research. Org file can be
converted to PDF and distributed to people.

> It looks like it would be easy to extend to something like a shared
> bibliography.

I wish I could understand better what you mean with shared
bibliography. Explain me better.

As side notes to bibliography subject, in the table of nodes I have
included some basic bibliographic entries, but not all, I have not
been thinking enough of that. I do use it for finely grained personal
bibliographic collection of hyperlinks and hyperdocuments and now I
have more than 20000 items, mostly PDF files with references to
articles inside of PDF file, their titles and similar. It is like
index of indexes. Subtrees or all system can be converted to Org files
on the fly and saved if necessary, but there is much work to make it
more detailed and nice, well hyperlinked.

Table `hlinks' should be maybe called hyperdocuments, but I chose this
name for now, has the following structure:

 hlinks_id               
 hlinks_datecreated      
 hlinks_datemodified     
 hlinks_usercreated      
 hlinks_usermodified     
 hlinks_hlinkstatuses    
 hlinks_date             
 hlinks_expiration       
 hlinks_curator          
 hlinks_hlinktypes       
 hlinks_mimetypes        
 hlinks_name             
 hlinks_link             
 hlinks_arguments        
 hlinks_description      
 hlinks_text             
 hlinks_tags             
 hlinks_parent           
 hlinks_author           
 hlinks_hlinkpermissions 
 hlinks_revision         
 hlinks_numberofpages    
 hlinks_language         
 hlinks_filesize         
 hlinks_timelength       
 hlinks_width            
 hlinks_height           
 hlinks_hash             
 hlinks_signature        p
 hlinks_pages            
 hlinks_accounts         
 hlinks_people           
 hlinks_businesses       
 hlinks_opportunities    
 hlinks_priorities       
 hlinks_authorname       
 hlinks_properties       
 hlinks_elisphash        
 hlinks_publisher        
 hlinks_groups           
 hlinks_groupassigned    
 hlinks_assignedperson   
 hlinks_rank             
 hlinks_active           
 hlinks_actionstatuses   
 hlinks_globalpriority   
 hlinks_relatedfile      

The nature of the database is collaborative. Adding new users and
allowing users finely grained access is possible. Users could edit
either all tables and fields, or just some tables and fields. Those
features are handled by the database and configuration, not by users'
run program (as otherwise user could circumvent it).

Users could access the database from any distant location and edit
nodes. Output could be Org file or any other output, but using Org
file as output is beneficial as Org itself has many various other
export formats.

This type of work is not well suitable for collaborative real time
online editing. However, it is suitable for concurrent non-real time
editing of any items.

For example, user A could designate new scheduled time, but user B
could change the scheduled time to something else. All changes can be
inside of the database backed version control to see who changed what
at which time. It is suitable for collaborative business or
organization work. Marketing, sales, communications, notes, anything
is possible when collaboration takes place.

Example workflow could be:

- manager views Org file of customer Joe

- assign customer Joe to salesman Jane

- salesman Jane calls customer and makes notes on customer Joe

- manager is notified of note visualized in Org file, it seem like
  customer Joe was handled in past better by salesman Juan

- manager schedules call to customer Joe by Juan at Tuesday 10 o'clock

- Juan calls customer at 10:30 Tuesday and closes the sale, invoice is
  generated

- all parties, manager, salesmen Jane and Juan can view what has taken
  place, visualization can take place as Org file

By using the package `crdt' from
https://code.librehq.com/qhong/crdt.el.git one can involve online
real-time collaboration of a single item. This works right now with
Org mode.

Each node of the database system could be edited with `crdt-mode'
enabled if one requires real time online collaboration.

> How far could you push it for something like a shared set of documents,
> e.g. a shared notebook on a project?

Each node is considered elementary object. There are no files. But
each node can be exported into a file.

As it is database backed, it is already shared notebook. Each subtree
can be considered Org file or Mixed-Object documents as defined by
Engelbart institute.

> How feasible would it be to access tables or src blocks in these if
> they were in a database?

Just as easy:

- concurrent updates, additions and deletions are possible with the
  automated database backed version control (no thinking).

- real time online collaborative editing is possible with `crdt'
  package for single entries

- editing any entry, any node in the database can be done by any
  mode, not just Org mode. If I define the type of the node to be Org
  type, then editing switches to Org mode and I can edit the tables by
  using Org mode. When saved, table is saved in the database, but can
  be if user wish and want, be also each time saved into the file as
  full Org or single node.

- src blocks I mostly constuct from SQL select statements, so they
  contain SQL, and I have implemented node with SQL, so I just put SQL
  statement there, it provides the report just as Org babel. It is not
  hard to implement any language and get literate programming
  functions.

Similar node based editing is provided by Leo editor, also inspired by
Org and other similar systems.

Leo programmable editor
http://leoeditor.com/

> or to build an agenda from entries in the database?

It is simple to provide agenda from entries in the database then from
entries in multiple Org files.

- each elementary object can have assigned "ACTION" at my side. Each
  assignement of ACTION is recorded with date/time/username, each
  COMPLETED designation is also recorded, each removal of designation
  is also recorded. One can know in future which items were actionable
  in past and which items were completed in past or maybe completed
  and then again assigned as actionable. This is currently not a
  feature in Org file. One could know that by editing Org file with
  file system version control. Single user without file system version
  control cannot know if some headings had TODO in past and now they
  do not have it.

- each item may be or may be not assigned to specific person. All
  people are in the database. So choice of people comes from the
  database itself. Assignment to specific person is recorded with
  date/time/username and deletion of assignment is recorded,
  modification is recorded automatically. This way it is possible to
  see which task was assigned to whom.

- agenda thus may be managed centralized for many people, or by people
  decentralized. Tasks may sent to assigned person with keys like "s
  a", either one task, or all tasks at once. Reminders can be
  programmed with various communication lines, including automated
  phone calls, SMS, emails, faxes, letters or visual reminders on
  computer, or printouts for dispatch.

- that allows collaborative based agenda, it is possible to list items
  assigned to person Joe, or person Jane, Joe may insert his own
  items, Jane may insert her own items, both Joe and Jane could see
  each other items, or finely grained permissions could be defined
  (database backed). Manager who is neither Joe or Jane could see all
  the items, regardless where users are located in the world. Access
  to the database could be through Org extensions, or through software
  I am making, or through other software from various
  devices. Software is independent of the database entries.

Special table collects information such as  SCHEDULED,  DEADLINE,
CLOCK-IN, CLOCK-OUT, ACTION, COMPLETED, ACTION-REMOVED. This table
may add different other timestamps by their types. So each node can
receive any of those timestamps.

It is then trivial to add new timestamp or find the difference between
CLOCK-IN, CLOCK-OUT, to list items by SCHEDULED or DEADLINES, or past
DEADLINES.

Your questions gave me to thinking that I should make some basic
functions:

- when editing Org mode, to easily insert database entry from
  Hyperscope system so that the entry can be edited in a file itself,
  and automatically updated in the database when file is saved, or
  special command invoked, or user switched to other heading

- inserting of such item from database entry would make Org
  properties, such as HOSTNAME, DATABASE used, (username, passwords
  could be externally saved), TABLE, COLUMN in the table and TABLE
  ID.

- then an extension function to Org would need to check if there is
  some database entry like that, so that saving of the file or
  invocation of that function finds the entry and updates it to the
  remote database. Database could be local or remote.

- if another user on other part of the world wish to insert or edit
  that Org entry, that user would get the updated version

With above extensions, user would still be editing file, it could be
saved file, file on the disk, but the database entries would be
updated either from the database or to the database in the
background and without interruptions to user.

Multiple users could then edit the Org files, which would get updated
on each new opening of a file or the invocation of a function.

This then limits entries to Org mode, while my entries are now in any
mode. I have enriched mode notes or Org type notes in the database, or
markdown nodes, or anything. When markdown note is viewed,
markdown-mode is invoked, when Org type node is viewed in Emacs, then
Org mode is invoked in read only mode.

The list of defined nodes that I have now available are:

Possible completions are:
1 File 	10 Directory 	11 Launch Program
12 Media 	13 Info Node 	14 PDF
15 HyperScope ID 	16 PDF Query 	17 Org Heading
18 Org 	19 PDF by Page Nr. 	2 WWW
20 DJVU 	21 Video 	22 Message-ID
23 WRS Page Annotation 	24 Directory Action ➜ 	25 Shell Command
26 Self 	27 Action 	
29 FOLLOW-UP 	3 MPV play video at exact time 	30 aMule
31 Task 	32 Password 	33 PostgreSQL
34 Markdown 	35 Paraset 	36 Enriched
37 txt2tags 	38 EPUB 	4 Local File
5 Set ➜ 	6 YouTube Video at exact time 	7 YouTube Video
8 Emacs Lisp 	9 Note

There are no limits to what a node can look like. Each node can be a
note in itself, even if it is a file, there can be note for the file,
number 28. I have deleted not to confuse people. Password is maybe
encrypted in the database, directory action has similar meaning like
in Hyperbole package where file ~/tmp/.hyperscope may contain this:

(hyperscope-directory-action 35243)

and the node ID 35243 decides which directory actions relate to
directory ~/tmp and then window splits with options from the database
such as maybe to read some notes related to directory, or remove all
docview directories or to remove all GIF files in ~/tmp - the idea
comes from Hyperbole package that has similar feature.

People in the database may be assigned specific subtrees or Org files
or nodes, so by finding person one can find the notes, tasks,
assignments, related to people.

At export to Org file, those nodes that are not compatible to Org file
such as Markdown, txt2tags, video, would be exported as separate file
and such file would be hyperlinked from the main Org file similar to
what is defined here:
https://www.dougengelbart.org/content/view/110/460/#2a1b

Complete exported package could then be dispatched onto DVD rom, USB
stick, sent by email or uploaded. Org file can be used in similar
fashion as MIME to define on higher level collection of various
documents.

As my goal is not to provide all features to users, but to use
Hyperscope, that is why my development goes slow and is not yet
available to public. I would need to verify many functions before it
becomes public software. But for individuals interested, I am willing
to help one on one to create the database and provide systems I am
using. 

Jean