Re: Remembrance Agents

Re: Remembrance Agents

[hpgisleropen]

> The more information user
> enters into the database and the more tags and relations have been
> created the better the relevance.

Isn't the real problem-to-solve finding the actual semantic context and
then to relate it matching information?

After all, words (TAGS, ...) have different meaning in different
semantic contexts.

I suspect, 'just' relating to tags (words) and showing the attached
document (part) to her is not very useful, yes?

Re: Remembrance Agents

[Jean Louis]

* hpgisleropen@bluewin.ch <hpgisleropen@bluewin.ch> [2020-12-02 12:57]:
> > The more information user
> > enters into the database and the more tags and relations have been
> > created the better the relevance.
> 
> Isn't the real problem-to-solve finding the actual semantic context and
> then to relate it matching information?

That problem may easily be solved by using external tools such as
PostgreSQL as it has a built-in relevance search and indexing
functions.

To index files into PostgreSQL database is not hard task. If you
happen to have a need, let me know and I will help.

Yesterday I have implemented relevance searć in relation to website
pages in the database. As often I wish to reference URLs from my
websites to our clients. Among 4000+ URLs one has to find it by
thinking and not by browsing hierarchy.

Until now I was using 2 inputs approach like:

1. Write one word or part of word that I remember being part of the
   page.

2. When faced with completion candidates narrow it down with second or
   third word.

Yesterday I have implemented this search by using relevance matching
feature and it works fast:

(format "SELECT pages_id || ' ' || pages_title || ' ' || ts_rank_cd(to_tsvector(pages_title || pages_description),%s,32 /* rank/(rank+1) */) AS rank FROM pages, to_tsquery(%s) query WHERE query @@ to_tsvector(pages_title || pages_description) ORDER BY rank DESC LIMIT 30;" query query)) ;; TODO this cannot order by rank

The query is prepared with this:

(defun rcd-sql-prepare-text-query (query)
  (let* ((query (string-trim query))
	 (query (if (and (string-match " " query)
			 (not (string-match "&" query)))
		    (string-replace " " " & " query))))
    query))

because I normally need WORD AND WORD. But I could make it any how, I
could search for (WORD AND WORD) OR WORD OR (WORD AND WORD) all is
possible. The function converts "my term words" into "my & term &
words" where & has the meaning of logical AND.

This shortens my search within Emacs to just one query and not having
two steps. It finds relevant results with accuracy and precision. My
searches locate the hyperlink within 10 first results, so I have
limited it to 30 results.

> There is als After all, words (TAGS, ...) have different meaning in
> different semantic contexts.

In their own meanings yes. But for the user's mind they may be
related and useful to be searched together. If tag is there it is
related to the object, search without tag and with tag may give quite
different results.

> I suspect, 'just' relating to tags (words) and showing the attached
> document (part) to her is not very useful, yes?

Thinking of many other systems, in many of other systems tags are main
method of locating things. But it all comes from well organized user's
mind. If mind is not organized user will not know how to tag to make
future locating features work well. If it is tagged well locating
things is very fast.

- todo family son joe bicycle
- todo family son kim drivers-license
- todo business appointment
- todo health kidneys

When tags are used that way then locating things related to one son
among two sons become easy. Locating things for family is larger group
and locating business appointments becomes easy. When something is
done, the TODO should be changed to something else.

If user tags well then searches can be very useful.

But what if user does not tag well but still has some attributes
attached such as titles, body text, etc. then the relevance search may
help.

Chapter 12. Full Text Search
https://www.postgresql.org/docs/current/textsearch-intro.html#TEXTSEARCH-MATCHING

emacs-libpq @ Github
https://github.com/anse1/emacs-libpq

Remembrance Agents

[Gerardo Moro]

[-- Attachment #1: Type: text/plain, Size: 101 bytes --]

Is there currently any (good) implementation of the idea of the Remembrance
Agents in Emacs?
Thanks!

[-- Attachment #2: Type: text/html, Size: 132 bytes --]

Re: Remembrance Agents

[George Mauer]

[-- Attachment #1: Type: text/plain, Size: 477 bytes --]

As far as I know the only thing remotely like that is the org-roam buffer
when you are in a file managed by org-roam. This would be simply a linkable
list of other roam notes which reference the currently viewed note

So not exactly what you're looking for I suspect but a similarish concept

On Sat, Nov 28, 2020, 11:01 Gerardo Moro <gerardomoro37@gmail.com> wrote:

> Is there currently any (good) implementation of the idea of the
> Remembrance Agents in Emacs?
> Thanks!
>

[-- Attachment #2: Type: text/html, Size: 816 bytes --]

Re: Remembrance Agents

[Jean Louis]

 * Gerardo Moro <gerardomoro37@gmail.com> [2020-11-28 20:02]:
> Is there currently any (good) implementation of the idea of the Remembrance
> Agents in Emacs?

According to this document:

Hyperlink: https://alumni.media.mit.edu/~rhodes/Papers/remembrance.html

the implementation was running already in Emacs!

There is more to it:

https://www.emacswiki.org/emacs/RemembranceAgents

And picture: https://alumni.media.mit.edu/~rhodes/Papers/ra.small.gif

As I am developing HyperScope that is to augment the knowledge, at
least for those objects displayed within HyperScope but also in other
context or modes I could invoke something similar as Speedbar on the
side or in separate window that would show all relations to specific
object. It could also look from time to time into what text is user
writing and search relevant terms or tags to display such as
hyperlinks that are accessible on the side. The more information user
enters into the database and the more tags and relations have been
created the better the relevance.

In the context of web applications providing CRM Customer Relationship
Management there are always various related objects hyperlinked on the
same page. Same is valid for ERM or Enterprise Resource Management and
similar systems as they are already based on the relational databases.

Automatic hypertexting is another good feature that may be implemented
within Emacs for any relevant terms.

Re: Remembrance Agents

[Gerardo Moro]

[-- Attachment #1: Type: text/plain, Size: 1996 bytes --]

Thanks, I had seen those pages. Seems an old project and I haven't used it.
The Github page is here https://github.com/emacsmirror/remem
Has somebody installed it and tried it? It would be great to have a
screen recording demo.
So as I see, it basically shows in the minibuffer documents that are
related to what you write.
Correct me if I'm wrong.
Your project sounds intriguing!
GM

El sáb, 28 nov 2020 a las 20:29, Jean Louis (<bugs@gnu.support>) escribió:

>  * Gerardo Moro <gerardomoro37@gmail.com> [2020-11-28 20:02]:
> > Is there currently any (good) implementation of the idea of the
> Remembrance
> > Agents in Emacs?
>
> According to this document:
>
> Hyperlink: https://alumni.media.mit.edu/~rhodes/Papers/remembrance.html
>
> the implementation was running already in Emacs!
>
> There is more to it:
>
> https://www.emacswiki.org/emacs/RemembranceAgents
>
> And picture: https://alumni.media.mit.edu/~rhodes/Papers/ra.small.gif
>
> As I am developing HyperScope that is to augment the knowledge, at
> least for those objects displayed within HyperScope but also in other
> context or modes I could invoke something similar as Speedbar on the
> side or in separate window that would show all relations to specific
> object. It could also look from time to time into what text is user
> writing and search relevant terms or tags to display such as
> hyperlinks that are accessible on the side. The more information user
> enters into the database and the more tags and relations have been
> created the better the relevance.
>
> In the context of web applications providing CRM Customer Relationship
> Management there are always various related objects hyperlinked on the
> same page. Same is valid for ERM or Enterprise Resource Management and
> similar systems as they are already based on the relational databases.
>
> Automatic hypertexting is another good feature that may be implemented
> within Emacs for any relevant terms.
>

[-- Attachment #2: Type: text/html, Size: 2795 bytes --]

Re: Remembrance Agents

[Eric S Fraga]

On Sunday, 29 Nov 2020 at 15:07, Gerardo Moro wrote:
> Has somebody installed it and tried it? It would be great to have a
> screen recording demo.

I've used it on and off in the past, within the past 2 years most
recently.  I populate the database it uses with both emails and org
files.  While writing, it shows lines from documents that match in a
separate window (whose size can be customized).

It does work but I find that I tend to forget about the remembrance
window most of the time so eventually I turn it off.  I never seem to
have enough screen real estate to justify its use (despite having 38"
and 27" monitors side by side on my desk ;-)).

There was a blog post early this year on this topic [1] which you might
find interesting.  It talks mostly about recoll [2] which is not so much
a remembrance agent but a search tool

HTH,
eric

Footnotes:
[1]  https://blog.jethro.dev/posts/remembrance_agents/

[2]  https://www.lesbonscomptes.com/recoll/

-- 
: Eric S Fraga via Emacs 28.0.50, Org release_9.4-143-g9a1549

Re: Remembrance Agents

[Jean Louis]

* Eric S Fraga <e.fraga@ucl.ac.uk> [2020-11-29 16:52]:
> On Sunday, 29 Nov 2020 at 15:07, Gerardo Moro wrote:
> > Has somebody installed it and tried it? It would be great to have a
> > screen recording demo.
> 
> I've used it on and off in the past, within the past 2 years most
> recently.  I populate the database it uses with both emails and org
> files.  While writing, it shows lines from documents that match in a
> separate window (whose size can be customized).
> 
> It does work but I find that I tend to forget about the remembrance
> window most of the time so eventually I turn it off.  I never seem to
> have enough screen real estate to justify its use (despite having 38"
> and 27" monitors side by side on my desk ;-)).

Sadly there are some errors today.

depmode=gcc3 /bin/sh ../depcomp \
gcc -DHAVE_CONFIG_H -I. -I. -I.. -I.. -I../savutil -I../savutil/parsedate -I../template -I../plugins -I\/usr/include/pcre -I\/sw/include    -g -O2 -c `test -f 'gbuf.c' || echo './'`gbuf.c
gbuf.c:43:8: error: conflicting types for ‘strnlen’
 size_t strnlen (char *s, size_t n)
        ^~~~~~~
In file included from ../savant.h:50:0,
                 from gbuf.c:39:
/usr/include/string.h:401:15: note: previous declaration of ‘strnlen’ was here
 extern size_t strnlen (const char *__string, size_t __maxlen)
               ^~~~~~~
make[2]: *** [Makefile:191: gbuf.o] Error 1
make[2]: Leaving directory '/home/data1/protected/.emacs.d/remem/savutil'
make[1]: *** [Makefile:394: ../savutil/libsavutil.a] Error 2
make[1]: Leaving directory '/home/data1/protected/.emacs.d/remem/main'
make: *** [Makefile:428: main/ra-index] Error 2
~/.emacs.d/remem
$

Re: Remembrance Agents

[Eric S Fraga]

On Sunday, 29 Nov 2020 at 20:29, Jean Louis wrote:
> Sadly there are some errors today.

On Debian, at least, there is a remembrance-agent package which has the
binaries and the Emacs package.

-- 
: Eric S Fraga via Emacs 28.0.50, Org release_9.4-143-g9a1549

Re: Remembrance Agents

[Jean Louis]

* Eric S Fraga <e.fraga@ucl.ac.uk> [2020-11-29 20:48]:
> On Sunday, 29 Nov 2020 at 20:29, Jean Louis wrote:
> > Sadly there are some errors today.
> 
> On Debian, at least, there is a remembrance-agent package which has the
> binaries and the Emacs package.

PostgreSQL supports relevance search with its built-in functions:
https://www.postgresql.org/docs/current/textsearch-intro.html

Emacs package emacs-libpq is soon coming to GNU ELPA.

emacs-libpq @ Github
https://github.com/anse1/emacs-libpq

Using Emacs Lisp functions to store documents for indexing and to get
relevant results is then integration that may be useful.

I may use this functions for semantic locating functions in
Hyperscope. Think of something, put some words and get it, without me
putting great effort to parse words, format them, juggle with terms,
lists and so on. 

PostgreSQL database support is provided in general for any language
and there is support for specific languages:

database=# \dF
               List of text search configurations
   Schema   |    Name    |              Description              
------------+------------+---------------------------------------
 pg_catalog | arabic     | configuration for arabic language
 pg_catalog | danish     | configuration for danish language
 pg_catalog | dutch      | configuration for dutch language
 pg_catalog | english    | configuration for english language
 pg_catalog | finnish    | configuration for finnish language
 pg_catalog | french     | configuration for french language
 pg_catalog | german     | configuration for german language
 pg_catalog | greek      | configuration for greek language
 pg_catalog | hungarian  | configuration for hungarian language
 pg_catalog | indonesian | configuration for indonesian language
 pg_catalog | irish      | configuration for irish language
 pg_catalog | italian    | configuration for italian language
 pg_catalog | lithuanian | configuration for lithuanian language
 pg_catalog | nepali     | configuration for nepali language
 pg_catalog | norwegian  | configuration for norwegian language
 pg_catalog | portuguese | configuration for portuguese language
 pg_catalog | romanian   | configuration for romanian language
 pg_catalog | russian    | configuration for russian language
 pg_catalog | simple     | simple configuration
 pg_catalog | spanish    | configuration for spanish language
 pg_catalog | swedish    | configuration for swedish language
 pg_catalog | tamil      | configuration for tamil language
 pg_catalog | turkish    | configuration for turkish language
(23 rows)

Re: Remembrance Agents

[Gerardo Moro]

[-- Attachment #1: Type: text/plain, Size: 1507 bytes --]

Thanks! I get an idea. Will try to use soon and let you know.
Thanks for the links, very helpful.
Question: when you say it uses emails, can it read any email database (I
have old ones from Microsoft outlook both Mac and Windows)? What if you
only use webmail, in which format shall you feed the RA?
Thanks

El dom, 29 nov 2020 a las 15:52, Eric S Fraga (<e.fraga@ucl.ac.uk>)
escribió:

> On Sunday, 29 Nov 2020 at 15:07, Gerardo Moro wrote:
> > Has somebody installed it and tried it? It would be great to have a
> > screen recording demo.
>
> I've used it on and off in the past, within the past 2 years most
> recently.  I populate the database it uses with both emails and org
> files.  While writing, it shows lines from documents that match in a
> separate window (whose size can be customized).
>
> It does work but I find that I tend to forget about the remembrance
> window most of the time so eventually I turn it off.  I never seem to
> have enough screen real estate to justify its use (despite having 38"
> and 27" monitors side by side on my desk ;-)).
>
> There was a blog post early this year on this topic [1] which you might
> find interesting.  It talks mostly about recoll [2] which is not so much
> a remembrance agent but a search tool
>
> HTH,
> eric
>
> Footnotes:
> [1]  https://blog.jethro.dev/posts/remembrance_agents/
>
> [2]  https://www.lesbonscomptes.com/recoll/
>
> --
> : Eric S Fraga via Emacs 28.0.50, Org release_9.4-143-g9a1549
>

[-- Attachment #2: Type: text/html, Size: 2075 bytes --]

Re: Remembrance Agents

[Eric S Fraga]

On Monday, 30 Nov 2020 at 08:48, Gerardo Moro wrote:
> Question: when you say it uses emails, can it read any email database (I
> have old ones from Microsoft outlook both Mac and Windows)? What if you
> only use webmail, in which format shall you feed the RA?

No, I don't think the remembrance agent can access webmail of any
sort.  It is file based.  In my case, all of my emails are stored
locally (I use gnus nnml groups and POP3 to retrieve my emails from the
mail host).

-- 
: Eric S Fraga via Emacs 28.0.50, Org release_9.4-143-g9a1549

Re: Remembrance Agents

[Gerardo Moro]

[-- Attachment #1: Type: text/plain, Size: 809 bytes --]

Ok, thanks. How do you specify the location and is the formatting ok when
showing you the relevance pieces of text from the local databases?

El lun, 30 nov 2020 a las 11:31, Eric S Fraga (<e.fraga@ucl.ac.uk>)
escribió:

> On Monday, 30 Nov 2020 at 08:48, Gerardo Moro wrote:
> > Question: when you say it uses emails, can it read any email database (I
> > have old ones from Microsoft outlook both Mac and Windows)? What if you
> > only use webmail, in which format shall you feed the RA?
>
> No, I don't think the remembrance agent can access webmail of any
> sort.  It is file based.  In my case, all of my emails are stored
> locally (I use gnus nnml groups and POP3 to retrieve my emails from the
> mail host).
>
> --
> : Eric S Fraga via Emacs 28.0.50, Org release_9.4-143-g9a1549
>

[-- Attachment #2: Type: text/html, Size: 1124 bytes --]

Re: Remembrance Agents

[Eric S Fraga]

On Monday, 30 Nov 2020 at 11:37, Gerardo Moro wrote:
> Ok, thanks. How do you specify the location and is the formatting ok when
> showing you the relevance pieces of text from the local databases?

The agent needs to be told which files to index (check the man page for
ra-index) and, in my case, I point it to all .org files and all text
files under ~/Mail.  The formatting is okay as far as I am concerned but
this is a subjective aspect that will differ for everybody!

-- 
: Eric S Fraga via Emacs 28.0.50, Org release_9.4-143-g9a1549

Re: Remembrance Agents

[Jean Louis]

* Eric S Fraga <e.fraga@ucl.ac.uk> [2020-11-30 13:11]:
> On Monday, 30 Nov 2020 at 11:37, Gerardo Moro wrote:
> > Ok, thanks. How do you specify the location and is the formatting ok when
> > showing you the relevance pieces of text from the local databases?
> 
> The agent needs to be told which files to index (check the man page for
> ra-index) and, in my case, I point it to all .org files and all text
> files under ~/Mail.  The formatting is okay as far as I am concerned but
> this is a subjective aspect that will differ for everybody!

SMART (String Matching Algorithm Research Tool)
https://github.com/smart-tool/smart

Does it use this algorithm? I am cloning that one to see how useful it
could be in relevance searches.

,----
| However, I can also see that the PostgreSQL database has already
| built-in features for relevance searches:
| 
| https://www.postgresql.org/docs/current/textsearch-intro.html
`----

This makes then many external software pieces redundant, it becomes
again trivial to work with a database as underlying algorithm and
functions have already been programmed. Making a new remembrance agent
based on various structured information is making few higher level
functions that integrate the underlying capability of PostgreSQL
database and Emacs users' editing environment:

0. Create appropriate table in the database.

1. Make automatic or semi-automatic list of Org files or other files.

2. Make simple Emacs Lisp function to index all those files in the
   database. This is probably few lines function.

3. Make few Emacs Lisp functions that observes what users is doing and
   display to user relevant results with the possibility to construct
   hyperlinks from the result buffer straight to Org buffer, or
   possibility to open up those relevant files for more references.

Jean

Re: Remembrance Agents

[Jean Louis]

* Eric S Fraga <e.fraga@ucl.ac.uk> [2020-11-30 12:31]:
> On Monday, 30 Nov 2020 at 08:48, Gerardo Moro wrote:
> > Question: when you say it uses emails, can it read any email database (I
> > have old ones from Microsoft outlook both Mac and Windows)? What if you
> > only use webmail, in which format shall you feed the RA?
> 
> No, I don't think the remembrance agent can access webmail of any
> sort.  It is file based.  In my case, all of my emails are stored
> locally (I use gnus nnml groups and POP3 to retrieve my emails from the
> mail host).

recoll and other indexers could be glued into something similar as
remembrance agents. While user is typing some words in a paragraph
could quickly be accessed and relevant documents with those words
could appear on the side of the buffer.

Re: Remembrance Agents

[Jean Louis]

* Gerardo Moro <gerardomoro37@gmail.com> [2020-11-30 09:49]:
> Thanks! I get an idea. Will try to use soon and let you know.
> Thanks for the links, very helpful.
> Question: when you say it uses emails, can it read any email database (I
> have old ones from Microsoft outlook both Mac and Windows)? What if you
> only use webmail, in which format shall you feed the RA?

I think that all those formats can be converted to something you could
be using centrally on your computer.

I am recommending Maildir format. But that would depend of your email
client that you use.

I am using Maildir since years as each message is its own file and
concurrency is there and I never lose messages. I was using mbox and
mh and various email formats, nothing was ever so stable as Maildir.

Putting each person's conversation in separate folder:

~/Maildir/name@example.com

then allows me to easily access all previous conversation to that
person. This implies that saving messages must be configured to save
into such specific maildir automatically.

recoll is not remembrance agent. It is a desktop search tool that
supports various formats. There are other desktop search tools on
GNU/Linux as free software:

Beagle - Quickly find the stuff you care about.
http://beagle-project.org/

Recoll is a desktop full-text search tool.
https://www.lesbonscomptes.com/recoll/

Tracker is a filesystem indexer, metadata storage system and search tool
https://wiki.gnome.org/Projects/Tracker

NEPOMUK - The Social Semantic Desktop
https://nepomuk.semanticdesktop.org/

Terrier is a highly flexible, efficient, and effective open source search engine
http://www.terrier.org/

In general as Emacs user you may need to choose one that works on
command line. I am not sure if all of those work on command line.

Command line output can then be customized to appear in Emacs.

Then hyperlink may be constructed in Org file as well for various
relevant semantic searches.

For Hyperscope dynamic knowledge repository since we talked about
remembrance I have at least implemented the table column hlinks_rank
that increases its rank by usage. Maybe I should call it different.

Org mode does not track every specific node and is not multi-user:

- when was the specific node edited?
- which person edited specific node?
- what is previous version of that node?
- which hyperlink was accessed how many times?

Would it track that information without disturbance, then users would
have list of most used nodes, most accessed or frequently edited
nodes, groupware based nodes.

In that specific sense of tracking which URL have been accessed how
many times, this gives a list that helps to remember what could be
more relevant in context user is working in.

As the rank increases, the hyperdocument has its own tags, title,
information, and those words mostly used can be indexed and while user
is typing those words could appear on the side to give more popular
references.

Then access could be tracked on files as PDF, images, videos,
directories within Dired, then last of most accessed files could be in
front of us. Recentf does similar function in Emacs. But Emacs
bookmarks are not tracked and sorted by their access.

I will try downloading Remembrance from Debian, as maybe they have
patches that work.

Jean

Re: Remembrance Agents

[Gerardo Moro]

[-- Attachment #1: Type: text/plain, Size: 896 bytes --]

> * Gerardo Moro <gerardomoro37@gmail.com> [2020-11-30 09:49]:
> > Thanks! I get an idea. Will try to use soon and let you know.
> > Thanks for the links, very helpful.
> > Question: when you say it uses emails, can it read any email database (I
> > have old ones from Microsoft outlook both Mac and Windows)? What if you
> > only use webmail, in which format shall you feed the RA?
>
> I think that all those formats can be converted to something you could
> be using centrally on your computer.
>
> I am recommending Maildir format. But that would depend of your email
> client that you use.
>

My emails are in format .emlx which is only readable by MAIL. I also have
used Windows' Microsoft Office which stores I think in PLIST format.
Is it possible to convert them to Maildir format? I don't know how any
search software could search within those files if they are not converted.

Best,
GM

[-- Attachment #2: Type: text/html, Size: 1304 bytes --]

Re: Remembrance Agents

[Jean Louis]

* Gerardo Moro <gerardomoro37@gmail.com> [2020-11-30 14:17]:
> > * Gerardo Moro <gerardomoro37@gmail.com> [2020-11-30 09:49]:
> > > Thanks! I get an idea. Will try to use soon and let you know.
> > > Thanks for the links, very helpful.
> > > Question: when you say it uses emails, can it read any email database (I
> > > have old ones from Microsoft outlook both Mac and Windows)? What if you
> > > only use webmail, in which format shall you feed the RA?
> >
> > I think that all those formats can be converted to something you could
> > be using centrally on your computer.
> >
> > I am recommending Maildir format. But that would depend of your email
> > client that you use.
> >
> 
> My emails are in format .emlx which is only readable by MAIL. I also have
> used Windows' Microsoft Office which stores I think in PLIST format.
> Is it possible to convert them to Maildir format? I don't know how any
> search software could search within those files if they are not converted.

emlx2maildir convert emlx format to maildir
http://mike.laiosa.org/software/emlx2maildir/

There must be PLIST converters too.

Life will become easy with maildir, but I do not know which email
software you use. I am using mutt, not Emacs. mu4e package supports
Maildir and database search.

Re: Remembrance Agents

[Jean Louis]

* Gerardo Moro <gerardomoro37@gmail.com> [2020-11-29 16:08]:
> Thanks, I had seen those pages. Seems an old project and I haven't used it.
> The Github page is here https://github.com/emacsmirror/remem

That is great, I am much interested in that. I was thinking it is not
available any more. I will test it.