Integration of notmuch email references in notes in org-mode

Integration of notmuch email references in notes in org-mode

[Rainer M Krug]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Hi

- From time to time, I have to include email references in notes in org. There is obviously
org-notmuch.el, but I am not to happy with it as, when exporting the notes, the link is shown but
neither clickable nor the resulting email is shown in the exported document (both understandable).

I could also include a bash source block:

#+header: :results output org
#+begin_src sh
  notmuch show thread:XXXXXXXXXXXXXXXX
#+end_src

the result ooks ugly formated - if I use raw, I can not fold the result, due to strange characters
at the beginning of the header lines.

I think that source block approaȟ could work very nbicely, if one could somehow format the result
as an email - but I have no idea how.

I actually would prefer the source block approach, as the email would be physically in the org
document and foldable.

Any suggestions how I could achieve this?

Thanks,

Rainer

- -- 
Rainer M. Krug, PhD (Conservation Ecology, SUN), MSc (Conservation Biology, UCT), Dipl. Phys.
(Germany)

Centre of Excellence for Invasion Biology
Stellenbosch University
South Africa

Tel :       +33 - (0)9 53 10 27 44
Cell:       +33 - (0)6 85 62 59 98
Fax :       +33 - (0)9 58 10 27 44

Fax (D):    +49 - (0)3 21 21 25 22 44

email:      Rainer@krugs.de

Skype:      RMkrug
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.11 (GNU/Linux)
Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/

iQEcBAEBAgAGBQJRIjFsAAoJENvXNx4PUvmCB0kH+QFL4jRInkoOdwVbOiiD5DcO
pNLeMND0t1N5zBxORwDdFfdjLvN4nJ7r3i0Bh6qldtAbOc/7hKQx1E1wVxQoSJp+
XyB+H7vycb29l4cSGgXjqRbkgnwLy8NyYZlHza3dej3Nv1ZLdu5trdg/QCwxtivP
kmJPDhiZepAFmjPcU+hYicOf0iT+B1NpfPa3ZumaJEld1yFgHw2ELGJbReKwlsbJ
yDbUq6kTK4HfHRpcNmqvavEWPCSQunAUGNJUV0aXPya9Fh4k2+1UXSDPX+oZch9Z
nq9lOLoHyR8U3qvTicojmfJgyZ9LzpDqdlKgMW+FGLJsNLVqZ24EpBaSQ2iHgzc=
=DwaG
-----END PGP SIGNATURE-----

Re: Integration of notmuch email references in notes in org-mode

[Suvayu Ali]

Hi Rainer

On Mon, Feb 18, 2013 at 02:49:32PM +0100, Rainer M Krug wrote:
> 
> - From time to time, I have to include email references in notes in org. There is obviously
> org-notmuch.el, but I am not to happy with it as, when exporting the notes, the link is shown but
> neither clickable nor the resulting email is shown in the exported document (both understandable).
> 

This has also been an irritation of mine.  I think I will take your
second idea (include the email on export) and see if I can come-up with
a patch for org-notmuch.  Since I'm still a newbie elisp programmer,
this might take a while.  :-p

> I could also include a bash source block:
> 
> #+header: :results output org
> #+begin_src sh
>   notmuch show thread:XXXXXXXXXXXXXXXX
> #+end_src
> 
> the result ooks ugly formated - if I use raw, I can not fold the result, due to strange characters
> at the beginning of the header lines.

Actually what you see is supposed to be easily parsed.  From the
notmuch-show man page:

  --format=(text|json|sexp|mbox|raw)
  
    text (default for messages)
      
      The default plain-text format has all text-content MIME parts
      decoded.  Various components in the output, (message, header,
      body, attachment, and MIME part), will be delimited by
      easily-parsed markers. Each marker consists of a Control-L
      character (ASCII decimal 12), the name of the marker, and then
      either an opening or closing brace, ('{' or '}'), to either open
      or close the component.  For a multipart MIME message, these parts
      will be nested.

I'm assuming what you presently see is in text format (default).  Since
you are not happy with this, I would suggest you try the mbox format.
    
    [...]
    
    mbox
    
      All matching messages are output in the traditional, Unix mbox
      format with each message being prefixed by a line beginning with
      "From " and a blank line separating each message. Lines in the
      message content beginning with "From " (preceded by zero or more
      '>' characters) have an additional '>' character added.  This
      reversible escaping is termed "mboxrd" format and described in
      detail here:
      
      http://homepage.ntlworld.com/jonathan.deboynepollard/FGA/mail-mbox-formats.html

A quick question though, how do you plan on handling attachments, I
presume you want to strip them?  In that case you will have to parse the
output anyway.

Hope this helps,

-- 
Suvayu

Open source is the future. It sets us free.

Re: Integration of notmuch email references in notes in org-mode

[Rainer M Krug]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On 18/02/13 16:12, Suvayu Ali wrote:
> Hi Rainer
> 
> On Mon, Feb 18, 2013 at 02:49:32PM +0100, Rainer M Krug wrote:
>> 
>> - From time to time, I have to include email references in notes in org. There is obviously 
>> org-notmuch.el, but I am not to happy with it as, when exporting the notes, the link is shown
>> but neither clickable nor the resulting email is shown in the exported document (both
>> understandable).
>> 
> 
> This has also been an irritation of mine.  I think I will take your second idea (include the
> email on export) and see if I can come-up with a patch for org-notmuch.  Since I'm still a
> newbie elisp programmer, this might take a while.  :-p

If there is a solution in sight, I am happy to wait a bit - email in export, possibly in a block
so that it is set apart from the rest of the document, would be brilliant. Possibly having an
option which can be set to have links exported as links or expanded on export, would be really nice.

> 
>> I could also include a bash source block:
>> 
>> #+header: :results output org #+begin_src sh notmuch show thread:XXXXXXXXXXXXXXXX #+end_src
>> 
>> the result ooks ugly formated - if I use raw, I can not fold the result, due to strange
>> characters at the beginning of the header lines.
> 
> Actually what you see is supposed to be easily parsed.  From the notmuch-show man page:
> 
> --format=(text|json|sexp|mbox|raw)
> 
> text (default for messages)
> 
> The default plain-text format has all text-content MIME parts decoded.  Various components in
> the output, (message, header, body, attachment, and MIME part), will be delimited by 
> easily-parsed markers. Each marker consists of a Control-L character (ASCII decimal 12), the
> name of the marker, and then either an opening or closing brace, ('{' or '}'), to either open 
> or close the component.  For a multipart MIME message, these parts will be nested.
> 
> I'm assuming what you presently see is in text format (default).  Since you are not happy with
> this, I would suggest you try the mbox format.
> 
> [...]
> 
> mbox
> 
> All matching messages are output in the traditional, Unix mbox format with each message being
> prefixed by a line beginning with "From " and a blank line separating each message. Lines in
> the message content beginning with "From " (preceded by zero or more '>' characters) have an
> additional '>' character added.  This reversible escaping is termed "mboxrd" format and
> described in detail here:
> 
> http://homepage.ntlworld.com/jonathan.deboynepollard/FGA/mail-mbox-formats.html
> 

Haven't thought about that - should be straight forward to find a suitable format.

> A quick question though, how do you plan on handling attachments, I presume you want to strip
> them?

Yes - attachments should be included simply as a line "Attachment: NameOfAttachment" - that would
be fine.


Thanks, and I am looking forward to the new additions to org-notmuch,

Rainer

> In that case you will have to parse the output anyway.
> 
> Hope this helps,
> 

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.11 (GNU/Linux)
Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/

iQEcBAEBAgAGBQJRIknZAAoJENvXNx4PUvmCo1AIAI+xSqnxwHLCitiFmPO/wDXV
ygMahUG58s1sy9qDlaKMUmO5GQoHXO9HBFlNyHYt++iBqbUU02LDeCIHbFOv2CT6
Fnzk6SH/eq2sZMIctHju3ybLYYKqV/jJJQwTHEBNjPyywC6IE3cpx0Xftv/xKERw
hKiUttnyucZm3JVoZGLA5kAssq6SAUXFONxaRDekDl4Bch6HNfjoNTSpl4kw0FQz
Jqym+8I8zRMSbrXGyo87KcIM6h3sKTZaeSmAzeuVD3I2E3ovWC2ufbAewLhgpM8P
K8NuR5tURz4DLaoE0Qy9jy0PTZsvzNBt11dB4v9KQHHjIzacGYdiKuWHElLxoc4=
=CMyd
-----END PGP SIGNATURE-----

Including linked emails during export (was Re: Integration of notmuch email references in notes in org-mode)

[Suvayu Ali]

Hi,

On Mon, Feb 18, 2013 at 04:33:45PM +0100, Rainer M Krug wrote:
> On 18/02/13 16:12, Suvayu Ali wrote:
> > On Mon, Feb 18, 2013 at 02:49:32PM +0100, Rainer M Krug wrote:
> >> 
> >> - From time to time, I have to include email references in notes in org. There is obviously 
> >> org-notmuch.el, but I am not to happy with it as, when exporting the notes, the link is shown
> >> but neither clickable nor the resulting email is shown in the exported document (both
> >> understandable).
> >> 
> > 
> > This has also been an irritation of mine.  I think I will take your second idea (include the
> > email on export) and see if I can come-up with a patch for org-notmuch.  Since I'm still a
> > newbie elisp programmer, this might take a while.  :-p
> 
> If there is a solution in sight, I am happy to wait a bit - email in export, possibly in a block
> so that it is set apart from the rest of the document, would be brilliant. Possibly having an
> option which can be set to have links exported as links or expanded on export, would be really nice.

The option to open the link is out of the question since different
people read their emails differently.  There is no way for a library to
get everything right, specially since the link won't be in Emacs.  It is
in the exported file (e.g. html, pdf, txt).

Now to include the email during the export at first I thought it should
be possible to write a filter, but then I realised I need to insert an
email quote block after whatever parent greater-element holds the link
and then redirect the link to this new block.  A filter allows you a way
to modify an element, but nothing more.  So I think the correct approach
here would be to write a derived backend.  This complicates things quite
significantly since now you need to support each backend separately.  I
wonder if there is any easier (generic) way to do the above for all
present and future backends.

I have CC'd Nicolas, lets see what he has to say.

Cheers,

-- 
Suvayu

Open source is the future. It sets us free.

Naive email "referencing" using copy&paste (was: Integration of notmuch email references in notes in org-mode)

[Karl Voit]

* Rainer M Krug <r.m.krug@gmail.com> wrote:
>
> Hi

Hi!

> I actually would prefer the source block approach, as the email
> would be physically in the org document and foldable.
>
> Any suggestions how I could achieve this?

In my case, MUA (mutt) and Org are on different hosts. I am
accessing MUA through ssh/screen and copy relevant parts into blocks
like this:

#+BEGIN_VERSE
Date: Mon, 18 Feb 2013 15:58:22 +0000
From: Kickstarter <no-reply@kickstarter.com>
To: xxxxxxxxxxxxx
From: Project Boss <boss@example.com>
Subject: Project Update xxxxxxxxxxxxxxxxxx
Message-ID: <513_6f6d48753ee22646e9@xxxxxxxxxxxxxxxxxx>

Project deadline is re-schedults to May 2014 since
external ressources are not available before autumn.
#+END_VERSE

Sometimes, I am using BEGIN_QUOTE instead of BEGIN_VERSE though.

With copy&paste, it's not as quick as capture but this way I get the
advantage to strip down the message to its relevant parts. With the
Message-ID, I am able to locate the original email in my MUA very
quickly.

-- 
Karl Voit

Re: Naive email "referencing" using copy&paste

[Rainer M Krug]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On 18/02/13 23:02, Karl Voit wrote:
> * Rainer M Krug <r.m.krug@gmail.com> wrote:
>> 
>> Hi
> 
> Hi!
> 
>> I actually would prefer the source block approach, as the email would be physically in the
>> org document and foldable.
>> 
>> Any suggestions how I could achieve this?
> 
> In my case, MUA (mutt) and Org are on different hosts. I am accessing MUA through ssh/screen
> and copy relevant parts into blocks like this:
> 
> #+BEGIN_VERSE Date: Mon, 18 Feb 2013 15:58:22 +0000 From: Kickstarter
> <no-reply@kickstarter.com> To: xxxxxxxxxxxxx From: Project Boss <boss@example.com> Subject:
> Project Update xxxxxxxxxxxxxxxxxx Message-ID: <513_6f6d48753ee22646e9@xxxxxxxxxxxxxxxxxx>
> 
> Project deadline is re-schedults to May 2014 since external ressources are not available before
> autumn. #+END_VERSE
> 
> Sometimes, I am using BEGIN_QUOTE instead of BEGIN_VERSE though.
> 
> With copy&paste, it's not as quick as capture but this way I get the advantage to strip down
> the message to its relevant parts. With the Message-ID, I am able to locate the original email
> in my MUA very quickly.

Yes - I did something similar, but didn't think about the verse and quote blocks - thanks.

But I was looking for something more dynamic, i.e. when further comments come in the conversation,
I would like to have them accessible as well.

So a notmuch search based system (through links or through source blocks) is the ideal situation
in my case.

Thanks,

Rainer

> 

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.11 (GNU/Linux)
Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/

iQEcBAEBAgAGBQJRIzDRAAoJENvXNx4PUvmCS5oH/29cLZq6yULqxbsIYoYyz4n7
6Vi7g4XBuh6qLBtygvhhMv7V515MtaqYSDWDQZoFMKK5LQGZnjvhSB60bBGwDIGu
/I2t9CV7ZO2qF1KpR98FTBI+Ch1MbOzjHtV5OSihZAZJnAfyOYpBAYWzppmlvxi8
fSvGX23ob0Q0F6H7pe+b46W4o+EGA9birR0iU3kXPzqKGN2o4rmOlHc2ro9JPve6
dNt1XGisjGX/XojKoMRm7YJgMRcADkftH7Eli1ZUfqQSuWJL2DBy7u9r89YKoU/A
5nCBmF6C5jSLK3sNyNu3CcjHbxq33ttBV8tFXvcZlptp1cgN3wNdA/KhWwbhBgc=
=VWE1
-----END PGP SIGNATURE-----

Re: Including linked emails during export (was Re: Integration of notmuch email references in notes in org-mode)

[Rainer M Krug]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On 18/02/13 22:41, Suvayu Ali wrote:
> Hi,
> 
> On Mon, Feb 18, 2013 at 04:33:45PM +0100, Rainer M Krug wrote:
>> On 18/02/13 16:12, Suvayu Ali wrote:
>>> On Mon, Feb 18, 2013 at 02:49:32PM +0100, Rainer M Krug wrote:
>>>> 
>>>> - From time to time, I have to include email references in notes in org. There is
>>>> obviously org-notmuch.el, but I am not to happy with it as, when exporting the notes, the
>>>> link is shown but neither clickable nor the resulting email is shown in the exported
>>>> document (both understandable).
>>>> 
>>> 
>>> This has also been an irritation of mine.  I think I will take your second idea (include
>>> the email on export) and see if I can come-up with a patch for org-notmuch.  Since I'm
>>> still a newbie elisp programmer, this might take a while.  :-p
>> 
>> If there is a solution in sight, I am happy to wait a bit - email in export, possibly in a
>> block so that it is set apart from the rest of the document, would be brilliant. Possibly
>> having an option which can be set to have links exported as links or expanded on export,
>> would be really nice.
> 
> The option to open the link is out of the question since different people read their emails
> differently.  There is no way for a library to get everything right, specially since the link
> won't be in Emacs. It is in the exported file (e.g. html, pdf, txt).

If I understand the link system correctly, one would need to create a notmuch protocoll so that
linux can react to clicking the link. But you are right, this is outside org.


> 
> Now to include the email during the export at first I thought it should be possible to write a
> filter, but then I realised I need to insert an email quote block after whatever parent
> greater-element holds the link and then redirect the link to this new block.  A filter allows
> you a way to modify an element, but nothing more.  So I think the correct approach here would
> be to write a derived backend.  This complicates things quite significantly since now you need
> to support each backend separately.  I wonder if there is any easier (generic) way to do the
> above for all present and future backends.

How is the export of links to pictures handled - as far as I remember, if the image is in a LaTeX
suitable format, the image is included in the LaTeX file, and the same for html. But you are right
- - I guess this is handled in the backend. I don't think that would be a suitable approach as it
would include to much maintenance.

But what about defining notmuch as a new language for babel, so that one could do the following:

#+begin_src notmuch :exports raw
  search id:XXXXXXXXXXXXXXXXx
#+end_src

Would not be much different then

#+begin_src sh :exports raw
  notmuch search id:XXXXXXXXXXXXXXXXx
#+end_src

but possibly open more possibilities for customizations?
XXXXXXXXXXXXXXXXx
Or inline it could be src_notmuch{search id:XXXXXXXXXXXXXXXXx}

The general advantage would be that one can use the existing infrastructure for code blocks and

Cheers,

Rainer

> 
> I have CC'd Nicolas, lets see what he has to say.
> 
> Cheers,
> 

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.11 (GNU/Linux)
Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/

iQEcBAEBAgAGBQJRIzQHAAoJENvXNx4PUvmCD7gH/3uS607nlfa31Nzxyt7j6bRH
UrFmDtpaU0zgjpPvSdqCRgF2V1R8O4lXTQd8P66swQ+tWglKSXaPaeP625I1zo0U
wjMHoAsppiENz4lMVV6S7Hya3DuegLgwU+5+KcC7m2XbiFqzStIs8uhczyIZ9EKH
GjrvMLrglkw3Nf/Gq4etvvPVnH05UqrBHsOF4j6W8ZcDAM0fufbv5EuMXt4gvlqy
E5fJbth06mphZRe/d8jpQ0XR97RIz5Vq4gOajwBa3kLT7BY7Y2SYUu2PDV44YjwF
8d4CSTgq0ReDQsg8qch6tE1dli/ftw2i0Qe9j3MCZ3QItB5aJ/y6rh4oAmMhzi0=
=0LBU
-----END PGP SIGNATURE-----

Re: Including linked emails during export (was Re: Integration of notmuch email references in notes in org-mode)

[Nicolas Goaziou]

Hello,

Rainer M Krug <r.m.krug@gmail.com> writes:

> On 18/02/13 22:41, Suvayu Ali wrote:
>> Hi,
>> 
>> On Mon, Feb 18, 2013 at 04:33:45PM +0100, Rainer M Krug wrote:
>>> On 18/02/13 16:12, Suvayu Ali wrote:
>>>> On Mon, Feb 18, 2013 at 02:49:32PM +0100, Rainer M Krug wrote:
>>>>> 
>>>>> - From time to time, I have to include email references in notes in org. There is
>>>>> obviously org-notmuch.el, but I am not to happy with it as, when exporting the notes, the
>>>>> link is shown but neither clickable nor the resulting email is shown in the exported
>>>>> document (both understandable).
>>>>> 
>>>> 
>>>> This has also been an irritation of mine.  I think I will take your second idea (include
>>>> the email on export) and see if I can come-up with a patch for org-notmuch.  Since I'm
>>>> still a newbie elisp programmer, this might take a while.  :-p
>>> 
>>> If there is a solution in sight, I am happy to wait a bit - email in export, possibly in a
>>> block so that it is set apart from the rest of the document, would be brilliant. Possibly
>>> having an option which can be set to have links exported as links or expanded on export,
>>> would be really nice.
>> 
>> The option to open the link is out of the question since different people read their emails
>> differently.  There is no way for a library to get everything right, specially since the link
>> won't be in Emacs. It is in the exported file (e.g. html, pdf, txt).
>
> If I understand the link system correctly, one would need to create a notmuch protocoll so that
> linux can react to clicking the link. But you are right, this is outside org.
>
>
>> 
>> Now to include the email during the export at first I thought it should be possible to write a
>> filter, but then I realised I need to insert an email quote block after whatever parent
>> greater-element holds the link and then redirect the link to this new block.  A filter allows
>> you a way to modify an element, but nothing more.  So I think the correct approach here would
>> be to write a derived backend.  This complicates things quite significantly since now you need
>> to support each backend separately.  I wonder if there is any easier (generic) way to do the
>> above for all present and future backends.
>
> How is the export of links to pictures handled - as far as I remember, if the image is in a LaTeX
> suitable format, the image is included in the LaTeX file, and the same for html. But you are right
> - I guess this is handled in the backend. I don't think that would be a suitable approach as it
> would include to much maintenance.
>
> But what about defining notmuch as a new language for babel, so that one could do the following:
>
> #+begin_src notmuch :exports raw
>   search id:XXXXXXXXXXXXXXXXx
> #+end_src
>
> Would not be much different then
>
> #+begin_src sh :exports raw
>   notmuch search id:XXXXXXXXXXXXXXXXx
> #+end_src
>
> but possibly open more possibilities for customizations?
> XXXXXXXXXXXXXXXXx
> Or inline it could be src_notmuch{search id:XXXXXXXXXXXXXXXXx}
>
> The general advantage would be that one can use the existing infrastructure for code blocks and
>
> Cheers,
>
> Rainer
>
>> 
>> I have CC'd Nicolas, lets see what he has to say.

FWIW, I think the approach suggested by Rainer is good, i.e. writing
a new "language" for Babel, with specific headers. Such blocks would
then be expanded into appropriate Org syntax the exporter can
understand.


Regards,

-- 
Nicolas Goaziou

Re: Including linked emails during export (was Re: Integration of notmuch email references in notes in org-mode)

[Suvayu Ali]

Hey Nicolas,

On Wed, Feb 27, 2013 at 02:35:04PM +0100, Nicolas Goaziou wrote:
> Hello,
> 
> Rainer M Krug <r.m.krug@gmail.com> writes:
> >
> > But what about defining notmuch as a new language for babel, so that
> > one could do the following:
> >
> > #+begin_src notmuch :exports raw
> >   search id:XXXXXXXXXXXXXXXXx
> > #+end_src
> >
> > Would not be much different then
> >
> > #+begin_src sh :exports raw
> >   notmuch search id:XXXXXXXXXXXXXXXXx
> > #+end_src
> >
> > but possibly open more possibilities for customizations?
> > XXXXXXXXXXXXXXXXx
> > Or inline it could be src_notmuch{search id:XXXXXXXXXXXXXXXXx}
> >
> > The general advantage would be that one can use the existing
> > infrastructure for code blocks and
> 
> FWIW, I think the approach suggested by Rainer is good, i.e. writing
> a new "language" for Babel, with specific headers. Such blocks would
> then be expanded into appropriate Org syntax the exporter can
> understand.

Thanks for your feedback.  I haven't had any time to look at this yet.
I'll keep this in mind.

Cheers,

-- 
Suvayu

Open source is the future. It sets us free.