Org Mode and PDF Notes!

Org Mode and PDF Notes!

[Matt Price]

[-- Attachment #1: Type: text/plain, Size: 317 bytes --]

I've just written up a post on my workflow for PDF's Since my blog has, I
think, a readership of 0 (surely there's a way to get emacsers to follow
me? ah well), I will post a link here in the hopes that someone will be
interested:

http://matt.hackinghistory.ca/2015/11/11/note-taking-with-pdf-tools/

All best!
Matt

[-- Attachment #2: Type: text/html, Size: 475 bytes --]

Re: Org Mode and PDF Notes!

[Kaushal Modi]

Have you had a look at https://github.com/rudolfochrist/interleave

The interleave package works great for me: taking notes in an org-mode
buffer in one side (buffer) corresponding to the pdf page open in
pdf-tools on the other side (buffer).

Re: Org Mode and PDF Notes!

[Matt Price]

[-- Attachment #1: Type: text/plain, Size: 834 bytes --]

On Wed, Nov 11, 2015 at 9:59 AM, Kaushal Modi <kaushal.modi@gmail.com>
wrote:

> Have you had a look at https://github.com/rudolfochrist/interleave


not till now!

>
>
> The interleave package works great for me: taking notes in an org-mode
> buffer in one side (buffer) corresponding to the pdf page open in
> pdf-tools on the other side (buffer).
>
Huh.  Looks pretty neat.  Doesn't seem to work with pdfview yet, and I'm
sort of addicted now. But it seems like a great idea.  I can think of some
changes, e.g., would be nice to keep all my notes for all
books/articles/etc. in a single file; if the interleave options could be
set on a per-heading basis that owuld be really cool. And if it worked with
pdfview, one could add highlights to the pdf text while writing notes in
the org window. Huh. Interesting. Thanks for sharing.

[-- Attachment #2: Type: text/html, Size: 1392 bytes --]

Re: Org Mode and PDF Notes!

[Kaushal Modi]

>  Doesn't seem to work with pdfview yet, and I'm sort of addicted now.

I use pdf-view too (the major mode for viewing PDF files that comes
with pdf-tools, right?) and interleave works great!

> I can think of some changes, e.g., would be nice to keep all my notes for all books/articles/etc. in a single file

Yeah, right now it is one org file per pdf.

> if the interleave options could be set on a per-heading basis that would be really cool.

What sort of options are you thinking about? Right now it creates one
org heading for each set of notes corresponding to a page.

May be you can work with the interleave.el dev to add in your ideas
and code. I had worked with him to make interleave support pdf-tools
and it was a pleasure to work with him.

Re: Org Mode and PDF Notes!

[Matt Price]

[-- Attachment #1: Type: text/plain, Size: 1028 bytes --]

On Wed, Nov 11, 2015 at 3:48 PM, Kaushal Modi <kaushal.modi@gmail.com>
wrote:

> >  Doesn't seem to work with pdfview yet, and I'm sort of addicted now.
>
> I use pdf-view too (the major mode for viewing PDF files that comes
> with pdf-tools, right?) and interleave works great!
>
ah cool, i didn't know that.

>
> > I can think of some changes, e.g., would be nice to keep all my notes
> for all books/articles/etc. in a single file
>
> Yeah, right now it is one org file per pdf.
>
> > if the interleave options could be set on a per-heading basis that would
> be really cool.
>
> What sort of options are you thinking about? Right now it creates one
> org heading for each set of notes corresponding to a page.
>

> May be you can work with the interleave.el dev to add in your ideas
> and code. I had worked with him to make interleave support pdf-tools
> and it was a pleasure to work with him.
>
I meant INTERLEAVE_PDF. I just opened an issue:
https://github.com/rudolfochrist/interleave/issues/13  so who knows,
maybe...

[-- Attachment #2: Type: text/html, Size: 1832 bytes --]

Re: Org Mode and PDF Notes!

[Sebastian Christ]

>>>>> On Wed, 11 Nov 2015 15:58:24 -0500, Matt Price <moptop99@gmail.com> said:

  > I meant INTERLEAVE_PDF. I just opened an issue:
  > https://github.com/rudolfochrist/interleave/issues/13  so who knows,
  > maybe...

Sound like a good idea. I'm currently a little short on time,but perhaps
I can look into that this weekend.

-Sebastian

Re: Org Mode and PDF Notes!

[Sebastian Christ]

>>>>> On Wed, 11 Nov 2015 15:48:15 -0500, Kaushal Modi <kaushal.modi@gmail.com> said:

  > May be you can work with the interleave.el dev to add in your ideas
  > and code. I had worked with him to make interleave support pdf-tools
  > and it was a pleasure to work with him.

You're so kind! It was also a pleasure for me.

Best,
Sebastian

Re: Org Mode and PDF Notes!

[Xebar Saram]

[-- Attachment #1: Type: text/plain, Size: 508 bytes --]

thanks for the interesting post!

you now have 0+1 readers (just subscribed via rss)  ;-)

best

Z

On Wed, Nov 11, 2015 at 4:42 PM, Matt Price <moptop99@gmail.com> wrote:

> I've just written up a post on my workflow for PDF's Since my blog has, I
> think, a readership of 0 (surely there's a way to get emacsers to follow
> me? ah well), I will post a link here in the hopes that someone will be
> interested:
>
> http://matt.hackinghistory.ca/2015/11/11/note-taking-with-pdf-tools/
>
> All best!
> Matt
>

[-- Attachment #2: Type: text/html, Size: 1144 bytes --]

Re: Org Mode and PDF Notes!

[Russell Adams]

On Wed, Nov 11, 2015 at 09:42:13AM -0500, Matt Price wrote:
> I've just written up a post on my workflow for PDF's Since my blog has, I
> think, a readership of 0 (surely there's a way to get emacsers to follow
> me? ah well), I will post a link here in the hopes that someone will be
> interested:
>
> http://matt.hackinghistory.ca/2015/11/11/note-taking-with-pdf-tools/
>
> All best!
> Matt

In the past I have just had the PDF open in emacs (docview?), made a
bookmark, and inserted that into Org. It only links to the page
though, not the text.

Thanks.


------------------------------------------------------------------
Russell Adams                            RLAdams@AdamsInfoServ.com

PGP Key ID:     0x1160DCB3           http://www.adamsinfoserv.com/

Fingerprint:    1723 D8CA 4280 1EC9 557F  66E8 1154 E018 1160 DCB3

Re: Org Mode and PDF Notes!

[Jeffrey DeLeo]

Matt Price <moptop99@gmail.com> writes:
> I've just written up a post on my workflow for PDF's

Thanks, this is helpful.

Re: Org Mode and PDF Notes!

[Matt Price]

[-- Attachment #1: Type: text/plain, Size: 455 bytes --]

FWIW, I've just updated the post with
- links that jump directly to the highlighted text
- a new custom link type that opens highlights in a temporary buffer
instead.

http://matt.hackinghistory.ca/2015/11/11/note-taking-with-pdf-tools/

On Wed, Nov 11, 2015 at 11:40 AM, Jeffrey DeLeo <JeffreyDeLeo@gmail.com>
wrote:

>
> Matt Price <moptop99@gmail.com> writes:
> > I've just written up a post on my workflow for PDF's
>
> Thanks, this is helpful.
>
>
>

[-- Attachment #2: Type: text/html, Size: 984 bytes --]

Re: Org Mode and PDF Notes!

[Memnon Anon]

Matt Price <moptop99@gmail.com> writes:

> [...] Since my blog
> has, I think, a readership of 0 (surely there's a way to get emacsers
> to follow me? ah well), I will post a link here [...]

http://planet.emacsen.org/
-- 
/-------------------------------------------\
| SDF and SDF-EU Public Access UNIX System  |
| http://sdf.org   ||   http://sdf-eu.org   |
=============================================

Re: Org Mode and PDF Notes!

[Matt Price]

[-- Attachment #1: Type: text/plain, Size: 435 bytes --]

On Wed, Nov 11, 2015 at 12:09 PM, Memnon Anon <memnon+usenet@freeshell.org>
wrote:

> Matt Price <moptop99@gmail.com> writes:
>
> > [...] Since my blog
> > has, I think, a readership of 0 (surely there's a way to get emacsers
> > to follow me? ah well), I will post a link here [...]
>
> http://planet.emacsen.org/\ <http://planet.emacsen.org/>

But I can't figure out how to add my blog to the list there! Can you tell
me the secret?

[-- Attachment #2: Type: text/html, Size: 933 bytes --]

Re: Org Mode and PDF Notes!

[Memnon Anon]

Matt Price <moptop99@gmail.com> writes:

> On Wed, Nov 11, 2015 at 12:09 PM, Memnon Anon <
> memnon+usenet@freeshell.org> wrote:
>
>     http://planet.emacsen.org/\
>
> But I can't figure out how to add my blog to the list there! Can you
> tell me the secret?

http://edward.oconnor.cx/2009/11/planet-emacsen-turns-3

,----
| If you’d like me to add your Emacs feed to Planet Emacsen, send me an
| email [mailto:hober0@gmail.com] and I’ll get right on it.
`----

While you are at it, please ask him to put that info somewhere more
visible on the site itself...
-- 
/-------------------------------------------\
| SDF and SDF-EU Public Access UNIX System  |
| http://sdf.org   ||   http://sdf-eu.org   |
=============================================

Re: Org Mode and PDF Notes!

[Ramon Diaz-Uriarte]

Dear Matt,


On Wed, 11-11-2015, at 15:42, Matt Price <moptop99@gmail.com> wrote:
> I've just written up a post on my workflow for PDF's Since my blog has, I
> think, a readership of 0 (surely there's a way to get emacsers to follow
> me? ah well), I will post a link here in the hopes that someone will be

Add another 1 :-)

> interested:
>
> http://matt.hackinghistory.ca/2015/11/11/note-taking-with-pdf-tools/
>

Really neat! A few comments/questions/ramblings:

- The type of highlights you get from RepliGo contain the text itself. I
  mean, when in your pdf I use C-c C-a l, the buffer showing the contents
  of each highlight contain  the highlighted text.

  This is not what I get from, say, EzPDF (which is what I use on Android),
  or from highlighting from pdf-tools itself using C-c C-a h, or from
  highlighting from Okular. The contents just gives the rectangle). Hummmm...


  Because of this, when I use your code on my pdfs, I only get things
  such as

Highlight
([[pdfview:/home/ramon/Zotero-data/storage/ESHHD4KW/Frank_2015_Commentary.pdf::5][Frank_2015_Commentary]],
5)


  instead of the text. Bummer! I wonder if RepliGO gives you a lot more
  than the rest, or if I am doing something silly.


- You have to call mwp/pdf-multi-extract on each file/set of files. I guess
  if I knew elisp, I'd find it trivial to iterate over a set of directories
  and subdirectories (and do this using a cron job at night), and also
  place everything in one single org file. Would this be something
  reasonable to do?

  (This might be related to your second Todo)


- I know nothing about how it works, and it does not use pdf-tools, but in
  your first Todo you mention: "extend the pdfview link type (in
  org-pdfview) to permit me to specify the precise location of an
  annotation,".  PDF.js (https://mozilla.github.io/pdf.js/), which is
  used for instance by zotfile (http://zotfile.com/) does that and it works
  out of the box with Okular (but I've not been able to get it to work with
  pdftools).


- In case it matters, I have somewhat similar modus operandi.  I do a lot
  of PDF reading, including note-taking and highlighting, in android
  tablets ---I use EzPDF, which also embeds the notes in the PDF. I have a
  cron job that extracts all the highlights and annotations of all the PDFs
  and places them in a single org file. The kludge is explained here:
  https://github.com/rdiaz02/Adios_Mendeley#extracting-all-pdf-annotations-and-placing-them-in-an-org-mode-file
  The truth is I use two mechanisms for PDF annotation and highlighting
  extraction, since none is fully satisfactory to me, but the one that uses
  Ruby (i.e., that does not depend on poppler) is able to actually extract
  the text of the highlights.


Best, and thanks again for sharing,


R.





> All best!
> Matt


-- 
Ramon Diaz-Uriarte
Department of Biochemistry, Lab B-25
Facultad de Medicina
Universidad Autónoma de Madrid 
Arzobispo Morcillo, 4
28029 Madrid
Spain

Phone: +34-91-497-2412

Email: rdiaz02@gmail.com
       ramon.diaz@iib.uam.es

http://ligarto.org/rdiaz

Re: Org Mode and PDF Notes!

[Matt Price]

[-- Attachment #1: Type: text/plain, Size: 4622 bytes --]

On Wed, Nov 11, 2015 at 3:17 PM, Ramon Diaz-Uriarte <rdiaz02@gmail.com>
wrote:

> Dear Matt,
>
>
> On Wed, 11-11-2015, at 15:42, Matt Price <moptop99@gmail.com> wrote:
> > I've just written up a post on my workflow for PDF's Since my blog has, I
> > think, a readership of 0 (surely there's a way to get emacsers to follow
> > me? ah well), I will post a link here in the hopes that someone will be
>
> Add another 1 :-)
>
> > interested:
> >
> > http://matt.hackinghistory.ca/2015/11/11/note-taking-with-pdf-tools/
> >
>
> Really neat! A few comments/questions/ramblings:
>
> - The type of highlights you get from RepliGo contain the text itself. I
>   mean, when in your pdf I use C-c C-a l, the buffer showing the contents
>   of each highlight contain  the highlighted text.
>
>   This is not what I get from, say, EzPDF (which is what I use on Android),
>   or from highlighting from pdf-tools itself using C-c C-a h, or from
>   highlighting from Okular. The contents just gives the rectangle).
> Hummmm...
>
>
>   Because of this, when I use your code on my pdfs, I only get things
>   such as
>
> Highlight
>
> ([[pdfview:/home/ramon/Zotero-data/storage/ESHHD4KW/Frank_2015_Commentary.pdf::5][Frank_2015_Commentary]],
> 5)
>
>
>   instead of the text. Bummer! I wonder if RepliGO gives you a lot more
>   than the rest, or if I am doing something silly.
>
> I think that there is no standard way of storing the highlight contents. I
chose Repligo over EZPDF because it gives you access to the text of the
highlights!

Okular, I think, stores your annotations in its own database, rather than
in the pdf. You can (I think!) attach the annotations to the pdf from
inside Okular.  At leasts, that's what I remember from when I was looking
around.

Repligo stores the highlighted text in the "subject" field of the
annotation. It's possible that the content of the annotation is stored in
some other field, like "content".  Maybe you can try:

M-: (pdf-annot-get-annots) and look at the output in the *Messages*
buffer.  Can you see any evidence of the the text? Can you share what you
learned?

Politza and I are discussing this here:
https://github.com/politza/pdf-tools/issues/137

that might be a good place to ocntinue the conversation.


>
> - You have to call mwp/pdf-multi-extract on each file/set of files. I guess
>   if I knew elisp, I'd find it trivial to iterate over a set of directories
>   and subdirectories (and do this using a cron job at night), and also
>   place everything in one single org file. Would this be something
>   reasonable to do?
>
> for sure.  My elisp sucks too but I bet someone will answer you here on
the list.


>   (This might be related to your second Todo)
>
> well, wasn't what I was planning but would still be useful.

>
> - I know nothing about how it works, and it does not use pdf-tools, but in
>   your first Todo you mention: "extend the pdfview link type (in
>   org-pdfview) to permit me to specify the precise location of an
>   annotation,".  PDF.js (https://mozilla.github.io/pdf.js/), which is
>   used for instance by zotfile (http://zotfile.com/) does that and it
> works
>   out of the box with Okular (but I've not been able to get it to work with
>   pdftools).
>
> Until I found pdf-tools, I had planned to write a node wrapper for pdf.js
and grab the annotations that way.  But I don't really know how to do that,
so this turned out to be easier :-)

Anyway, I've judated the post, and it's now possible to create links to
individualt annotations, though you will have to use my updated version of
org-pdfview, until/unless Markus accepts my patch.

>
> - In case it matters, I have somewhat similar modus operandi.  I do a lot
>   of PDF reading, including note-taking and highlighting, in android
>   tablets ---I use EzPDF, which also embeds the notes in the PDF. I have a
>   cron job that extracts all the highlights and annotations of all the PDFs
>   and places them in a single org file. The kludge is explained here:
>
> https://github.com/rdiaz02/Adios_Mendeley#extracting-all-pdf-annotations-and-placing-them-in-an-org-mode-file
>   The truth is I use two mechanisms for PDF annotation and highlighting
>   extraction, since none is fully satisfactory to me, but the one that uses
>   Ruby (i.e., that does not depend on poppler) is able to actually extract
>   the text of the highlights.
>
> ah, man, that looks really cool and I'm sorry I didn't know about it
earlier! I haven't read through your whole document but looks like there's
a lot useful stuff there.



>
> Best, and thanks again for sharing,
>
> you're welcome & thank you!
m

[-- Attachment #2: Type: text/html, Size: 6761 bytes --]

Re: Org Mode and PDF Notes!

[Matt Lundin]

Matt Price <moptop99@gmail.com> writes:
>
> I think that there is no standard way of storing the highlight
> contents. I chose Repligo over EZPDF because it gives you access to
> the text of the highlights!

> Okular, I think, stores your annotations in its own database, rather
> than in the pdf. You can (I think!) attach the annotations to the pdf
> from inside Okular. At leasts, that's what I remember from when I was
> looking around.

> Repligo stores the highlighted text in the "subject" field of the
> annotation. It's possible that the content of the annotation is stored
> in some other field, like "content". Maybe you can try:

Repligo (as you say) grabs the text in the highlighted region and adds
it to the "subject" connected to the highlight. I think it is rather
unique in doing so. AFAIK, Okular, pdf-tools, or other android tools
(mupdf, xodo reader) do not not do this. In other words, it is thanks of
Repligo's previous extraction that the function on Matt's blog is able
get the text of the highlighted region.

Otherwise, you would need a means of going back into the region
highlighted and extracting the embedded text. Pdf-tools doesn't
currently extract the text, but it might not be that difficult to modify
it to do so, since it already has the bounds of the highlighted region
and can select and copy text embedded in the pdf.

Best,
Matt

Re: Org Mode and PDF Notes!

[Ramon Diaz-Uriarte]

On Wed, 11-11-2015, at 21:33, Matt Price <moptop99@gmail.com> wrote:
>>
>>
>>   instead of the text. Bummer! I wonder if RepliGO gives you a lot more
>>   than the rest, or if I am doing something silly.
>>
>> I think that there is no standard way of storing the highlight contents. I
> chose Repligo over EZPDF because it gives you access to the text of the
> highlights!


I'll try to see if I can get repligo (I had it a few years ago)

>
> Okular, I think, stores your annotations in its own database, rather than
> in the pdf. You can (I think!) attach the annotations to the pdf from
> inside Okular.  At leasts, that's what I remember from when I was looking
> around.

Actually, Okular stores the annotations in the PDF itself if you do "Save
As". (It still keeps an internal db, but I never use it anymore). It is
easy to check by doing that and then opening the file with another reader
in another machine (e.g., in an android).



>
> Repligo stores the highlighted text in the "subject" field of the
> annotation. It's possible that the content of the annotation is stored in
> some other field, like "content".  Maybe you can try:
>
> M-: (pdf-annot-get-annots) and look at the output in the *Messages*
> buffer.  Can you see any evidence of the the text? Can you share what you
> learned?

Nope, no evidence of the text. I get things such as

(((buffer . #<buffer Frank_2015_Commentary.pdf>) (page . 13) (edges
0.113553 0.31717 0.868657 0.361746) (type . highlight) (id . annot-13-0)
(flags . 4) (color . "#ffff00") (contents . "") (modified 22081 45188)
(label . "TF201") (subject . "Highlight") (opacity . 1.0) ...)


so we get the location of the highlight (and its properties), but not the
textual contents. And this is the case whether I make the annotation with
EzPDF or Okular or, for that matter, with pdf-tools itself.

So it seems RepliGO is actually giving you a lot more by default :-)


>
> Politza and I are discussing this here:
> https://github.com/politza/pdf-tools/issues/137
>
> that might be a good place to ocntinue the conversation.
>

I'll do. In the meantime, I think this is a limitation coming from
poppler. Other people have mentioned similar things (e.g.,
http://coda.caseykuhlman.com/entries/2014/pdf-extract.html) and using other
tools that depend on poppler (such as Leela:
https://github.com/TrilbyWhite/Leela) also will not give us the text
itself. 



>>
>> Until I found pdf-tools, I had planned to write a node wrapper for pdf.js
> and grab the annotations that way.  But I don't really know how to do that,
> so this turned out to be easier :-)
>
> Anyway, I've judated the post, and it's now possible to create links to
> individualt annotations, though you will have to use my updated version of
> org-pdfview, until/unless Markus accepts my patch.


I just updated packages, and things are working perfectly: I am jumping to
the page and location.



Thanks,


R.




-- 
Ramon Diaz-Uriarte
Department of Biochemistry, Lab B-25
Facultad de Medicina
Universidad Autónoma de Madrid 
Arzobispo Morcillo, 4
28029 Madrid
Spain

Phone: +34-91-497-2412

Email: rdiaz02@gmail.com
       ramon.diaz@iib.uam.es

http://ligarto.org/rdiaz

Re: Org Mode and PDF Notes!

[Matt Price]

[-- Attachment #1: Type: text/plain, Size: 3669 bytes --]

On Thu, Nov 12, 2015 at 7:23 AM, Ramon Diaz-Uriarte <rdiaz02@gmail.com>
wrote:

>
>
>
> On Wed, 11-11-2015, at 21:33, Matt Price <moptop99@gmail.com> wrote:
> >>
> >>
> >>   instead of the text. Bummer! I wonder if RepliGO gives you a lot more
> >>   than the rest, or if I am doing something silly.
> >>
> >> I think that there is no standard way of storing the highlight
> contents. I
> > chose Repligo over EZPDF because it gives you access to the text of the
> > highlights!
>
>
> I'll try to see if I can get repligo (I had it a few years ago)
>
> >
> > Okular, I think, stores your annotations in its own database, rather than
> > in the pdf. You can (I think!) attach the annotations to the pdf from
> > inside Okular.  At leasts, that's what I remember from when I was looking
> > around.
>
> Actually, Okular stores the annotations in the PDF itself if you do "Save
> As". (It still keeps an internal db, but I never use it anymore). It is
> easy to check by doing that and then opening the file with another reader
> in another machine (e.g., in an android).
>
>
> my bad, thx.

>
> >
> > Repligo stores the highlighted text in the "subject" field of the
> > annotation. It's possible that the content of the annotation is stored in
> > some other field, like "content".  Maybe you can try:
> >
> > M-: (pdf-annot-get-annots) and look at the output in the *Messages*
> > buffer.  Can you see any evidence of the the text? Can you share what you
> > learned?
>
> Nope, no evidence of the text. I get things such as
>
> (((buffer . #<buffer Frank_2015_Commentary.pdf>) (page . 13) (edges
> 0.113553 0.31717 0.868657 0.361746) (type . highlight) (id . annot-13-0)
> (flags . 4) (color . "#ffff00") (contents . "") (modified 22081 45188)
> (label . "TF201") (subject . "Highlight") (opacity . 1.0) ...)
>
>
> so we get the location of the highlight (and its properties), but not the
> textual contents. And this is the case whether I make the annotation with
> EzPDF or Okular or, for that matter, with pdf-tools itself.
>
> So it seems RepliGO is actually giving you a lot more by default :-)
>
>
Try replacing

(text (assoc-default 'subject annot))

with

(text (pdf-info-gettext page (assoc-default 'edges annot)))


in the lambda function in pdf-annot-markups-as-org-text.  This will fail on
cropped pdfs if you have added highlights using the most recent pdf-tools,
which stores negative values in the 'edges field, but I've found it works
otherwise.  I'd love to hear if it works for you too. (I know you're
following the relevant bug report on the pdf-tools github repo).


>
> >
> > Politza and I are discussing this here:
> > https://github.com/politza/pdf-tools/issues/137
> >
> > that might be a good place to ocntinue the conversation.
> >
>
> I'll do. In the meantime, I think this is a limitation coming from
> poppler. Other people have mentioned similar things (e.g.,
> http://coda.caseykuhlman.com/entries/2014/pdf-extract.html) and using
> other
> tools that depend on poppler (such as Leela:
> https://github.com/TrilbyWhite/Leela) also will not give us the text
> itself.
>
>
>
> >>
> >> Until I found pdf-tools, I had planned to write a node wrapper for
> pdf.js
> > and grab the annotations that way.  But I don't really know how to do
> that,
> > so this turned out to be easier :-)
> >
> > Anyway, I've judated the post, and it's now possible to create links to
> > individualt annotations, though you will have to use my updated version
> of
> > org-pdfview, until/unless Markus accepts my patch.
>
>
> I just updated packages, and things are working perfectly: I am jumping to
> the page and location.
>
>
>
> Thanks,
>
>
> R.
>
>
>
>
>

[-- Attachment #2: Type: text/html, Size: 5449 bytes --]

Re: Org Mode and PDF Notes!

[Ramon Diaz-Uriarte]

On Thu, 12-11-2015, at 14:11, Matt Price <moptop99@gmail.com> wrote:
>> >
>> > M-: (pdf-annot-get-annots) and look at the output in the *Messages*
>> > buffer.  Can you see any evidence of the the text? Can you share what you
>> > learned?
>>
>> Nope, no evidence of the text. I get things such as
>>
>> (((buffer . #<buffer Frank_2015_Commentary.pdf>) (page . 13) (edges
>> 0.113553 0.31717 0.868657 0.361746) (type . highlight) (id . annot-13-0)
>> (flags . 4) (color . "#ffff00") (contents . "") (modified 22081 45188)
>> (label . "TF201") (subject . "Highlight") (opacity . 1.0) ...)
>>
>>
>> so we get the location of the highlight (and its properties), but not the
>> textual contents. And this is the case whether I make the annotation with
>> EzPDF or Okular or, for that matter, with pdf-tools itself.
>>
>> So it seems RepliGO is actually giving you a lot more by default :-)
>>
>>
> Try replacing
>
> (text (assoc-default 'subject annot))
>
> with
>
> (text (pdf-info-gettext page (assoc-default 'edges annot)))
>
>
> in the lambda function in pdf-annot-markups-as-org-text.  This will fail on
> cropped pdfs if you have added highlights using the most recent pdf-tools,
> which stores negative values in the 'edges field, but I've found it works
> otherwise.  I'd love to hear if it works for you too. (I know you're
> following the relevant bug report on the pdf-tools github repo).


It works! I get the highlighted text (I have just tried with a pdf I
highlighted with EzPDF, the one that, above, was giving just the location
and properties). And it still does the correct thing in your pdf (the one
from repligo).

Awesome! Thanks.


R.





-- 
Ramon Diaz-Uriarte
Department of Biochemistry, Lab B-25
Facultad de Medicina
Universidad Autónoma de Madrid 
Arzobispo Morcillo, 4
28029 Madrid
Spain

Phone: +34-91-497-2412

Email: rdiaz02@gmail.com
       ramon.diaz@iib.uam.es

http://ligarto.org/rdiaz

Re: Org Mode and PDF Notes!

[Matt Lundin]

Ramon Diaz-Uriarte <rdiaz02@gmail.com> writes:

>
> so we get the location of the highlight (and its properties), but not the
> textual contents. And this is the case whether I make the annotation with
> EzPDF or Okular or, for that matter, with pdf-tools itself.
>
> So it seems RepliGO is actually giving you a lot more by default :-)
>
>>
>> Politza and I are discussing this here:
>> https://github.com/politza/pdf-tools/issues/137
>>
>> that might be a good place to ocntinue the conversation.
>>
>
> I'll do. In the meantime, I think this is a limitation coming from
> poppler. Other people have mentioned similar things (e.g.,
> http://coda.caseykuhlman.com/entries/2014/pdf-extract.html) and using other
> tools that depend on poppler (such as Leela:
> https://github.com/TrilbyWhite/Leela) also will not give us the text
> itself. 

I don't think this is a limitation of poppler so much as the way that
pdf annotations work. Typically, the subject/text field is not populated
by the text of the highlighted region. Rather, a highlight annotation
specifies bounds, color, style, etc. Basically what Repligo does (I
wouldn't recommend using it, as it is closed source and severely out of
date) is to grab the text *at the time of highlighting* and add it to
the notes field. I don't know of any other annotation tool that does the
same thing. Applications built on poppler could do it, though they
currently do not.

For extracting the text of highlighted regions *after the fact*, I've
had good luck with this script that relies on the pdf-reader gem for
ruby:

https://gist.github.com/danlucraft/5277732

Matt

Re: Org Mode and PDF Notes!

[Matt Price]

[-- Attachment #1: Type: text/plain, Size: 1565 bytes --]

On Thu, Nov 12, 2015 at 9:28 AM, Matt Lundin <mdl@imapmail.org> wrote:

> Ramon Diaz-Uriarte <rdiaz02@gmail.com> writes:
> >
> > I'll do. In the meantime, I think this is a limitation coming from
> > poppler. Other people have mentioned similar things (e.g.,
> > http://coda.caseykuhlman.com/entries/2014/pdf-extract.html) and using
> other
> > tools that depend on poppler (such as Leela:
> > https://github.com/TrilbyWhite/Leela) also will not give us the text
> > itself.
>
> I don't think this is a limitation of poppler so much as the way that
> pdf annotations work. Typically, the subject/text field is not populated
> by the text of the highlighted region. Rather, a highlight annotation
> specifies bounds, color, style, etc. Basically what Repligo does (I
> wouldn't recommend using it, as it is closed source and severely out of
> date) is to grab the text *at the time of highlighting* and add it to
> the notes field. I don't know of any other annotation tool that does the
> same thing. Applications built on poppler could do it, though they
> currently do not.
>
> For extracting the text of highlighted regions *after the fact*, I've
> had good luck with this script that relies on the pdf-reader gem for
> ruby:
>
> https://gist.github.com/danlucraft/5277732
>
> This looks interesting. It searches for file "./markup_receiver", but
doesn't provide that file, which does not appear to be a gem.  Any hints?

With politza's help am getting close to being able to extract annotation
text from within pdf-tools, but am not quite there yet.


> Matt
>

[-- Attachment #2: Type: text/html, Size: 2615 bytes --]

Re: Org Mode and PDF Notes!

[Ramon Diaz-Uriarte]

On Thu, 12-11-2015, at 23:52, Matt Price <moptop99@gmail.com> wrote:
> On Thu, Nov 12, 2015 at 9:28 AM, Matt Lundin <mdl@imapmail.org> wrote:
>
>> Ramon Diaz-Uriarte <rdiaz02@gmail.com> writes:
>> >
>> > I'll do. In the meantime, I think this is a limitation coming from
>> > poppler. Other people have mentioned similar things (e.g.,
>> > http://coda.caseykuhlman.com/entries/2014/pdf-extract.html) and using
>> other
>> > tools that depend on poppler (such as Leela:
>> > https://github.com/TrilbyWhite/Leela) also will not give us the text
>> > itself.
>>
>> I don't think this is a limitation of poppler so much as the way that
>> pdf annotations work. Typically, the subject/text field is not populated
>> by the text of the highlighted region. Rather, a highlight annotation
>> specifies bounds, color, style, etc. Basically what Repligo does (I
>> wouldn't recommend using it, as it is closed source and severely out of
>> date) is to grab the text *at the time of highlighting* and add it to
>> the notes field. I don't know of any other annotation tool that does the
>> same thing. Applications built on poppler could do it, though they
>> currently do not.
>>
>> For extracting the text of highlighted regions *after the fact*, I've
>> had good luck with this script that relies on the pdf-reader gem for
>> ruby:
>>
>> https://gist.github.com/danlucraft/5277732
>>
>> This looks interesting. It searches for file "./markup_receiver", but
> doesn't provide that file, which does not appear to be a gem.  Any hints?


I think I got it from

https://www.omniref.com/github/danlucraft/pyranine/HEAD/files/lib/pyranine/markup_receiver.rb

>
> With politza's help am getting close to being able to extract annotation
> text from within pdf-tools, but am not quite there yet.


Neat!


R.


>
>
>> Matt
>>


-- 
Ramon Diaz-Uriarte
Department of Biochemistry, Lab B-25
Facultad de Medicina
Universidad Autónoma de Madrid 
Arzobispo Morcillo, 4
28029 Madrid
Spain

Phone: +34-91-497-2412

Email: rdiaz02@gmail.com
       ramon.diaz@iib.uam.es

http://ligarto.org/rdiaz

Re: Org Mode and PDF Notes!

[Ramon Diaz-Uriarte]

On Thu, 12-11-2015, at 15:28, Matt Lundin <mdl@imapmail.org> wrote:
> Ramon Diaz-Uriarte <rdiaz02@gmail.com> writes:
>
>>
>> so we get the location of the highlight (and its properties), but not the
>> textual contents. And this is the case whether I make the annotation with
>> EzPDF or Okular or, for that matter, with pdf-tools itself.
>>
>> So it seems RepliGO is actually giving you a lot more by default :-)
>>
>>>
>>> Politza and I are discussing this here:
>>> https://github.com/politza/pdf-tools/issues/137
>>>
>>> that might be a good place to ocntinue the conversation.
>>>
>>
>> I'll do. In the meantime, I think this is a limitation coming from
>> poppler. Other people have mentioned similar things (e.g.,
>> http://coda.caseykuhlman.com/entries/2014/pdf-extract.html) and using other
>> tools that depend on poppler (such as Leela:
>> https://github.com/TrilbyWhite/Leela) also will not give us the text
>> itself. 
>
> I don't think this is a limitation of poppler so much as the way that
> pdf annotations work. Typically, the subject/text field is not populated
> by the text of the highlighted region. Rather, a highlight annotation
> specifies bounds, color, style, etc. Basically what Repligo does (I
> wouldn't recommend using it, as it is closed source and severely out of
> date) is to grab the text *at the time of highlighting* and add it to
> the notes field. I don't know of any other annotation tool that does the
> same thing. Applications built on poppler could do it, though they
> currently do not.

I stand corrected. You are right; sorry for the sloppiness in the wording
and ideas.


>
> For extracting the text of highlighted regions *after the fact*, I've
> had good luck with this script that relies on the pdf-reader gem for
> ruby:
>
> https://gist.github.com/danlucraft/5277732

That is also what I use for extracting the text from the highlighted
regions.


R.


>
> Matt


-- 
Ramon Diaz-Uriarte
Department of Biochemistry, Lab B-25
Facultad de Medicina
Universidad Autónoma de Madrid 
Arzobispo Morcillo, 4
28029 Madrid
Spain

Phone: +34-91-497-2412

Email: rdiaz02@gmail.com
       ramon.diaz@iib.uam.es

http://ligarto.org/rdiaz

Re: Org Mode and PDF Notes!

[Karl Voit]

* Matt Price <moptop99@gmail.com> wrote:
> --047d7b41411eff32e6052444d409
> Content-Type: text/plain; charset=UTF-8
>
> I've just written up a post on my workflow for PDF's Since my blog has, I
> think, a readership of 0 (surely there's a way to get emacsers to follow
> me? ah well), I will post a link here in the hopes that someone will be
> interested:
>
> http://matt.hackinghistory.ca/2015/11/11/note-taking-with-pdf-tools/

Thanks for your blog post and the URL!

I did something very similar a couple of years ago while writing my
PhD thesis:
https://github.com/novoid/extract_pdf_annotations_to_orgmode

And I also have a blog which I use for Emacs/Org-mode stuff from
time to time: http://karl-voit.at/

This reminds me of implementing a tag-based RSS/Atom-feed for
Emacs-related stuff to my blogging system ...

-- 
mail|git|SVN|photos|postings|SMS|phonecalls|RSS|CSV|XML to Org-mode:
       > get Memacs from https://github.com/novoid/Memacs <

https://github.com/novoid/extract_pdf_annotations_to_orgmode + more on github

Re: Org Mode and PDF Notes!

[Peter Davis]

Matt Price <moptop99@gmail.com> writes:

> I've just written up a post on my workflow for PDF's Since my blog
> has, I think, a readership of 0 (surely there's a way to get emacsers
> to follow me? ah well), I will post a link here in the hopes that
> someone will be interested:
>
> http://matt.hackinghistory.ca/2015/11/11/note-taking-with-pdf-tools/

Thanks, Matt. This looks interesting. I've run into poppler before, but was not familiar with pdf-tools. I'll have to take a closer
look.

-pd

Re: Org Mode and PDF Notes!

[Matti Minkkinen]

Ramon Diaz-Uriarte writes:

> On Thu, 12-11-2015, at 14:11, Matt Price gmail.com> wrote:
> It works! I get the highlighted text (I have just tried with a pdf I
> highlighted with EzPDF, the one that, above, was giving just the location
> and properties). And it still does the correct thing in your pdf (the one
> from repligo).
> Awesome! Thanks.
> R.
>
> link

I also have a strong interest in getting PDF annotations from Android to
org-mode notes easily, so I've been following this thread with great
interest.

Very recently, on 11 November, there were a couple of commits to this
pdf-tools fork which also let you export highlighted text:
https://github.com/pinguim06/pdf-tools

Matt Price already referred to the pull request from that fork, but
those commits are more recent than the original pull request. I think it
uses the same principle as the solution mentioned here but maybe it's
more elaborate. I don't know enough elisp to confirm that.

--
Matti

Re: Org Mode and PDF Notes!

[Ramon Diaz-Uriarte]

Thanks for the pointer.


On Fri, 13-11-2015, at 09:04, Matti Minkkinen <matti.minkkinen@iki.fi> wrote:
> Ramon Diaz-Uriarte writes:
>
>> On Thu, 12-11-2015, at 14:11, Matt Price gmail.com> wrote:
>> It works! I get the highlighted text (I have just tried with a pdf I
>> highlighted with EzPDF, the one that, above, was giving just the location
>> and properties). And it still does the correct thing in your pdf (the one
>> from repligo).
>> Awesome! Thanks.
>> R.
>>
>> link
>
> I also have a strong interest in getting PDF annotations from Android to
> org-mode notes easily, so I've been following this thread with great
> interest.
>
> Very recently, on 11 November, there were a couple of commits to this
> pdf-tools fork which also let you export highlighted text:
> https://github.com/pinguim06/pdf-tools
>
> Matt Price already referred to the pull request from that fork, but
> those commits are more recent than the original pull request. I think it
> uses the same principle as the solution mentioned here but maybe it's
> more elaborate. I don't know enough elisp to confirm that.


-- 
Ramon Diaz-Uriarte
Department of Biochemistry, Lab B-25
Facultad de Medicina
Universidad Autónoma de Madrid 
Arzobispo Morcillo, 4
28029 Madrid
Spain

Phone: +34-91-497-2412

Email: rdiaz02@gmail.com
       ramon.diaz@iib.uam.es

http://ligarto.org/rdiaz