Re: [Feature] add a new org-attach dispatcher command to offline save web page

[Ihor Radchenko <yantar92@gmail.com>]

> My uneasiness has more to do with the specificity of the dependence on
> monolith and the way that is hard-coded into the patch. When it comes to
> patches, I think priority should go to those that are configurable,
> accessible, and useful for everyone as opposed to those that have
> hard-coded work-flows or highly-specific user configurations.

Agree. Though I can see a use of having monolith as one of the options
to help people discover what kind of tools they can use. I personally
had a hard time finding command-line cli like monolith. Actually, it is
the first time I heard about some offline tool handling js without a
need to write python or ruby code. 

> The question is: which functionality? A simple downloading tool or a
> full archival tool? Achieving similar functionality to org-board or
> monolith would a big task, since they aim to download an archival
> version of a webpage (including all resources). 

My view on this is bare-bones download, in a spirit of org-attach
itself. There is already 'url method in org-attach-attach, but it is
hard-coded to url-retrieve-synchronously. It would be handy if user
could configure alternative retrievers (like monolith, wget, curl, or
some user-defined function).

Note that monolith does not crawl the website. It only collects
everything needed to show the page as you see it in browser into single
html file. This behaviour is what one expects to obtain when saving a
full web-page from browser.

> In addition, with
> archiving you also quickly run into the complexity of versioning based
> on time archived. 

I guess that org-attach-git can be used for versioning, but I don't
think that versioning is within scope of this patch. Monolith does not
even support versioning.

> There's also the challenge of mapping the downloaded
> files to metadata (specifically the original url). Org-board currently
> handles both of these very well.

org-board is a great package, but it is not built-in. I do not think
that all the org-board functionality needs to be included into
org-attach. At least not within scope of this patch as I understand it.

Best,
Ihor

Matthew Lundin <mdl@imapmail.org> writes:

> Ihor Radchenko <yantar92@gmail.com> writes:
>
>> It does not mean that attaching URL directly is not worth including
>> into org. This sounds pretty common use case, especially considering
>> the number of packages providing similar feature. You mentioned
>> org-board, but there is also org-download and org-web-tools.
>
> My uneasiness has more to do with the specificity of the dependence on
> monolith and the way that is hard-coded into the patch. When it comes to
> patches, I think priority should go to those that are configurable,
> accessible, and useful for everyone as opposed to those that have
> hard-coded work-flows or highly-specific user configurations.
>
>> I agree that monolith is completely uncommon tool and I would not expect
>> the majority of users to have it installed, but the same functionality
>> utilising built-in url.el (as a default) should be acceptable.
>
> The question is: which functionality? A simple downloading tool or a
> full archival tool? Achieving similar functionality to org-board or
> monolith would a big task, since they aim to download an archival
> version of a webpage (including all resources). In addition, with
> archiving you also quickly run into the complexity of versioning based
> on time archived. There's also the challenge of mapping the downloaded
> files to metadata (specifically the original url). Org-board currently
> handles both of these very well.
>
> I suppose there would be a few options depending on what the aims are:
>
> 1. At the simple end, include little more than than a quick and dirty
>    way of downloading a single resource (html, pdf, jpeg) using url.el
>    or wget (or optionally, monolith) and putting that in the attachment
>    folder. Those who want full archiving of all resources could use
>    other tools like org-board or org-web-tools.
>
> 2. At the (much) more complex end, it would be to code out a robust
>    archiving solution on top of url.el or wget.
>
> 3. Another, possibly simpler option... Add a command to the dispatcher
>    that allows the user to invoke a custom function that is called with
>    the attachment directory as the default-directory. This would enable
>    more end-user flexibility, such as the ability to use
>    wkhtmtoimage/wkhtmltopdf, monolith, phantom.js, archive.is, etc.
>
> Best,
>
> Matt

-- 
Ihor Radchenko,
PhD,
Center for Advancing Materials Performance from the Nanoscale (CAMP-nano)
State Key Laboratory for Mechanical Behavior of Materials, Xi'an Jiaotong University, Xi'an, China
Email: yantar92@gmail.com, ihor_radchenko@alumni.sutd.edu.sg