Include does not work when doing org-export-as-org

Include does not work when doing org-export-as-org

[Henry Hirsch]

Hello everyone,

I am trying to export an org-file with includes.
When I export as PDF all includes work but when
I export as org-file the includes are getting ignored.

I am on org-mode 7.6.

Hope somebody knows how to fix this.

Sincerly yours,
Henry
 

Re: Include does not work when doing org-export-as-org

[Nick Dokos]

Henry Hirsch <henry.hirsch@adition.com> wrote:

> Hello everyone,
> 
> I am trying to export an org-file with includes.
> When I export as PDF all includes work but when
> I export as org-file the includes are getting ignored.
> 
> I am on org-mode 7.6.
> 
> Hope somebody knows how to fix this.
> 

Sorry to be a grouch, but can you *please* provide an example?  If you
have a small toy example, provide that. Otherwise, cut down your example
to a minimal case that exhibits the problem (what Seb calls an ECM).

You see, if you provide an example that I can cut-n-past and test in a
few seconds, I probably will do it. If not, I have to *create* an
example based on your instructions: that not only takes longer (which
cuts down the number of people who might be willing to help), but it is
also error-prone - if I misunderstand your instructions, then I end up
wasting my time testing something that is not the problem, or providing
wrong information and wasting the time of every *other* person who might
be looking at the problem.

And please don't take it personally - this is a plea to *everybody*
who reports a problem:

    o if you report odd behavior, please provide an ECM.
    o if you report an error, please provide a backtrace.

Thanks,
Nick

Re: Include does not work when doing org-export-as-org

[Henry Hirsch]

On Tue, Oct 11, 2011 at 09:41:43AM -0400, Nick Dokos wrote:
> Sorry to be a grouch, but can you *please* provide an example?  
Content of a.org:
#+INCLUDE: "b.org" 

Content of b.org:
Hello World

Content of a_source.org after export:

Thanks,
Henry

Re: Include does not work when doing org-export-as-org

[Henry Hirsch]

On Tue, Oct 11, 2011 at 03:53:00PM +0200, Henry Hirsch wrote:
I forgot something:

> Content of a_source.org after export:

is actually:
#+INCLUDE "b.org"

Where I expected it to be "Hello World".
 
Thanks,
Henry

Re: Include does not work when doing org-export-as-org

[Nick Dokos]

Henry Hirsch <henry.hirsch@adition.com> wrote:

> Hello everyone,
> 
> I am trying to export an org-file with includes.
> When I export as PDF all includes work but when
> I export as org-file the includes are getting ignored.
> 

What does "export as org-file" mean? What do you do exactly?

FWIW, I tried with your minimal example (thanks!), and I can export to
ascii/latex/PDF/HTML with

	C-c C-e A (or D or L or H)

They all include "Hello world" in the output.

I downgraded to org 7.6 as well: same result.

Nick

> I am on org-mode 7.6.
> 
> Hope somebody knows how to fix this.
> 
> Sincerly yours,
> Henry
>  
> 

Re: Include does not work when doing org-export-as-org

[Nick Dokos]

Nick Dokos <nicholas.dokos@hp.com> wrote:

> FWIW, I tried with your minimal example (thanks!), and I can export to
> ascii/latex/PDF/HTML with
> 
> 	C-c C-e A (or D or L or H)
> 

I meant:

        C-c C-e A (or L or d or H)

Nick

Re: Include does not work when doing org-export-as-org

[Nick Dokos]

Nick Dokos <nicholas.dokos@hp.com> wrote:

> Henry Hirsch <henry.hirsch@adition.com> wrote:
> 
> > Hello everyone,
> > 
> > I am trying to export an org-file with includes.
> > When I export as PDF all includes work but when
> > I export as org-file the includes are getting ignored.
> > 
> 
> What does "export as org-file" mean? What do you do exactly?
> 

Never mind: I didn't read the subject line.

Sorry,
Nick

Re: Include does not work when doing org-export-as-org

[Henry Hirsch]

On Tue, Oct 11, 2011 at 10:09:49AM -0400, Nick Dokos wrote:
> Henry Hirsch <henry.hirsch@adition.com> wrote:
> 
> > Hello everyone,
> > 
> > I am trying to export an org-file with includes.
> > When I export as PDF all includes work but when
> > I export as org-file the includes are getting ignored.
> > 
> 
> What does "export as org-file" mean? What do you do exactly?
> 
> FWIW, I tried with your minimal example (thanks!), and I can export to
> ascii/latex/PDF/HTML with

Yes. As I mentioned none of the above is producing any problem for me.

As I explicitly stated only org-export-as-org produces the unwanted output.

So when I say export as org I mean I hit M-x and then type "org-export-as-org"
without the quotes and then enter. Then I get a new file in the same directory
with the name of a-source.org, which you can open by hitting C-x C-f then typing
"a-source.org" without the quotes and then enter. 

Now if both our systems react the same, you should see the Line #+INCLUDE "b.org".
Though I expected it to say "Hello World".

Sincerly yours,
Henry

-- 
Mit freundlichem Gruß,

Henry Hirsch <henry.hirsch@adition.com>

--
Entwickler

---------------------------------------------------------------------

ADITION technologies AG
Basler Landstraße 8
79111 Freiburg

http://www.adition.com

T +49 / (0)761 / 88147 - 30
F +49 / (0)761 / 88147 - 77
Support +49 /(0)1805 - ADITION

(Festnetzpreis 14 ct/min; Mobilfunkpreise maximal 42 ct/min)    

Eingetragen beim Amtsgericht Düsseldorf unter HRB 54076
Vorstände: Andreas Kleiser, Jörg Klekamp, Tihomir Perkovic, Marcus Schlüter
Aufsichtsratsvorsitzender: Rechtsanwalt Daniel Raimer
UStIDNr.: DE 218 858 434

Re: Include does not work when doing org-export-as-org

[Nick Dokos]

Nick Dokos <nicholas.dokos@hp.com> wrote:

> Nick Dokos <nicholas.dokos@hp.com> wrote:
> 
> > Henry Hirsch <henry.hirsch@adition.com> wrote:
> > 
> > > Hello everyone,
> > > 
> > > I am trying to export an org-file with includes.
> > > When I export as PDF all includes work but when
> > > I export as org-file the includes are getting ignored.
> > > 
> > 
> > What does "export as org-file" mean? What do you do exactly?
> > 
> 
> Never mind: I didn't read the subject line.
> 

C-h f org-export-as-org says:

,----
| org-export-as-org is an interactive Lisp function in `org-exp.el'.
| 
| (org-export-as-org ARG &optional HIDDEN EXT-PLIST TO-BUFFER BODY-ONLY
| PUB-DIR)
| 
| Make a copy with not-exporting stuff removed.
| The purpose of this function is to provide a way to export the source
| Org file of a webpage in Org format, but with sensitive and/or irrelevant
| stuff removed.  This command will remove the following:
| 
| - archived trees (if the variable `org-export-with-archived-trees' is nil)
| - comment blocks and trees starting with the COMMENT keyword
| - only trees that are consistent with `org-export-select-tags'
|   and `org-export-exclude-tags'.
`----

It is not a general purpose exporter. It is supposed to allow you to post
org examples on web pages. I presume it is used when exporting to Worg.

In particular, I assume it does not process includes, because they need
to be preserved in examples.

Nick

Re: Include does not work when doing org-export-as-org

[Henry Hirsch]

On Tue, Oct 11, 2011 at 10:27:56AM -0400, Nick Dokos wrote:
 
> In particular, I assume it does not process includes, because they need
> to be preserved in examples.
> 
> Nick
 
So do you think it is reasonable that org-export-as-org is the only mode
of export which is exhibiting a different behaviour?

For my part I think it is not. 
I used to love org-mode above everything.
But this just leaves me not amused.

Henry

Re: Include does not work when doing org-export-as-org

[Nick Dokos]

Henry Hirsch <henry.hirsch@adition.com> wrote:

> On Tue, Oct 11, 2011 at 10:27:56AM -0400, Nick Dokos wrote:
>  
> > In particular, I assume it does not process includes, because they need
> > to be preserved in examples.
> > 
> > Nick
>  
> So do you think it is reasonable that org-export-as-org is the only mode
> of export which is exhibiting a different behaviour?
> 
It's certainly reasonable: it does not pretend to be a general-purpose
exporter. It may be misnamed though: if org-export-as-org is only used
in batch processing of e.g. Worg, then it can be renamed to something more
obscure and made non-interactive, so that it would not confuse the unwary.

One could imagine a general-purpose org-export-as-org that processes
includes, but I'm not sure what else it would do: just copy it's input
to its output mostly - other than processing includes, is there anything
else that it should do?

> For my part I think it is not.  I used to love org-mode above
> everything.  But this just leaves me not amused.
> 

Well, there are a couple of options:

      o org is very much a scratch-your-itch project - so you can
      certainly go ahead and implement what you want.

      o explain your use case: if it is compelling enough, somebody
      might be motivated to implement what you ask for (but you need
      to specify it exactly - and if somebody else implements it, be
      prepared to compromise...)


But it is certainly *not* the case that org is some ivory tower project
that exists for theoretical purity only: there are hacks, work-arounds,
inconsistencies and bugs.

Nick



      

Re: Include does not work when doing org-export-as-org

[Henry Hirsch]

On Tue, Oct 11, 2011 at 10:57:58AM -0400, Nick Dokos wrote:
>       o org is very much a scratch-your-itch project - so you can
>       certainly go ahead and implement what you want.

I get it. If I want something fixed I gotta do it myself.

Have a nice day, and thank you for sobering me up.

Re: Include does not work when doing org-export-as-org

[Nick Dokos]

Henry Hirsch <henry.hirsch@adition.com> wrote:

> On Tue, Oct 11, 2011 at 10:57:58AM -0400, Nick Dokos wrote:
> >       o org is very much a scratch-your-itch project - so you can
> >       certainly go ahead and implement what you want.
> 
> I get it. If I want something fixed I gotta do it myself.
> 
> Have a nice day, and thank you for sobering me up.
> 

I'm sorry I offended you: that was not my intention at all.
But the above is certainly a misrepresentation of what I said.
Let me repeat:

<quote>
... there are a couple of options:

      o org is very much a scratch-your-itch project - so you can
      certainly go ahead and implement what you want.

      o explain your use case: if it is compelling enough, somebody
      might be motivated to implement what you ask for (but you need
      to specify it exactly - and if somebody else implements it, be
      prepared to compromise...)

</quote>

A lot of people do follow the first way - many follow the second.
You are certainly free to propose some other way to proceed.

So far what you have done is identified it a behavior that does not
meet your expectations. It may be considered a bug or it may be not.
If the "bug" is fixed, it would break an existing use case (exporting
org example to web pages), which was in fact the primary reason for
providing the interface (somebody scratched that itch). So why should
it be changed (or more likely: why should another interface be added)
to meet your expectations? What is *your* use case? That is not at
all clear, so far at least.

Nick

Re: Include does not work when doing org-export-as-org

[Henry Hirsch]

On Tue, Oct 11, 2011 at 11:37:54AM -0400, Nick Dokos wrote:
> What is *your* use case? That is not at all clear, so far at least.
> 
> Nick

I want to apply the don't repeat yourself (dry) principle to my org files.
For the documentation of software quality I have huge org documents
with a lot of tables which tend to repeat themselves since I 
have to document the same parameters in different environments.

To prevent having to go through a lot of lines and files every time I
add a parameter to be documented or change anything else on the table
for that matter. 

I would like to export it to an org file and have it put the included
files in that file. And while I realy don't get why org-export-as-org
does not do this. I am sure implementing this does not need to break
established api. It could very well be added as a seperate function.

If this needs any more clarification I will contribute every
information I can. Also I understand that there is not much motivation
to implement this since nobody besides me seems to mind the
absence of the functionality.

Thank you,
Henry

Re: Include does not work when doing org-export-as-org

[Nick Dokos]

Henry Hirsch <henry.hirsch@adition.com> wrote:

> On Tue, Oct 11, 2011 at 11:37:54AM -0400, Nick Dokos wrote:
> > What is *your* use case? That is not at all clear, so far at least.
> > 
> > Nick
> 
> I want to apply the don't repeat yourself (dry) principle to my org files.
> For the documentation of software quality I have huge org documents
> with a lot of tables which tend to repeat themselves since I 
> have to document the same parameters in different environments.
> 
> To prevent having to go through a lot of lines and files every time I
> add a parameter to be documented or change anything else on the table
> for that matter. 
> 
> I would like to export it to an org file and have it put the included
> files in that file. And while I realy don't get why org-export-as-org
> does not do this. I am sure implementing this does not need to break
> established api. It could very well be added as a seperate function.
> 
> If this needs any more clarification I will contribute every
> information I can. Also I understand that there is not much motivation
> to implement this since nobody besides me seems to mind the
> absence of the functionality.
> 

The following should do basically what you want, but there are several
assumptions built into it: it assumes that

1) you are visiting the buffer of the top file (a.org in your example).

2) there is a file associated with the buffer (i.e. (buffer-file-name)
   returns a path, not nil).

3) the file is named something like "/path/to/my/file.org" and the
   output file is named "/path/to/my/file.I.org"

4) the output file is sacrificial: it is *clobbered* by the function.

5) if there is narrowing in effect, only the narrowed portion is processed.

There may be other limitations as well.

Nick

--8<---------------cut here---------------start------------->8---
(defun org-to-org-handle-includes ()
  "Copy the contents of the current buffer to OUTFILE,
recursively processing #+INCLUDEs."
  (let* ((s (buffer-string))
	 (fname (buffer-file-name))
	 (ofname (format "%s.I.org" (file-name-sans-extension fname))))
    (setq result
	  (with-temp-buffer
	    (insert s)
	    (org-export-handle-include-files-recurse)
	    (buffer-string)))
    (find-file ofname)
    (delete-region (point-min) (point-max))
    (insert result)
    (save-buffer)))
--8<---------------cut here---------------end--------------->8---

Re: Include does not work when doing org-export-as-org

[Carsten Dominik]

On 11.10.2011, at 16:34, Henry Hirsch wrote:

> On Tue, Oct 11, 2011 at 10:27:56AM -0400, Nick Dokos wrote:
> 
>> In particular, I assume it does not process includes, because they need
>> to be preserved in examples.
>> 
>> Nick
> 
> So do you think it is reasonable that org-export-as-org is the only mode
> of export which is exhibiting a different behaviour?
> 
> For my part I think it is not. 
> I used to love org-mode above everything.
> But this just leaves me not amused.

What made you get out of bed today in a bad mood?

Maybe I can help to improve the communication
by explaining where the problem lies.
Both functionalities, the one that is implemented, and the
one that you want make a lot of sense.  The problem
is merely a misnaming of the function.

The reason why `org-export-to-org' was introduced was
solely to provide a way for a website to publish
the source file that was used to create a particular
website, along with the website itself.  An example can
be found on Bernt's site.

http://doc.norang.ca/org-mode.html#sec-18

What the function does is (Nick already collected this info)

- remove archived trees and commented trees
- remove trees that are excluded by special tags
- output the resulting file with -source.org extension
  and optionally produce an htmlized version of this
  file so that the web visitor can look at the file.

Processing of #include files was left out on purpose, because
that often contains stuff which you want to have out of the way
because it is tedious or complicated.  In fact, a website can choose
to also publish the source of the #include files as separate files.

What you are looking for a mechanism to expand more directives,
in order to produce another org file.  You want (IIUC) a preprocessor
much like a C compiler preprocessor.  However, it is not
automatic what this should do and what it should not do.
For example

 - do you think it should expand macros?
 - oo you think it should process babel blocks and insert the
   evaluation results, as the other exporters do?
 - do you think it should process links into a standard format
   and pre-expand link searches?

I suspect (but that is a guess, based on the description
of your use-case) that your answer would be No to all
these questions.  Yet someone else could come in and
take issue with the absence of this processing, with all
the same right as you have to wonder about the absence
of #include processing.  It all depends and what you
actually want to achieve, and the answer is *not*
obvious.

So while I am guilty of maybe misnaming this function, you will
have noticed that this "exporter" actually does not show up in
the export menu as an independent exporter - it is mainly used
by publishing projects.

We could use a new function which does a configurable amount
of processing, maybe it could get a number of arguments which
define what exactly should be done - and it should definitely
write to a different file, without clobbering the current file.
Maybe file_expanded.org or so?

The way to write this function is to start at `org-export-as-org',
keep what you like and add more stuff from org-export-preprocess,
for example a call `org-export-handle-include-files-recurse' as
one of the first things.  Really, `org-export-preprocess-string'
can be the guide of what can and should be done.

I could write such a function, Nick could, but first there needs to
be clarity of what this should achieve, and how to address this issue
in a general way.

- Carsten

Re: Include does not work when doing org-export-as-org

[Henry Hirsch]

On Tue, Oct 11, 2011 at 12:51:24PM -0400, Nick Dokos wrote:
> The following should do basically what you want,
Yes, that realy hit the spot. Sorry for being so destructive earlier.
For a moment I thought you are telling me off.

Carsten: 
I think by default it should not expand macros, process babel blocks
or process links. Though it could be very usefull to have the
processing of babel-blocks with an optional on and off-switch.

But like you indicated. This is only the ideal solution for my use
case. The next person could need some other behaviour.

Thank you very much for caring about my problem.

Re: Include does not work when doing org-export-as-org

[Carsten Dominik]

On 12.10.2011, at 09:44, Henry Hirsch wrote:

> On Tue, Oct 11, 2011 at 12:51:24PM -0400, Nick Dokos wrote:
>> The following should do basically what you want,
> Yes, that realy hit the spot. Sorry for being so destructive earlier.
> For a moment I thought you are telling me off.
> 
> Carsten: 
> I think by default it should not expand macros, process babel blocks
> or process links. Though it could be very usefull to have the
> processing of babel-blocks with an optional on and off-switch.
> 
> But like you indicated. This is only the ideal solution for my use
> case. The next person could need some other behaviour.
> 
> Thank you very much for caring about my problem.

You are welcome - and I think it might be good in integrate
Nicks function into Org and make it available through the menu
or a key.

- Carsten

Re: Include does not work when doing org-export-as-org

[Bastien]

Carsten Dominik <carsten.dominik@gmail.com> writes:

> You are welcome - and I think it might be good in integrate
> Nicks function into Org and make it available through the menu
> or a key.

Indeed -- IIRC the problem is that Nick cannot submit non-tiny 
patches :(  I added Nick's code to Worg/org-hacks.org.  Carsten,
could you apply a patch with a rewrite of Nick's code and the 
changes in the menu/keymap?  (If Nick doesn't mind, of course.)

-- 
 Bastien

Re: Include does not work when doing org-export-as-org

[Nick Dokos]

Bastien <bzg@altern.org> wrote:

> Carsten Dominik <carsten.dominik@gmail.com> writes:
> 
> > You are welcome - and I think it might be good in integrate
> > Nicks function into Org and make it available through the menu
> > or a key.
> 
> Indeed -- IIRC the problem is that Nick cannot submit non-tiny 
> patches :(  I added Nick's code to Worg/org-hacks.org.  Carsten,
> could you apply a patch with a rewrite of Nick's code and the 
> changes in the menu/keymap?  (If Nick doesn't mind, of course.)
> 

I don't mind at all - you can do with it what you will.

Nick