[RFC] Org version of the Org manual

[RFC] Org version of the Org manual

[Thomas S. Dye]

Aloha all,

An experiment to see if it is possible to convert the Org manual to Org
is now (hopefully) at the "proof of concept" stage and I would like to
determine whether or not the approach is viable. The experiment is here:

https://github.com/tsdye/orgmanual.git

The file orgmanual.org has a heading, * Instructions for use, that
explains how to set up asynchronous export with the new exporter. The
repo also contains orgmanual.info in case you don't want to hassle with
exporting the org file and just want to see what it looks like in info.
I'm able to generate an info file without makeinfo raising any errors or
warnings. 

There is also a heading, * Improvements and fixes, that includes a list
of problems that either need to be fixed or are in the process of being
fixed. Obviously, the file is not perfect, but I do believe it is good
enough to judge whether success will be possible. At any rate, I need to
pull myself away from this project for a week or so to do other things
and can't get caught up making editorial changes or working on the (to
me) thorny issues that remain.

Sections with problems have been given FIXME keywords. In some cases,
problem text (especially in macros) has been replaced with `XXX'.
Proposed fixes for these nagging issues will be greatly appreciated.

By way of background, my initial goal was to replicate the Org manual
precisely. This goal was abandoned when I realized that the various
contributors to the manual (including me a few years ago) had different
ideas about what a .texi file should look like and had created a somewhat
eclectic product.  The goal now is to have a "functional equivalent" of
the manual.

The ultimate goal is to maintain the Org manual in Org, rather than
.texi, and to have the infrastructure in place in time to support the
major revisions required by the forthcoming release of version 8.0.

Many thanks to Nicolas Goaziou and to Jonathan Leech-Pepin for their
excellent work on the new exporter framework and the texinfo exporter,
and for their patience with my sometimes exasperated (and exasperating)
queries.  Jonathan showed me the way forward at an early stage and
devised most of the macros.

I look forward to picking this project up again in a week or so, or
learning from my Org colleagues that the project ought to be scuttled.

All the best,
Tom

-- 
T.S. Dye & Colleagues, Archaeologists
735 Bishop St, Suite 315, Honolulu, HI 96813
Tel: 808-529-0866, Fax: 808-529-0884
http://www.tsdye.com

Re: [RFC] Org version of the Org manual

[Carsten Dominik]

Hi Tom,

wow, a daring project.  This would indeed be a testimony to the quality of the exporter if this works and gives proper output!  I am curious and looking forward to seeing the results.

- Carsten

On 25.2.2013, at 21:21, Thomas S. Dye <tsd@tsdye.com> wrote:

> Aloha all,
> 
> An experiment to see if it is possible to convert the Org manual to Org
> is now (hopefully) at the "proof of concept" stage and I would like to
> determine whether or not the approach is viable. The experiment is here:
> 
> https://github.com/tsdye/orgmanual.git
> 
> The file orgmanual.org has a heading, * Instructions for use, that
> explains how to set up asynchronous export with the new exporter. The
> repo also contains orgmanual.info in case you don't want to hassle with
> exporting the org file and just want to see what it looks like in info.
> I'm able to generate an info file without makeinfo raising any errors or
> warnings. 
> 
> There is also a heading, * Improvements and fixes, that includes a list
> of problems that either need to be fixed or are in the process of being
> fixed. Obviously, the file is not perfect, but I do believe it is good
> enough to judge whether success will be possible. At any rate, I need to
> pull myself away from this project for a week or so to do other things
> and can't get caught up making editorial changes or working on the (to
> me) thorny issues that remain.
> 
> Sections with problems have been given FIXME keywords. In some cases,
> problem text (especially in macros) has been replaced with `XXX'.
> Proposed fixes for these nagging issues will be greatly appreciated.
> 
> By way of background, my initial goal was to replicate the Org manual
> precisely. This goal was abandoned when I realized that the various
> contributors to the manual (including me a few years ago) had different
> ideas about what a .texi file should look like and had created a somewhat
> eclectic product.  The goal now is to have a "functional equivalent" of
> the manual.
> 
> The ultimate goal is to maintain the Org manual in Org, rather than
> .texi, and to have the infrastructure in place in time to support the
> major revisions required by the forthcoming release of version 8.0.
> 
> Many thanks to Nicolas Goaziou and to Jonathan Leech-Pepin for their
> excellent work on the new exporter framework and the texinfo exporter,
> and for their patience with my sometimes exasperated (and exasperating)
> queries.  Jonathan showed me the way forward at an early stage and
> devised most of the macros.
> 
> I look forward to picking this project up again in a week or so, or
> learning from my Org colleagues that the project ought to be scuttled.
> 
> All the best,
> Tom
> 
> -- 
> T.S. Dye & Colleagues, Archaeologists
> 735 Bishop St, Suite 315, Honolulu, HI 96813
> Tel: 808-529-0866, Fax: 808-529-0884
> http://www.tsdye.com
> 

Re: [RFC] Org version of the Org manual

[Achim Gratz]

Thomas S. Dye writes:
[…]
> I look forward to picking this project up again in a week or so, or
> learning from my Org colleagues that the project ought to be scuttled.

Hi Tom,

this looks good.

I've implemented a rough draft of a Makefile that integrates with the
build system (a clean target is still missing, etc.pp.).  To use, clone
orgmanual.git in the Org worktree (don't add it, or add as a submodule),
then drop this Makefile into orgmanual/:

--8<---------------cut here---------------start------------->8---
BEXP=$(BATCH) \
	--eval '(add-to-list '"'"'load-path "../lisp")' \
	--eval '(setq org-footnote-auto-adjust nil)' \
	-l ox-texinfo \
	--eval '(add-to-list '"'"'org-export-snippet-translation-alist '"'"'("info" . "texinfo"))'
ORG2TEXI=-f org-texinfo-export-to-texinfo
ORG2INFO=--eval "(org-texinfo-compile \"./$<\")"

orgmanual.texi:	orgmanual.org
	$(BEXP) $< $(ORG2TEXI)
orgmanual.info:	orgmanual.texi
#	$(BEXP) $< $(ORG2INFO)
	$(MAKEINFO)  --no-split $< -o $@
orgmanual.pdf:	LC_ALL=C        # work around a bug in texi2dvi
orgmanual.pdf:	LANG=C          # work around a bug in texi2dvi
orgmanual.pdf:	orgmanual.texi
	$(TEXI2PDF) $<
--8<---------------cut here---------------end--------------->8---

Also make a link (for now) to ../doc/org-version.inc and add this to
local.mk:

--8<---------------cut here---------------start------------->8---
.PHONY:	orgmanual
orgmanual:
	$(MAKE) -C $@ $@.info $@.pdf
--8<---------------cut here---------------end--------------->8---


A "make orgmanual" then exports to orgmanual.texi (slowly) and an info
file is produced by makeinfo, but texi2dvi does not want to produce a
PDF yet and errors out.  The info file contains a few easily spotted
mistakes in the macro section like "~@{@{@{name(arg1,arg2)@~@}@}}" that
are quite probably the source of some of these errors.


Regards,
Achim.
-- 
+<[Q+ Matrix-12 WAVE#46+305 Neuron microQkb Andromeda XTk Blofeld]>+

Wavetables for the Terratec KOMPLEXER:
http://Synth.Stromeko.net/Downloads.html#KomplexerWaves

Re: [RFC] Org version of the Org manual

[Thomas S. Dye]

Hi Achim,

Achim Gratz <Stromeko@nexgo.de> writes:

> Thomas S. Dye writes:
> […]
>> I look forward to picking this project up again in a week or so, or
>> learning from my Org colleagues that the project ought to be scuttled.
>
> Hi Tom,
>
> this looks good.

Thanks for reviewing a work in progress.  I'm really pleased it looks
good to you.

>
> I've implemented a rough draft of a Makefile that integrates with the
> build system (a clean target is still missing, etc.pp.).  To use, clone
> orgmanual.git in the Org worktree (don't add it, or add as a submodule),
> then drop this Makefile into orgmanual/:

These instructions assume more knowledge than I have. Could you be more
specific about "in the Org worktree"? Should I create a branch where I
do this? Or, should I put this in one of the upstream branches and
arrange to push it into the Org repo?  In any case, can you suggest a
location?  /contrib?  /doc?

>
> BEXP=$(BATCH) \
> 	--eval '(add-to-list '"'"'load-path "../lisp")' \
> 	--eval '(setq org-footnote-auto-adjust nil)' \
> 	-l ox-texinfo \
> 	--eval '(add-to-list '"'"'org-export-snippet-translation-alist '"'"'("info" . "texinfo"))'
> ORG2TEXI=-f org-texinfo-export-to-texinfo
> ORG2INFO=--eval "(org-texinfo-compile \"./$<\")"
>
> orgmanual.texi:	orgmanual.org
> 	$(BEXP) $< $(ORG2TEXI)
> orgmanual.info:	orgmanual.texi
> #	$(BEXP) $< $(ORG2INFO)
> 	$(MAKEINFO)  --no-split $< -o $@
> orgmanual.pdf:	LC_ALL=C        # work around a bug in texi2dvi
> orgmanual.pdf:	LANG=C          # work around a bug in texi2dvi
> orgmanual.pdf:	orgmanual.texi
> 	$(TEXI2PDF) $<

This seems straightforward.

>
> Also make a link (for now) to ../doc/org-version.inc and add this to
> local.mk:
>
> .PHONY:	orgmanual
> orgmanual:
> 	$(MAKE) -C $@ $@.info $@.pdf
>

I can do this, but then the orgmanual.info generated from the source
will indicate the wrong version. Is that OK? I have instructions from
Bastien how to cull all the changes to org.texi that have taken place
since I started translating the manual to org several months ago. When
I'm done going through those, then the link will indicate the correct
version.

>
> A "make orgmanual" then exports to orgmanual.texi (slowly) and an info
> file is produced by makeinfo, but texi2dvi does not want to produce a
> PDF yet and errors out.  The info file contains a few easily spotted
> mistakes in the macro section like "~@{@{@{name(arg1,arg2)@~@}@}}" that
> are quite probably the source of some of these errors.

Thanks, I found these and have corrected them.  There are probably more ...

All the best,
Tom

-- 
Thomas S. Dye
http://www.tsdye.com

Re: [RFC] Org version of the Org manual

[Yagnesh Raghava Yakkala]

Hello Thomas,

> These instructions assume more knowledge than I have. Could you be more
> specific about "in the Org worktree"? Should I create a branch where I
> do this? Or, should I put this in one of the upstream branches and
> arrange to push it into the Org repo?  In any case, can you suggest a
> location?  /contrib?  /doc?

IIUC, Achim means,

--8<---------------cut here---------------start------------->8---
cd /source/directory/of/org-mode
git clone http:github.com/tsdye/orgmanual  # or your most updated repo of orgmanual

git submodule add orgmanual                # optional
git commit -m "Commit Message"             # optional

cd orgmanual
Create_Makefile                            # [1] 
ln -s ../doc/org-version.inc .

cd ..
update_local.mk_file                       # [2]
make orgmanual
--8<---------------cut here---------------end--------------->8---

For me also texi2dvi failed, but orgmanual.info is generated fine and looks
great.

Given texi generation is very slow, I wonder which format of manual will be
included in Emacs trunk.?

Anyway, very nice to see org manual in org Format. 

Thanks.,

[1],[2] are code bits from Achim's mail.

-- 
ఎందరో మహానుభావులు అందరికి వందనములు.
YYR

Re: [RFC] Org version of the Org manual

[Achim Gratz]

[-- Attachment #1: Type: text/plain, Size: 508 bytes --]

Yagnesh Raghava Yakkala writes:
> IIUC, Achim means,

Thanks for spelling that out for me. :-)

> Given texi generation is very slow, I wonder which format of manual
> will be included in Emacs trunk.?

Well, there obviously needs to be improvement on that front.  I've
profiled the export already, but haven't had time yet to make sense of
the results.  Attached is the output from elp.  I also have profiler
results, but these are lisp data and I first need to figure out how to
load them in Emacs again.


[-- Attachment #2: elp.txt --]
[-- Type: text/plain, Size: 15700 bytes --]

org-entry-get                                       350676      469.21990999  0.0013380439
org-babel-exp-process-buffer                        1           317.362436    317.362436
org-babel-exp-src-block                             138         308.52577799  2.2356940434
org-babel-get-src-block-info                        3834        305.86685299  0.0797774786
org-babel-parse-src-block-match                     3834        301.70857400  0.0786928988
org-babel-exp-do-export                             138         299.98738899  2.1738216594
org-babel-exp-code                                  138         299.34870600  2.1691935217
org-babel-expand-noweb-references                   24          299.114988    12.4631245
org-babel-params-from-properties                    3834        297.16263699  0.0775072083
org-entry-get-with-inheritance                      95850       290.83956400  0.0030343199
org-macro-replace-all                               2           222.371024    111.185512
org-element-context                                 8587        210.50228799  0.0245140663
org-element-at-point                                8998        198.89139700  0.0221039561
org-element--current-element                        237338      185.80111699  0.0007828544
org-get-property-block                              314005      142.07183900  0.0004524508
org-up-heading-safe                                 254825      98.096140999  0.0003849549
org-element--parse-elements                         3119        78.158617     0.0250588704
org-element-headline-parser                         59179       62.237870000  0.0010516884
org-back-to-heading                                 1177194     54.685840999  4.645...e-05
org-list-struct                                     8118        50.615231000  0.0062349385
org-element-map                                     2159        41.052958000  0.0190148022
org-before-first-heading-p                          263824      36.090307000  0.0001367969
org-element--get-next-object-candidates             43922       26.156263000  0.0005955162
org-outline-level                                   472987      23.837515999  5.039...e-05
org-end-of-subtree                                  59187       22.566596999  0.0003812762
org-get-indentation                                 529429      17.841042000  3.369...e-05
org-entry-properties                                59179       15.035857000  0.0002540741
org-element-parse-buffer                            1           14.286519     14.286519
org-element--parse-objects                          6307        10.982457     0.0017413123
org-element-paragraph-parser                        71441       9.5795360000  0.0001340901
org-element-item-parser                             25674       7.3659629999  0.0002869035
org-heading-components                              59179       6.7734890000  0.0001144576
org-element-normalize-string                        10028       6.3973670000  0.0006379504
org-at-heading-p                                    359654      5.7468979999  1.597...e-05
org-element-plain-list-parser                       9210        5.5201879999  0.0005993689
org-element-latex-or-entity-successor               14616       5.0801999999  0.0003475779
org-list-context                                    8118        4.9218190000  0.0006062846
org-babel-exp-non-block-elements                    139         4.8030860000  0.0345545755
org-element-footnote-definition-parser              32897       4.1083099999  0.0001248840
org-babel-merge-params                              8709        3.5205869999  0.0004042469
org-element--collect-affiliated-keywords            144443      3.3394739999  2.311...e-05
org-element-property-drawer-parser                  21129       2.9038599999  0.0001374348
org-get-limited-outline-regexp                      241703      2.8068669999  1.161...e-05
org-element-link-successor                          33549       2.7400770000  8.167...e-05
org-list-get-list-end                               9210        2.2022560000  0.0002391157
org-invisible-p2                                    110997      1.8746269999  1.688...e-05
org-list-get-last-item                              9210        1.8070339999  0.0001962034
org-element-parse-secondary-string                  1548        1.7770869999  0.0011479890
org-at-table-p                                      80490       1.7516579999  2.176...e-05
org-element-keyword-parser                          5407        1.6982869999  0.0003140904
org-element-export-snippet-successor                38156       1.6902520000  4.429...e-05
org-element-sub/superscript-successor               14996       1.4842760000  9.897...e-05
org-element-line-break-successor                    10676       1.4662499999  0.0001373407
org-element-timestamp-successor                     14512       1.4495880000  9.988...e-05
org-element-src-block-parser                        2741        1.4298379999  0.0005216483
org-trim                                            99454       1.4216499999  1.429...e-05
org-babel-parse-header-arguments                    8709        1.4209139999  0.0001631546
org-element-inline-src-block-successor              13626       1.3379629999  9.819...e-05
org-macro-initialize-templates                      1           1.248365      1.248365
org-macro--collect-macros                           1           1.248089      1.248089
org-element-macro-parser                            8502        1.1616150000  0.0001366284
org-unescape-code-in-string                         9311        1.1497540000  0.0001234834
org-macro-expand                                    8474        1.0761649999  0.0001269961
org-element-inline-babel-call-successor             13626       0.9184050000  6.740...e-05
org-element-macro-successor                         14571       0.9024950000  6.193...e-05
org-list-struct-assoc-end                           8118        0.8850810000  0.0001090269
org-element-text-markup-successor                   17857       0.8849819999  4.955...e-05
org-element-export-snippet-parser                   23904       0.8848189999  3.701...e-05
org-list-get-list-type                              9210        0.8797819999  9.552...e-05
org-item-re                                         86556       0.8766279999  1.012...e-05
org-list-prevs-alist                                9210        0.8290719999  9.001...e-05
org-element-footnote-reference-successor            14838       0.7596100000  5.119...e-05
org-not-nil                                         95850       0.7501719999  7.826...e-06
org-element-normalize-contents                      3919        0.6525249999  0.0001665029
org-list-get-next-item                              67249       0.6363409999  9.462...e-06
org-babel-remove-result                             138         0.634481      0.0045976884
org-babel-where-is-src-block-result                 138         0.632719      0.0045849202
org-babel-ref-goto-headline-id                      22          0.61628       0.0280127272
org-element-set-contents                            29511       0.5933719999  2.010...e-05
org-element-statistics-cookie-successor             13740       0.5776040000  4.203...e-05
org-babel-balanced-split                            5712        0.5430289999  9.506...e-05
org-list-parents-alist                              9210        0.5095729999  5.532...e-05
org-list-get-list-begin                             9210        0.4939830000  5.363...e-05
org-element-example-block-parser                    2736        0.4815410000  0.0001760018
org-babel-find-named-result                         54          0.4734820000  0.0087681851
org-do-remove-indentation                           4156        0.4653900000  0.0001119802
org-get-category                                    59127       0.4292040000  7.259...e-06
org-element-table-parser                            1026        0.4162909999  0.0004057417
org-element-target-successor                        14529       0.4006059999  2.757...e-05
org-element-table-row-parser                        5052        0.3823159999  7.567...e-05
org-at-table.el-p                                   1026        0.3236109999  0.0003154103
org-reduced-level                                   59179       0.3232659999  5.462...e-06
org-element-radio-target-successor                  14503       0.2978549999  2.053...e-05
org-split-string                                    36231       0.2893910000  7.987...e-06
org-uniquify                                        40152       0.2739410000  6.822...e-06
org-list-get-prev-item                              9210        0.2735659999  2.970...e-05
org-element-table-cell-successor                    1291        0.2522169999  0.0001953656
org-babel-chomp                                     6306        0.2347340000  3.722...e-05
org-find-entry-with-id                              22          0.2240680000  0.0101849090
org-babel-read                                      5580        0.2164680000  3.879...e-05
org-babel-where-is-src-block-head                   3972        0.2142490000  5.393...e-05
org-babel-parse-multiple-vars                       9194        0.1842510000  2.004...e-05
org-element-property                                47128       0.1440620000  3.056...e-06
org-babel-get-inline-src-block-matches              138         0.1359030000  0.0009848043
org-fill-template                                   138         0.1243740000  0.0009012608
org-string-nw-p                                     5483        0.1167720000  2.129...e-05
org-list-get-item-end                               34884       0.1068650000  3.063...e-06
org-element-code-parser                             3608        0.0825499999  2.287...e-05
org-skip-whitespace                                 8999        0.0773929999  8.600...e-06
org-element-link-parser                             582         0.0719579999  0.0001236391
org-remove-indentation                              322         0.0622920000  0.0001934534
org-update-radio-target-regexp                      1           0.058251      0.058251
org-all-targets                                     1           0.058217      0.058217
org-element-section-parser                          377         0.0538779999  0.0001429124
org-add-props                                       14213       0.0532790000  3.748...e-06
org-refresh-category-properties                     2           0.038394      0.019197
org-element-table-cell-parser                       1291        0.0378999999  2.935...e-05
org-kill-new                                        1           0.036258      0.036258
org-babel-number-p                                  5580        0.0318689999  5.711...e-06
org-babel-combine-header-arg-lists                  3834        0.0273989999  7.146...e-06
org-element-type                                    8784        0.0224829999  2.559...e-06
org-list-get-top-point                              9210        0.0213349999  2.316...e-06
org-string-match-p                                  2742        0.0202060000  7.369...e-06
org-element-node-property-parser                    399         0.0194659999  4.878...e-05
org-babel-trim                                      352         0.0149620000  4.250...e-05
org-element-footnote-reference-parser               335         0.014787      4.414...e-05
org-link-unescape                                   578         0.0096020000  1.661...e-05
org-link-expand-abbrev                              578         0.0088600000  1.532...e-05
org-element-italic-parser                           332         0.0072709999  2.190...e-05
org-version                                         2           0.006288      0.003144
org-babel-join-splits-near-ch                       352         0.0057630000  1.637...e-05
org-escape-code-in-string                           138         0.004417      3.200...e-05
org-element-subscript-parser                        104         0.002996      2.880...e-05
org-reverse-string                                  704         0.0029590000  4.203...e-06
org-file-equal-p                                    2           0.002958      0.001479
org-element-comment-parser                          50          0.0023529999  4.705...e-05
org-reduce                                          352         0.0021599999  6.136...e-06
org-babel-noweb-p                                   138         0.0017369999  1.258...e-05
org-element-plain-link-successor                    45          0.0015850000  3.522...e-05
org-element-statistics-cookie-parser                115         0.0015400000  1.339...e-05
org-translate-link                                  578         0.0012740000  2.204...e-06
org-remove-if                                       355         0.0010609999  2.988...e-06
org-get-agenda-file-buffer                          22          0.00084       3.818...e-05
org-element-bold-parser                             36          0.0006630000  1.841...e-05
org-element-drawer-parser                           10          0.000658      6.58e-05
org-element-timestamp-parser                        5           0.0003999999  7.999...e-05
org-element-superscript-parser                      15          0.0003909999  2.606...e-05
org-find-base-buffer-visiting                       22          0.000366      1.663...e-05
org-element-latex-fragment-parser                   5           0.000319      6.38e-05
org-combine-plists                                  7           0.00029       4.142...e-05
org-element-target-parser                           19          0.0002800000  1.473...e-05
org-element-fixed-width-parser                      9           0.000277      3.077...e-05
org-defkey                                          20          0.000236      1.179...e-05
org-babel-noweb-wrap                                46          0.0001609999  3.499...e-06
org-add-link-type                                   1           0.000129      0.000129
org-parse-time-string                               5           0.000126      2.52e-05
org-element-underline-parser                        5           0.000115      2.3e-05
org-make-link-regexps                               1           9.6e-05       9.6e-05
org-babel-result-end                                1           9.2e-05       9.2e-05
org-element-verbatim-parser                         3           7.2e-05       2.4e-05
org-link-unescape-compound                          3           6.5e-05       2.166...e-05
org-timestamp-translate                             2           6e-05         3e-05
org-element-quote-block-parser                      1           5.9e-05       5.9e-05
org-element-interpret-data                          6           5.600...e-05  9.333...e-06
org-key                                             20          3.700...e-05  1.850...e-06
org-region-active-p                                 3           3.1e-05       1.033...e-05
org-at-item-p                                       1           2.7e-05       2.7e-05
org-release                                         2           9e-06         4.5e-06
org-translate-time                                  2           6e-06         3e-06
org-restart-font-lock                               1           6e-06         6e-06
org-babel-exp-get-export-buffer                     2           6e-06         3e-06
org-git-version                                     2           4.999...e-06  2.499...e-06
org-property-inherit-p                              1           3e-06         3e-06
org-make-target-link-regexp                         1           2e-06         2e-06

[-- Attachment #3: Type: text/plain, Size: 189 bytes --]




Regards,
Achim.
-- 
+<[Q+ Matrix-12 WAVE#46+305 Neuron microQkb Andromeda XTk Blofeld]>+

SD adaptation for Waldorf microQ V2.22R2:
http://Synth.Stromeko.net/Downloads.html#WaldorfSDada

Re: [RFC] Org version of the Org manual

[Bastien]

Hi Achim,

thanks for doing the profiling.

Achim Gratz <Stromeko@nexgo.de> writes:

> org-macro-replace-all    2    222.371024    111.185512

This one looks suspicious: if the function does what it says,
why does it run twice?  If we can spare 111 secs here, that's
already an improvement.  I'll have a look at some point but
feel free to beat me on this!

-- 
 Bastien

Re: [RFC] Org version of the Org manual

[Nicolas Goaziou]

Hello,

Bastien <bzg@altern.org> writes:

> Hi Achim,
>
> thanks for doing the profiling.
>
> Achim Gratz <Stromeko@nexgo.de> writes:
>
>> org-macro-replace-all    2    222.371024    111.185512
>
> This one looks suspicious: if the function does what it says,
> why does it run twice?  If we can spare 111 secs here, that's
> already an improvement.  I'll have a look at some point but
> feel free to beat me on this!

It has to be run twice. See comments in `org-export-as':

  Expand export-specific set of macros: {{{author}}},
  {{{date}}}, {{{email}}} and {{{title}}}.  It must be done
  once regular macros have been expanded, since document
  keywords may contain one of them.

IOW it handles macros in #+TITLE and other document keywords. I don't
think the second run is as long as the first one, though.


Regards,

-- 
Nicolas Goaziou

Re: [RFC] Org version of the Org manual

[Achim Gratz]

Bastien writes:
>> org-macro-replace-all    2    222.371024    111.185512
>
> This one looks suspicious: if the function does what it says,
> why does it run twice?  If we can spare 111 secs here, that's
> already an improvement.  I'll have a look at some point but
> feel free to beat me on this!

That interpretation makes assumptions that may not be valid.  I'm not
even sure that you can rely on the number of calls (elp may miss some at
least), but especially not that each call takes the same time.  So it is
perfectly possible that the first call took 222 seconds and the second
one just 0.371024…

The top entry for profiler btw is "called-interactively".  How helpful.

Regards,
Achim.
-- 
+<[Q+ Matrix-12 WAVE#46+305 Neuron microQkb Andromeda XTk Blofeld]>+

Factory and User Sound Singles for Waldorf rackAttack:
http://Synth.Stromeko.net/Downloads.html#WaldorfSounds

Re: [RFC] Org version of the Org manual

[Jambunathan K]

Achim Gratz <Stromeko@nexgo.de> writes:

> Bastien writes:
>>> org-macro-replace-all    2    222.371024    111.185512
>>
>> This one looks suspicious: if the function does what it says,
>> why does it run twice?  If we can spare 111 secs here, that's
>> already an improvement.  I'll have a look at some point but
>> feel free to beat me on this!
>
> That interpretation makes assumptions that may not be valid.  I'm not
> even sure that you can rely on the number of calls (elp may miss some at
> least), but especially not that each call takes the same time.  So it is
> perfectly possible that the first call took 222 seconds and the second
> one just 0.371024…

See (info "(elisp) Profiling") on a fairly recent Emacs.

,---- C-h n => Lisp Changes in Emacs 24.3
| ** New sampling-based Elisp profiler.
| Try M-x profiler-start, do some work, and then call M-x profiler-report.
| When finished, use M-x profiler-stop.  The sampling rate can be based on
| CPU time or memory allocations.
`----


>
> The top entry for profiler btw is "called-interactively".  How helpful.
>
> Regards,
> Achim.

-- 

Re: [RFC] Org version of the Org manual

[Thomas S. Dye]

Aloha Yagnesh Raghava Yakkala,

Yagnesh Raghava Yakkala <hi@yagnesh.org> writes:

> Hello Thomas,
>
>> These instructions assume more knowledge than I have. Could you be more
>> specific about "in the Org worktree"? Should I create a branch where I
>> do this? Or, should I put this in one of the upstream branches and
>> arrange to push it into the Org repo?  In any case, can you suggest a
>> location?  /contrib?  /doc?
>
> IIUC, Achim means,
>
> cd /source/directory/of/org-mode
> git clone http:github.com/tsdye/orgmanual  # or your most updated repo of orgmanual
>
> git submodule add orgmanual                # optional
> git commit -m "Commit Message"             # optional
>
> cd orgmanual
> Create_Makefile                            # [1] 
> ln -s ../doc/org-version.inc .
>
> cd ..
> update_local.mk_file                       # [2]
> make orgmanual
>
> For me also texi2dvi failed, but orgmanual.info is generated fine and looks
> great.

Thanks for the detailed instructions, which helped me greatly.

The orgmanual now creates a pdf file, too, but one lacking the indexes
and with links that look like they need another run through TeX.

When I'm done running `make orgmanual', my directory is very neat and
tidy, which is nice, but I'll need to look at the auxiliary files used
to build the indexes, so I can debug.  Should I run texi2dvi outside
`make orgmanual', or would some other workflow better fit my needs?
Suggestions warmly welcomed.

Thanks again for your help.

All the best,
Tom
-- 
Thomas S. Dye
http://www.tsdye.com

Re: [RFC] Org version of the Org manual

[Achim Gratz]

Thomas S. Dye writes:
> The orgmanual now creates a pdf file, too, but one lacking the indexes
> and with links that look like they need another run through TeX.

In that case, texi2dvi and hence make should have signaled an error
(I've just pulled from your repo and it does).

> When I'm done running `make orgmanual', my directory is very neat and
> tidy, which is nice, but I'll need to look at the auxiliary files used
> to build the indexes, so I can debug.

Add

TEXI2PDF+=--tidy

at the beginning of orgmanual/Makefile, then look in the orgmanual.t2d
directory.  Or if you'd rather have all the files in the same directory
like before:

TEXI2PDF+=--build=local


Regards,
Achim.
-- 
+<[Q+ Matrix-12 WAVE#46+305 Neuron microQkb Andromeda XTk Blofeld]>+

SD adaptation for Waldorf Blofeld V1.15B11:
http://Synth.Stromeko.net/Downloads.html#WaldorfSDada

Re: [RFC] Org version of the Org manual

[Thomas S. Dye]

Achim Gratz <Stromeko@nexgo.de> writes:

> Thomas S. Dye writes:
>> The orgmanual now creates a pdf file, too, but one lacking the indexes
>> and with links that look like they need another run through TeX.
>
> In that case, texi2dvi and hence make should have signaled an error
> (I've just pulled from your repo and it does).

Yes, it signals two errors here:

Output written on orgmanual.pdf (269 pages, 1122108 bytes).
Transcript written on orgmanual.log.
/opt/local/bin/texi2dvi: pdftex exited with bad status, quitting.
make[1]: *** [orgmanual.pdf] Error 1
make: *** [orgmanual] Error 2

>
>> When I'm done running `make orgmanual', my directory is very neat and
>> tidy, which is nice, but I'll need to look at the auxiliary files used
>> to build the indexes, so I can debug.
>
> Add
>
> TEXI2PDF+=--tidy
>
> at the beginning of orgmanual/Makefile, then look in the orgmanual.t2d
> directory.

There they are, in orgmanual.t2d/pdf/build.  Thanks!

All the best,
Tom

-- 
Thomas S. Dye
http://www.tsdye.com

Re: [RFC] Org version of the Org manual

[Thomas S. Dye]

Achim Gratz <Stromeko@nexgo.de> writes:

> Thomas S. Dye writes:
>> The orgmanual now creates a pdf file, too, but one lacking the indexes
>> and with links that look like they need another run through TeX.
>
> In that case, texi2dvi and hence make should have signaled an error
> (I've just pulled from your repo and it does).
>
>> When I'm done running `make orgmanual', my directory is very neat and
>> tidy, which is nice, but I'll need to look at the auxiliary files used
>> to build the indexes, so I can debug.
>
> Add
>
> TEXI2PDF+=--tidy

That works nicely.  I found the error and orgmanual.pdf is now produced
without errors.

I found that the common combination of `keybinding org-command', which I
simplified in the translation to org, looks good in the info file but
not in the pdf file, where the elements weren't easy to distinguish
visually. I've changed all these to `keybinding, org-command' which
still looks simple but works (for me) in both info and pdf formats.

Is the html version of the Org manual generated from the .texi source?
If so, could you show me how to augment Makefile so the html
document is generated by `make orgmanual'?  I want to check if the html
document looks reasonable.

My next step will be to bring orgmanual up-to-date with the changes
that have been made to org.texi since I started the translation several
months ago.

There are still about a half dozen places where I've substituted a `XXX'
placeholder for a problem I haven't been able to solve. Sections with
problems have been marked FIXME. Most of the problems have to do with
exporting a "special" character, such as `\' or `,'. I'm afraid these
nagging problems will need to be solved by someone clever in the Org
community.

All the best,
Tom

-- 
Thomas S. Dye
http://www.tsdye.com

Re: [RFC] Org version of the Org manual

[Achim Gratz]

[-- Attachment #1: Type: text/plain, Size: 1876 bytes --]

Thomas S. Dye writes:
> That works nicely.  I found the error and orgmanual.pdf is now produced
> without errors.

Progress! :-)

With the current version from git I cannot export to texinfo
successfully, though, I get this error near the end of the export:

Debugger entered--Lisp error: (wrong-type-argument stringp nil)
  string-match("\\`[ \n.]+" nil) (if (string-match "\\`[ \n.]+" s)
  (setq s (replace-match "" t t s))) org-trim(nil) (concat "\n@item "
  (if tag desc) "\n" (org-trim contents) "\n") (let* ((tag
  (org-element-property :tag item)) (desc (org-export-data tag info)))
  (concat "\n@item " (if tag desc) "\n" (org-trim contents) "\n"))
  org-texinfo-item((item (:bullet "- " :begin 41929 :end 42016
  :contents-begin 42016 :contents-end 42016 :checkbox nil :counter nil
  :hiddenp outline :structure ((40825 2 "- " nil nil "@@info:@kbd{@@C-c
  /@@info:}@@, ~org-sparse-tree~" 41031) (41031 2 "- " nil nil
  "@@info:@kbd{@@C-c / r@@info:}@@, ~org-occur~" 41929) (41929 2 "- "
  nil nil "@@info:@kbd{@@M-g n@@info:}@@ or @@info:@kbd{@@M-g
  M-n@@info:}@@, ~next-error~" 42016))…

This may actually a bug in the texinfo exporter.

> Is the html version of the Org manual generated from the .texi source?
> If so, could you show me how to augment Makefile so the html
> document is generated by `make orgmanual'?  I want to check if the html
> document looks reasonable.

I've extended the Makefile to approximate the one in doc/, HTML is
produced both via makeinfo and as an export via ox-html.  To proceed in
an orderly manner and prepare for an eventual integration into Org, can
you please do the following in your Org clone:

git checkout master
git checkout -b orgmanual master
git submodule add https://github.com/tsdye/orgmanual.git
git commit -am 'make orgmanual/ a submodule'

cd orgmanual
git checkout -b orgmanual master
git am orgmanual.patch


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: orgmanual.patch --]
[-- Type: text/x-patch, Size: 2647 bytes --]

From 5511736802ced4e08bd432ebdb12dfeebe282a22 Mon Sep 17 00:00:00 2001
From: Achim Gratz <Stromeko@Stromeko.DE>
Date: Sun, 10 Mar 2013 12:07:19 +0100
Subject: [PATCH] add Makefile, org-version.inc (symbolic link), modify
 .gitignore

ignore make products
---
 .gitignore      |  7 ++++---
 Makefile        | 49 +++++++++++++++++++++++++++++++++++++++++++++++++
 org-version.inc |  1 +
 3 files changed, 54 insertions(+), 3 deletions(-)
 create mode 100644 Makefile
 create mode 120000 org-version.inc

diff --git a/.gitignore b/.gitignore
index 1f940d9..fde636e 100644
--- a/.gitignore
+++ b/.gitignore
@@ -1,10 +1,8 @@
 *.*~
 *.texi
 *.info
-Makefile
 init.el
 export-test.org
-org-version.inc
 *.aux
 *.cp
 *.fn
@@ -17,4 +15,7 @@ org-version.inc
 *.vr
 *.cps
 *.pgs
-.DS_Store
\ No newline at end of file
+.DS_Store
+/orgmanual.html
+/orgmanual.t2d/
+/orgmanual/
diff --git a/Makefile b/Makefile
new file mode 100644
index 0000000..7c45664
--- /dev/null
+++ b/Makefile
@@ -0,0 +1,49 @@
+TEXI2PDF+=--tidy
+BEXP=$(BATCH) \
+	--eval '(add-to-list '"'"'load-path "../lisp")' \
+	--eval '(setq org-footnote-auto-adjust nil)'
+EXTEXI=	-l ox-texinfo \
+	--eval '(add-to-list '"'"'org-export-snippet-translation-alist '"'"'("info" . "texinfo"))'
+EXHTML=	-l ox-html \
+	$(BTEST_POST) \
+	--eval '(add-to-list '"'"'org-export-snippet-translation-alist '"'"'("info" . "texinfo"))'
+ORG2TEXI=-f org-texinfo-export-to-texinfo
+ORG2HTML=-f org-html-export-to-html
+ORG2INFO=--eval "(org-texinfo-compile \"./$<\")"
+
+.SUFFIXES:	# we don't need default suffix rules
+ifeq ($(MAKELEVEL), 0)
+  $(error This make needs to be started as a sub-make from the toplevel directory.)
+endif
+.PHONY:		all info html pdf
+
+all:		$(ORG_MAKE_DOC)
+
+info:		orgmanual.info
+
+html:		orgmanual orgmanual.html
+
+pdf:		orgmanual.pdf
+
+orgmanual.texi:	orgmanual.org
+	$(BEXP) $(EXTEXI) $< $(ORG2TEXI)
+orgmanual.info:	orgmanual.texi
+	$(MAKEINFO)  --no-split $< -o $@
+orgmanual.pdf:	LC_ALL=C        # work around a bug in texi2dvi
+orgmanual.pdf:	LANG=C          # work around a bug in texi2dvi
+orgmanual.pdf:	orgmanual.texi
+	$(TEXI2PDF) $<
+orgmanual:	orgmanual.texi
+	$(TEXI2HTML) $<
+orgmanual.html: orgmanual.org
+	$(BEXP) $(EXHTML) $< $(ORG2HTML)
+
+clean:
+	$(RM) org *.pdf *.html \
+	      *.aux *.cp *.cps *.dvi *.fn *.fns *.ky *.kys *.pg *.pgs \
+	      *.toc *.tp *.tps *.vr *.vrs *.log *.html *.ps
+cleanall:	clean
+	$(RMR) orgmanual.t2d orgmanual
+
+
+
diff --git a/org-version.inc b/org-version.inc
new file mode 120000
index 0000000..fd0c05b
--- /dev/null
+++ b/org-version.inc
@@ -0,0 +1 @@
+../doc/org-version.inc
\ No newline at end of file
-- 
1.8.1.4


[-- Attachment #3: Type: text/plain, Size: 69 bytes --]


cd ..
git commit -am 'update submodule orgmanual'
git am org.patch


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #4: org.patch --]
[-- Type: text/x-patch, Size: 777 bytes --]

From 488e1aa75d9420a238851217c39f52ecf317e8a0 Mon Sep 17 00:00:00 2001
From: Achim Gratz <Stromeko@Stromeko.DE>
Date: Sun, 10 Mar 2013 12:11:29 +0100
Subject: [PATCH] add $(EXTRADIRS) to targets.mk for build system integration
 of orgmanual/

---
 mk/targets.mk | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/mk/targets.mk b/mk/targets.mk
index aef11eb..b7fe9cc 100644
--- a/mk/targets.mk
+++ b/mk/targets.mk
@@ -6,7 +6,7 @@ DISTFILES_extra=  Makefile request-assign-future.txt contrib etc
 LISPDIRS      = lisp
 OTHERDIRS     = doc etc
 CLEANDIRS     = contrib testing mk
-SUBDIRS       = $(OTHERDIRS) $(LISPDIRS)
+SUBDIRS       = $(OTHERDIRS) $(LISPDIRS) $(EXTRADIRS)
 INSTSUB       = $(SUBDIRS:%=install-%)
 ORG_MAKE_DOC ?= info html pdf
 
-- 
1.8.1.4


[-- Attachment #5: Type: text/plain, Size: 1075 bytes --]


If you are unsure about any of this, please ask.  You can now edit/add
these lines

--8<---------------cut here---------------start------------->8---
.PHONY: orgmanual
EXTRADIRS=orgmanual
orgmanual:
        $(MAKE) -C $@
--8<---------------cut here---------------end--------------->8---

to the top of your local.mk and should now be able to do a "make
orgmanual".  Which types of documentation are produced can be controlled
with ORG_MAKE_DOC (default is "info pdf html"), just like for the
official manuals.  Also, "make cleanall" will now clean up in orgmanual
also.  BTEST_POST should be configured to have a load-path pointing to a
sufficiently advanced htmlize version for the HTML export.

> My next step will be to bring orgmanual up-to-date with the changes
> that have been made to org.texi since I started the translation several
> months ago.

I'm not envious…


Regards,
Achim.
-- 
+<[Q+ Matrix-12 WAVE#46+305 Neuron microQkb Andromeda XTk Blofeld]>+

SD adaptations for Waldorf Q V3.00R3 and Q+ V3.54R2:
http://Synth.Stromeko.net/Downloads.html#WaldorfSDada

Re: [RFC] Org version of the Org manual

[Jonathan Leech-Pepin]

[-- Attachment #1: Type: text/plain, Size: 3753 bytes --]

Hello Achim,

On 10 March 2013 08:24, Achim Gratz <Stromeko@nexgo.de> wrote:

> Thomas S. Dye writes:
> > That works nicely.  I found the error and orgmanual.pdf is now produced
> > without errors.
>
> Progress! :-)
>
> With the current version from git I cannot export to texinfo
> successfully, though, I get this error near the end of the export:
>
> Debugger entered--Lisp error: (wrong-type-argument stringp nil)
>   string-match("\\`[ \n.]+" nil) (if (string-match "\\`[ \n.]+" s)
>   (setq s (replace-match "" t t s))) org-trim(nil) (concat "\n@item "
>   (if tag desc) "\n" (org-trim contents) "\n") (let* ((tag
>   (org-element-property :tag item)) (desc (org-export-data tag info)))
>   (concat "\n@item " (if tag desc) "\n" (org-trim contents) "\n"))
>   org-texinfo-item((item (:bullet "- " :begin 41929 :end 42016
>   :contents-begin 42016 :contents-end 42016 :checkbox nil :counter nil
>   :hiddenp outline :structure ((40825 2 "- " nil nil "@@info:@kbd{@@C-c
>   /@@info:}@@, ~org-sparse-tree~" 41031) (41031 2 "- " nil nil
>   "@@info:@kbd{@@C-c / r@@info:}@@, ~org-occur~" 41929) (41929 2 "- "
>   nil nil "@@info:@kbd{@@M-g n@@info:}@@ or @@info:@kbd{@@M-g
>   M-n@@info:}@@, ~next-error~" 42016))…
>
> This may actually a bug in the texinfo exporter.
>
>
The error is actually on line 6069 of the manual.  The {{{vindex[...]}}}
line and
subsequent paragraph.  As far as the list is concerned there is no
associated
content for that list entry.  Indenting them appropriately to be recognized
as part
of the list allows for successful export.

This may also be partly a bug, should the exporter allow for a list item
without any contents?

Regards,

Jon

> > Is the html version of the Org manual generated from the .texi source?
> > If so, could you show me how to augment Makefile so the html
> > document is generated by `make orgmanual'?  I want to check if the html
> > document looks reasonable.
>
> I've extended the Makefile to approximate the one in doc/, HTML is
> produced both via makeinfo and as an export via ox-html.  To proceed in
> an orderly manner and prepare for an eventual integration into Org, can
> you please do the following in your Org clone:
>
> git checkout master
> git checkout -b orgmanual master
> git submodule add https://github.com/tsdye/orgmanual.git
> git commit -am 'make orgmanual/ a submodule'
>
> cd orgmanual
> git checkout -b orgmanual master
> git am orgmanual.patch
>
>
>
> cd ..
> git commit -am 'update submodule orgmanual'
> git am org.patch
>
>
>
> If you are unsure about any of this, please ask.  You can now edit/add
> these lines
>
> --8<---------------cut here---------------start------------->8---
> .PHONY: orgmanual
> EXTRADIRS=orgmanual
> orgmanual:
>         $(MAKE) -C $@
> --8<---------------cut here---------------end--------------->8---
>
> to the top of your local.mk and should now be able to do a "make
> orgmanual".  Which types of documentation are produced can be controlled
> with ORG_MAKE_DOC (default is "info pdf html"), just like for the
> official manuals.  Also, "make cleanall" will now clean up in orgmanual
> also.  BTEST_POST should be configured to have a load-path pointing to a
> sufficiently advanced htmlize version for the HTML export.
>
> > My next step will be to bring orgmanual up-to-date with the changes
> > that have been made to org.texi since I started the translation several
> > months ago.
>
> I'm not envious…
>
>
> Regards,
> Achim.
> --
> +<[Q+ Matrix-12 WAVE#46+305 Neuron microQkb Andromeda XTk Blofeld]>+
>
> SD adaptations for Waldorf Q V3.00R3 and Q+ V3.54R2:
> http://Synth.Stromeko.net/Downloads.html#WaldorfSDada
>
>

[-- Attachment #2: Type: text/html, Size: 4878 bytes --]

Re: [RFC] Org version of the Org manual

[Achim Gratz]

Jonathan Leech-Pepin writes:
> The error is actually on line 6069 of the manual. The {{{vindex
> [...]}}} line and subsequent paragraph. As far as the list is
> concerned there is no associated content for that list
> entry. Indenting them appropriately to be recognized as part of the
> list allows for successful export.

Thanks for tracking that down.

> This may also be partly a bug, should the exporter allow for a list
> item without any contents?

Well it should maybe not allow for it, but I think it should either
expect to get a nil in that situation or otherwsie not let the error
propagate.  Anyway, it's great to have such a large and complex document
available to train the exporter on.


Regards,
Achim.
-- 
+<[Q+ Matrix-12 WAVE#46+305 Neuron microQkb Andromeda XTk Blofeld]>+

Waldorf MIDI Implementation & additional documentation:
http://Synth.Stromeko.net/Downloads.html#WaldorfDocs

Re: [RFC] Org version of the Org manual

[Jonathan Leech-Pepin]

[-- Attachment #1: Type: text/plain, Size: 1416 bytes --]

On 10 March 2013 15:25, Achim Gratz <Stromeko@nexgo.de> wrote:

> Jonathan Leech-Pepin writes:
> > The error is actually on line 6069 of the manual. The {{{vindex
> > [...]}}} line and subsequent paragraph. As far as the list is
> > concerned there is no associated content for that list
> > entry. Indenting them appropriately to be recognized as part of the
> > list allows for successful export.
>
> Thanks for tracking that down.
>
> > This may also be partly a bug, should the exporter allow for a list
> > item without any contents?
>
> Well it should maybe not allow for it, but I think it should either
> expect to get a nil in that situation or otherwsie not let the error
> propagate.  Anyway, it's great to have such a large and complex document
> available to train the exporter on.
>

The error is not actually within ox-texinfo in this case however I suspect.
The exact same code (at the location of the error in ox-texinfo) appears
within
ox-latex and ox-ascii.

`(org-trim contents)', which removes whitespace at start and end of string,
in this case with the string being the contents of the list item.  When
there is
no contents there is no string to clean up.

Regards,

Jon

>
>
> Regards,
> Achim.
> --
> +<[Q+ Matrix-12 WAVE#46+305 Neuron microQkb Andromeda XTk Blofeld]>+
>
> Waldorf MIDI Implementation & additional documentation:
> http://Synth.Stromeko.net/Downloads.html#WaldorfDocs
>
>
>

[-- Attachment #2: Type: text/html, Size: 2104 bytes --]

Re: [RFC] Org version of the Org manual

[Nicolas Goaziou]

Hello,

Jonathan Leech-Pepin <jonathan.leechpepin@gmail.com> writes:

> The error is not actually within ox-texinfo in this case however I suspect.
> The exact same code (at the location of the error in ox-texinfo) appears
> within
> ox-latex and ox-ascii.
>
> `(org-trim contents)', which removes whitespace at start and end of string,
> in this case with the string being the contents of the list item.  When
> there is
> no contents there is no string to clean up.

I'm unsure about the part of the code you're referring to: in
ox-latex.el, there is (and contents (org-trim contents)), which
shouldn't generate an error.

Would you mind providing an ECM for the problem you're describing?

Thank you.


Regards,

-- 
Nicolas Goaziou

Re: [RFC] Org version of the Org manual

[Jonathan Leech-Pepin]

[-- Attachment #1: Type: text/plain, Size: 990 bytes --]

Hello,

On 10 March 2013 16:23, Nicolas Goaziou <n.goaziou@gmail.com> wrote:

> Hello,
>
> Jonathan Leech-Pepin <jonathan.leechpepin@gmail.com> writes:
>
> > The error is not actually within ox-texinfo in this case however I
> suspect.
> > The exact same code (at the location of the error in ox-texinfo) appears
> > within
> > ox-latex and ox-ascii.
> >
> > `(org-trim contents)', which removes whitespace at start and end of
> string,
> > in this case with the string being the contents of the list item.  When
> > there is
> > no contents there is no string to clean up.
>
> I'm unsure about the part of the code you're referring to: in
> ox-latex.el, there is (and contents (org-trim contents)), which
> shouldn't generate an error.


Oops.  I'd forgotten to include the (and contents ...) around the (org-trim
contents).

Sorry for the noise.

Regards,

Jon

> Would you mind providing an ECM for the problem you're describing?
>
> Thank you.
>
>
> Regards,
>
> --
> Nicolas Goaziou
>

[-- Attachment #2: Type: text/html, Size: 1653 bytes --]

Re: [RFC] Org version of the Org manual

[Thomas S. Dye]

Achim Gratz <Stromeko@nexgo.de> writes:

> Thomas S. Dye writes:
>> That works nicely.  I found the error and orgmanual.pdf is now produced
>> without errors.
>
> Progress! :-)
>
> With the current version from git I cannot export to texinfo

This is fixed now.

>
> I've extended the Makefile to approximate the one in doc/, HTML is
> produced both via makeinfo and as an export via ox-html.  To proceed in
> an orderly manner and prepare for an eventual integration into Org, can
> you please do the following in your Org clone:
>
> git checkout master
> git checkout -b orgmanual master
> git submodule add https://github.com/tsdye/orgmanual.git
> git commit -am 'make orgmanual/ a submodule'
>
> cd orgmanual
> git checkout -b orgmanual master
> git am orgmanual.patch
>
>
>
> cd ..
> git commit -am 'update submodule orgmanual'
> git am org.patch
>
>

This all worked fine.  Thanks for the detailed instructions.

>
> If you are unsure about any of this, please ask.  You can now edit/add
> these lines
>
> .PHONY: orgmanual
> EXTRADIRS=orgmanual
> orgmanual:
>         $(MAKE) -C $@
>
> to the top of your local.mk and should now be able to do a "make
> orgmanual".  Which types of documentation are produced can be controlled
> with ORG_MAKE_DOC (default is "info pdf html"), just like for the
> official manuals.  Also, "make cleanall" will now clean up in orgmanual
> also.  BTEST_POST should be configured to have a load-path pointing to a
> sufficiently advanced htmlize version for the HTML export.
>

I do have a problem here.  I think I have mis-configured BTEST_POST.

I have this:

BTEST_POST  =
              # -L <path-to>/ert      # needed for Emacs23, Emacs24 has ert built in
              # -L <path-to>/ess      # needed for running R tests
		-L contrib/lisp/htmlize.el  # need at least version 1.34 for source code formatting

Now 'make orgmanual' returns this:

local.mk:55: *** commands commence before first target.  Stop.

All the best,
Tom

-- 
Thomas S. Dye
http://www.tsdye.com

Re: [RFC] Org version of the Org manual

[Achim Gratz]

Thomas S. Dye writes:
> I do have a problem here.  I think I have mis-configured BTEST_POST.
>
> I have this:
>
> BTEST_POST  =
>               # -L <path-to>/ert      # needed for Emacs23, Emacs24 has ert built in
>               # -L <path-to>/ess      # needed for running R tests
> 		-L contrib/lisp/htmlize.el  # need at least version 1.34 for source code formatting

This should work (put it all on one line and drop the comments):

BTEST_POST  = -L ../contrib/lisp/htmlize.el

In general, any load-path given on the command line should be an
absolute path or you need to know which directory the invocation of
emacs happens in and make the path relative to that (./orgmanual in this
case).


Regards,
Achim.
-- 
+<[Q+ Matrix-12 WAVE#46+305 Neuron microQkb Andromeda XTk Blofeld]>+

Samples for the Waldorf Blofeld:
http://Synth.Stromeko.net/Downloads.html#BlofeldSamplesExtra

Re: [RFC] Org version of the Org manual

[Thomas S. Dye]

Aloha Achim,

Achim Gratz <Stromeko@nexgo.de> writes:

> Thomas S. Dye writes:
>> I do have a problem here.  I think I have mis-configured BTEST_POST.
>>
>> I have this:
>>
>> BTEST_POST  =
>>               # -L <path-to>/ert # needed for Emacs23, Emacs24 has
>> ert built in
>>               # -L <path-to>/ess      # needed for running R tests
>> 		-L contrib/lisp/htmlize.el # need at least version
>> 1.34 for source code formatting
>
> This should work (put it all on one line and drop the comments):
>
> BTEST_POST  = -L ../contrib/lisp/htmlize.el
>
> In general, any load-path given on the command line should be an
> absolute path or you need to know which directory the invocation of
> emacs happens in and make the path relative to that (./orgmanual in this
> case).

Thanks.  An absolute path works on one line with comments removed works.

All the best,
Tom

-- 
Thomas S. Dye
http://www.tsdye.com

Re: [RFC] Org version of the Org manual

[Achim Gratz]

Hi Tom,

I'm playing around with the current version, here is what I have so far:


The arguments to #+attr_texinfo need to lose the double quotes

Index entries in description lists must be separated from their entry by
at least one blank line like in the example below, otherwise the second
such entry gets indented in the exported .texi file and that somehow
confuses texinfo for some of them.

--8<---------------cut here---------------start------------->8---
- {{{kbd(v d)}}} or short {{{kbd(d)}}}, ~org-agenda-day-view~ ::

  {{{kindex(v d)}}}
  {{{kindex(d)}}}
  {{{vindex(org-agenda-span)}}}
--8<---------------cut here---------------end--------------->8---


With these hanges I get the manual exported correctly to all formats.


Regards,
Achim.
-- 
+<[Q+ Matrix-12 WAVE#46+305 Neuron microQkb Andromeda XTk Blofeld]>+

SD adaptation for Waldorf microQ V2.22R2:
http://Synth.Stromeko.net/Downloads.html#WaldorfSDada

Re: [RFC] Org version of the Org manual

[Thomas S. Dye]

Hi Achim,

Achim Gratz <Stromeko@nexgo.de> writes:

> Hi Tom,
>
> I'm playing around with the current version, here is what I have so far:
>
>
> The arguments to #+attr_texinfo need to lose the double quotes
>
> Index entries in description lists must be separated from their entry by
> at least one blank line like in the example below, otherwise the second
> such entry gets indented in the exported .texi file and that somehow
> confuses texinfo for some of them.
>
> - {{{kbd(v d)}}} or short {{{kbd(d)}}}, ~org-agenda-day-view~ ::
>
>   {{{kindex(v d)}}}
>   {{{kindex(d)}}}
>   {{{vindex(org-agenda-span)}}}
>
>
> With these hanges I get the manual exported correctly to all formats.

Done.  Thanks.

I brought the manual up-to-date and used @@info:@@ for the problem macro
characters, so it should be completely converted to Org now.

I want to finish editing the Installation instructions, then tag the
commit to indicate that it is "current" with org.texi.

Then, unless you think there is more to fix with the conversion, I'll
get down to documenting Nicolas' work according to the instructions he
gave me months ago. 

All the best,
Tom

-- 
Thomas S. Dye
http://www.tsdye.com

Re: [RFC] Org version of the Org manual

[Carsten Dominik]

On 17.3.2013, at 02:19, Thomas S. Dye <tsd@tsdye.com> wrote:

> Hi Achim,
> 
> Achim Gratz <Stromeko@nexgo.de> writes:
> 
>> Hi Tom,
>> 
>> I'm playing around with the current version, here is what I have so far:
>> 
>> 
>> The arguments to #+attr_texinfo need to lose the double quotes
>> 
>> Index entries in description lists must be separated from their entry by
>> at least one blank line like in the example below, otherwise the second
>> such entry gets indented in the exported .texi file and that somehow
>> confuses texinfo for some of them.
>> 
>> - {{{kbd(v d)}}} or short {{{kbd(d)}}}, ~org-agenda-day-view~ ::
>> 
>>  {{{kindex(v d)}}}
>>  {{{kindex(d)}}}
>>  {{{vindex(org-agenda-span)}}}
>> 
>> 
>> With these hanges I get the manual exported correctly to all formats.
> 
> Done.  Thanks.
> 
> I brought the manual up-to-date and used @@info:@@ for the problem macro
> characters, so it should be completely converted to Org now.


I tried the file on github, and export to texinfo fails with the attached backtrace (exports to other backends as well).

I love this idea, so I really want to try.  Any ideas what might be wrong on my side?

- Carsten

Debugger entered--Lisp error: (error "Unknown marker at 281861")
  signal(error ("Unknown marker at 281861"))
  error("Unknown marker at %d" 281861)
  org-element-text-markup-successor(281883)
  #[(res) "\302\303\304\b\"!	!\207" [res limit intern format "org-element-%s-successor"] 4](text-markup)
  mapcar(#[(res) "\302\303\304\b\"!	!\207" [res limit intern format "org-element-%s-successor"] 4] (export-snippet footnote-reference inline-babel-call inline-src-block latex-or-entity line-break link macro radio-target statistics-cookie sub/superscript target text-markup timestamp))
  org-element--get-next-object-candidates(281883 (export-snippet footnote-reference inline-babel-call inline-src-block latex-or-entity line-break link macro radio-target statistics-cookie sub/superscript target text-markup timestamp) initial)
  byte-code("\306\b	\n#\211\x12\203\230\x01\307\310\311\312\313\n\"\"\n\"\211^[A\fV\203$\0\314\315
\"\210\202\224\x01\vAb\210\316\317\320\v@\"! \x1e\x1d\321\x0e\x1d\x1e\x1e\x1e\x1f\x0e\x1e;\203I\0\322\323\x0e\x1f\x0e\x1e#\202Q\0\324\x0e\x1eA@\x0e\x1f\"*\x1e \325\x0e\x1d\x1e\x1e\x1e\x1f\x0e\x1e;\203k\0\322\323\x0e\x1f\x0e\x1e#\202s\0\324\x0e\x1eA@\x0e\x1f\"*\x1e!\326\x0e\x1d\x1e\x1e\x1e\x1f\x0e\x1e;\203\215\0\322\323\x0e\x1f\x0e\x1e#\202\225\0\324\x0e\x1eA@\x0e\x1f\"*\211\x1e\"\fX\203\340\0\x0e\"\bU\204\254\0\x0e\"b\210\202\223\x01\314\315\x0e\x1d\327
\x1e#\x1e\x1f\211\x1e\x1e;\203\311\0\330\x0e\x1e\331\x0e\x1f\x0e#$\202\332\0\x0e\x1eA\332\x0e\x1eA@\x0e\x1f\x0e##\240\210\x0e\x1e+\"\210\202\223\x01\x0e \203\363\0\x0e \fV\204\363\0\x0e!\fW\203'\x01\314\315\x0e\x1d\327
\x1e#\x1e\x1f\211\x1e\x1e;\203\x10\x01\330\x0e\x1e\331\x0e\x1f\x0e#$\202!\x01\x0e\x1eA\332\x0e\x1eA@\x0e\x1f\x0e##\240\210\x0e\x1e+\"\210\202\223\x01\x0e b\210\x0e\x1d\327
\x1e#\x1e\x1f\211\x1e\x1e;\203F\x01\330\x0e\x1e\331\x0e\x1f\x0e#$\202W\x01\x0e\x1eA\332\x0e\x1eA@\x0e\x1f\x0e##\240\210\x0e\x1e+\210\x0e\x1d\x15\x0e\x1d\211\x1e\x1e9\203j\x01\x0e\x1e\202\210\x01\x0e\x1e\211\x1e\x1e:\204}\x01\x0e\x1e;\205\207\x01\333\202\207\x01\x0e\x1e@9\205\207\x01\x0e\x1e@)\x0e$\236A)\x11\334\x12\x0e!\x10,)\202\0\0
\207" [end restriction candidates closest-cand origin parent org-element--get-next-object-candidates rassq apply min mapcar cdr throw exit intern format "org-element-%s-parser" :contents-begin get-text-property 0 plist-get :contents-end :end :parent org-add-props nil plist-put plain-text initial object element property cbeg cend obj-end value org-element-object-restrictions] 8)
  org-element-context()
  org-macro-replace-all((("modification-time" . "(eval (format-time-string \"$1\" '(20805 22656)))") ("input-file" . "om.org") ("time" . "(eval (format-time-string \"$1\"))") ("property" . "(eval (org-entry-get nil \"$1\" 'selective))") ("page" . "@@info:@page@@") ("result" . "@@info:@result{}@@") ("enddefopt" . "@@info:@end defopt@@") ("defopt" . "@@info:@defopt@@") ("enddefun" . "@@info:@end defun@@") ("defun" . "@@info:@defun@@") ("noindent" . "@@info:@noindent@@") ("kbdkeys" . "{{{kbd($1{{{key($2)}}}{{{key($3)}}})}}}") ("ksksksksk" . "{{{kbd($1 {{{key($2)}}} $3 {{{key($4)}}} {{{key($5)}}})}}}") ("ksksksk" . "{{{kbd($1 {{{key($2)}}} $3 {{{key($4)}}})}}}") ("kbdspckey" . "{{{kbd($1 {{{key($2)}}})}}}") ("kbdkey" . "{{{kbd($1{{{key($2)}}})}}}") ("printindex" . "@@info:@printindex $1@@") ("value" . "{{{markup(value,$1)}}}") ("cite" . "{{{markup(cite,$1)}}}") ("var" . "{{{markup(var,$1)}}}") ("file" . "{{{markup(file,$1)}}}") ("command" . "{{{markup(command,$1)}}}") ("samp" . "{{{markup(samp,$1)}}}") ("key" . "{{{markup(key,$1)}}}") ("kbd" . "{{{markup(kbd,$1)}}}") ("markup" . "@@info:@$1{@@$2@@info:}@@") ("kindex" . "{{{index(k,$1)}}}") ("vindex" . "{{{index(v,$1)}}}") ("findex" . "{{{index(f,$1)}}}") ("pindex" . "{{{index(p,$1)}}}") ("cindex" . "{{{index(c,$1)}}}") ("index" . "@@info:@$1index $2@@")))
  org-export-as(texinfo nil nil nil nil)
  org-export-to-file(texinfo "./om.texi" nil nil nil nil)
  org-texinfo-export-to-texinfo(nil nil nil nil)
  org-export-dispatch(nil)
  call-interactively(org-export-dispatch nil nil)


> 
> I want to finish editing the Installation instructions, then tag the
> commit to indicate that it is "current" with org.texi.
> 
> Then, unless you think there is more to fix with the conversion, I'll
> get down to documenting Nicolas' work according to the instructions he
> gave me months ago. 
> 
> All the best,
> Tom
> 
> -- 
> Thomas S. Dye
> http://www.tsdye.com
> 

Re: [RFC] Org version of the Org manual

[Achim Gratz]

Carsten Dominik writes:
> I tried the file on github, and export to texinfo fails with the
> attached backtrace (exports to other backends as well).

Works for me.

> I love this idea, so I really want to try.  Any ideas what might be
> wrong on my side?
[…]
>   org-export-to-file(texinfo "./om.texi" nil nil nil nil)

It seems you are not using the build system?
You need to set at least 

(add-to-list 'org-export-snippet-translation-alist
             '("info" . "texinfo"))


Regards,
Achim.
-- 
+<[Q+ Matrix-12 WAVE#46+305 Neuron microQkb Andromeda XTk Blofeld]>+

SD adaptations for Waldorf Q V3.00R3 and Q+ V3.54R2:
http://Synth.Stromeko.net/Downloads.html#WaldorfSDada

Re: [RFC] Org version of the Org manual

[Carsten Dominik]

On 17.3.2013, at 07:54, Achim Gratz <Stromeko@nexgo.de> wrote:

> Carsten Dominik writes:
>> I tried the file on github, and export to texinfo fails with the
>> attached backtrace (exports to other backends as well).
> 
> Works for me.
> 
>> I love this idea, so I really want to try.  Any ideas what might be
>> wrong on my side?
> […]
>>  org-export-to-file(texinfo "./om.texi" nil nil nil nil)
> 
> It seems you are not using the build system?

Hi Achim, not sure what exactly you mean.  I do compile using make which I think is what you mean?

> You need to set at least 
> 
> (add-to-list 'org-export-snippet-translation-alist
>             '("info" . "texinfo"))

Have that now, does not fix the issue.

- Carsten

Re: [RFC] Org version of the Org manual

[Achim Gratz]

Carsten Dominik writes:
>> It seems you are not using the build system?

> Hi Achim, not sure what exactly you mean.  I do compile using make
> which I think is what you mean?

It seems you're not producing orgmanual.texi with make, since I can't
see how it would use "om.texi" as the target filename.  Anyway, the
latest changes from Tom are only in the orgmanual branch of the github
repo.  You'll need to complement that with the changes to Org that I
posted earlier for him to try.


Regards,
Achim.
-- 
+<[Q+ Matrix-12 WAVE#46+305 Neuron microQkb Andromeda XTk Blofeld]>+

Factory and User Sound Singles for Waldorf Q+, Q and microQ:
http://Synth.Stromeko.net/Downloads.html#WaldorfSounds

Re: [RFC] Org version of the Org manual

[Carsten Dominik]

On 17.3.2013, at 14:34, Achim Gratz <Stromeko@nexgo.de> wrote:

> Carsten Dominik writes:
>>> It seems you are not using the build system?
> 
>> Hi Achim, not sure what exactly you mean.  I do compile using make
>> which I think is what you mean?
> 
> It seems you're not producing orgmanual.texi with make,

You are right, I am not, did not know that the Makefile already supports making the manual in this way.  Don't see how either.

> since I can't
> see how it would use "om.texi" as the target filename.  Anyway, the
> latest changes from Tom are only in the orgmanual branch of the github
> repo.  You'll need to complement that with the changes to Org that I
> posted earlier for him to try.

So these are not yet in his github repository?

I am a bit confused, so I give up for now, maybe you or Tom can send around some instructions once there is a version accessible that will work out of the box.

Thanks

- Carsten

Re: [RFC] Org version of the Org manual

[Achim Gratz]

Carsten Dominik writes:
> So these are not yet in his github repository?

Yes they are, but not in the master branch.  You'll want to switch to
the orgmanual branch for now.

> I am a bit confused, so I give up for now, maybe you or Tom can send
> around some instructions once there is a version accessible that will
> work out of the box.

This has been described elsewhere in this thread, it seems you missed
that part:

http://permalink.gmane.org/gmane.emacs.orgmode/68117

Now that there is an orgmanual branch already in Tom's repo on github,
simply clone that one and the patch that instantiates the Makefile is
already applied:

cd orgmanual
git checkout -b orgmanual orgmanual

You still need to branch the Org repo and add the patch there, plus
modify your local.mk, but then you'll be able to let the build system
produce the manual for you.


Regards,
Achim.
-- 
+<[Q+ Matrix-12 WAVE#46+305 Neuron microQkb Andromeda XTk Blofeld]>+

Factory and User Sound Singles for Waldorf rackAttack:
http://Synth.Stromeko.net/Downloads.html#WaldorfSounds

Re: [RFC] Org version of the Org manual

[Carsten Dominik]

Hi Achim,

thank you for the additional information.
On 17.3.2013, at 18:36, Achim Gratz <Stromeko@nexgo.de> wrote:

> Carsten Dominik writes:
>> So these are not yet in his github repository?
> 
> Yes they are, but not in the master branch.  You'll want to switch to
> the orgmanual branch for now.
> 
>> I am a bit confused, so I give up for now, maybe you or Tom can send
>> around some instructions once there is a version accessible that will
>> work out of the box.
> 
> This has been described elsewhere in this thread, it seems you missed
> that part:

Quite likely, I only spontaneously tried to compile the new manual and wanted to go the route with C-c C-e i t or something like this.  Inclusion in the built system is great, if this project succeeds.

I think I am doing everything right, but I still get the marker error.  So I fear something in my config might be acting up - there are several people who have reported now that they have produced the manual successfully.  Will try to isolate my issue.

Thanks!

- Carsten

> 
> http://permalink.gmane.org/gmane.emacs.orgmode/68117
> 
> Now that there is an orgmanual branch already in Tom's repo on github,
> simply clone that one and the patch that instantiates the Makefile is
> already applied:
> 
> cd orgmanual
> git checkout -b orgmanual orgmanual
> 
> You still need to branch the Org repo and add the patch there, plus
> modify your local.mk, but then you'll be able to let the build system
> produce the manual for you.
> 
> 
> Regards,
> Achim.
> -- 
> +<[Q+ Matrix-12 WAVE#46+305 Neuron microQkb Andromeda XTk Blofeld]>+
> 
> Factory and User Sound Singles for Waldorf rackAttack:
> http://Synth.Stromeko.net/Downloads.html#WaldorfSounds
> 
> 

Re: [RFC] Org version of the Org manual

[Sebastien Vauban]

Hi Carsten,

Carsten Dominik wrote:
> On 17.3.2013, at 02:19, Thomas S. Dye <tsd-P0awH739Ni4AvxtiuMwx3w@public.gmane.org> wrote:
>
>> Hi Achim,
>> 
>> Achim Gratz <Stromeko-i47jiTeKxPI@public.gmane.org> writes:
>> 
>>> Hi Tom,
>>> 
>>> I'm playing around with the current version, here is what I have so far:
>>> 
>>> 
>>> The arguments to #+attr_texinfo need to lose the double quotes
>>> 
>>> Index entries in description lists must be separated from their entry by
>>> at least one blank line like in the example below, otherwise the second
>>> such entry gets indented in the exported .texi file and that somehow
>>> confuses texinfo for some of them.
>>> 
>>> - {{{kbd(v d)}}} or short {{{kbd(d)}}}, ~org-agenda-day-view~ ::
>>> 
>>>  {{{kindex(v d)}}}
>>>  {{{kindex(d)}}}
>>>  {{{vindex(org-agenda-span)}}}
>>> 
>>> 
>>> With these hanges I get the manual exported correctly to all formats.
>> 
>> Done.  Thanks.
>> 
>> I brought the manual up-to-date and used @@info:@@ for the problem macro
>> characters, so it should be completely converted to Org now.
>
>
> I tried the file on github, and export to texinfo fails with the attached backtrace (exports to other backends as well).
>
> I love this idea, so I really want to try.  Any ideas what might be wrong on my side?
>
> - Carsten
>
> Debugger entered--Lisp error: (error "Unknown marker at 281861")

Could you try M-x goto-char 281861?

I have such a msg in old files where I had my own @ markes.

Best regards,
  Seb

-- 
Sebastien Vauban

Re: [RFC] Org version of the Org manual

[Carsten Dominik]

On 17.3.2013, at 08:01, "Sebastien Vauban" <wxhgmqzgwmuf@spammotel.com> wrote:

> Hi Carsten,
> 
> Carsten Dominik wrote:
>> On 17.3.2013, at 02:19, Thomas S. Dye <tsd@tsdye.com> wrote:
>> 
>>> Hi Achim,
>>> 
>>> Achim Gratz <Stromeko@nexgo.de> writes:
>>> 
>>>> Hi Tom,
>>>> 
>>>> I'm playing around with the current version, here is what I have so far:
>>>> 
>>>> 
>>>> The arguments to #+attr_texinfo need to lose the double quotes
>>>> 
>>>> Index entries in description lists must be separated from their entry by
>>>> at least one blank line like in the example below, otherwise the second
>>>> such entry gets indented in the exported .texi file and that somehow
>>>> confuses texinfo for some of them.
>>>> 
>>>> - {{{kbd(v d)}}} or short {{{kbd(d)}}}, ~org-agenda-day-view~ ::
>>>> 
>>>> {{{kindex(v d)}}}
>>>> {{{kindex(d)}}}
>>>> {{{vindex(org-agenda-span)}}}
>>>> 
>>>> 
>>>> With these hanges I get the manual exported correctly to all formats.
>>> 
>>> Done.  Thanks.
>>> 
>>> I brought the manual up-to-date and used @@info:@@ for the problem macro
>>> characters, so it should be completely converted to Org now.
>> 
>> 
>> I tried the file on github, and export to texinfo fails with the attached backtrace (exports to other backends as well).
>> 
>> I love this idea, so I really want to try.  Any ideas what might be wrong on my side?
>> 
>> - Carsten
>> 
>> Debugger entered--Lisp error: (error "Unknown marker at 281861")
> 
> Could you try M-x goto-char 281861?

You mean in orgmanual.org?


It jumos to "see" in
---------------------------------------------------------
- %:keyword ::   

  Specific information for certain link types, see below.
----------------------------------------------------------

> 
> I have such a msg in old files where I had my own @ markes.
> 
> Best regards,
>  Seb
> 
> -- 
> Sebastien Vauban
> 
> 

Re: [RFC] Org version of the Org manual

[Achim Gratz]

[-- Attachment #1: Type: text/plain, Size: 97 bytes --]

Hi Tom,

I have a patch that should fix your problems with some characters in
macro expansions:


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: 0001-fix-macro-expansion-with-separators-and-backslashes.patch --]
[-- Type: text/x-patch, Size: 2589 bytes --]

From 27b22d17f629a50bd485a0320dac45616d7ceb7f Mon Sep 17 00:00:00 2001
From: Achim Gratz <Stromeko@Stromeko.DE>
Date: Sun, 17 Mar 2013 10:20:10 +0100
Subject: [PATCH] fix macro expansion with separators and backslashes

* lisp/org-element.el (org-element-macro-parser): Do not try to
  "repair bad splits", only split at the correct places.
* lisp/org-macro.el (org-macro-expand): Do not try to interpret the
  macro replacement text as a regex.

Allow to write macros like {{{kbd(\\)}}} and {{{kbd(\\,)}}} and expand
them correctly.  A backslash at the end of an argument was incorrectly
trying to cons the next argument (which may not exist).
---
 lisp/org-element.el | 16 +++++-----------
 lisp/org-macro.el   |  2 +-
 2 files changed, 6 insertions(+), 12 deletions(-)

diff --git a/lisp/org-element.el b/lisp/org-element.el
index ba2461a..337cad0 100644
--- a/lisp/org-element.el
+++ b/lisp/org-element.el
@@ -3117,20 +3117,14 @@ (defun org-element-macro-parser ()
 	  (post-blank (progn (goto-char (match-end 0))
 			     (skip-chars-forward " \t")))
 	  (end (point))
-	  (args (let ((args (org-match-string-no-properties 3)) args2)
+	  (args (let ((args (org-match-string-no-properties 3)))
 		  (when args
 		    ;; Do not use `org-split-string' since empty
 		    ;; strings are meaningful here.
-		    (setq args (split-string args ","))
-		    (while args
-		      (while (string-match "\\\\\\'" (car args))
-			;; Repair bad splits, when comma is protected,
-                        ;; and thus not a real separator.
-			(setcar (cdr args) (concat (substring (car args) 0 -1)
-						   "," (nth 1 args)))
-			(pop args))
-		      (push (pop args) args2))
-		    (mapcar 'org-trim (nreverse args2))))))
+		    (setq args (replace-regexp-in-string "," "\000" args))
+		    (setq args (replace-regexp-in-string "\\\\\000" "," args))
+		    (setq args (split-string args "\000"))
+		    (mapcar 'org-trim args)))))
       (list 'macro
 	    (list :key key
 		  :value value
diff --git a/lisp/org-macro.el b/lisp/org-macro.el
index 88cbdf7..3e07295 100644
--- a/lisp/org-macro.el
+++ b/lisp/org-macro.el
@@ -137,7 +137,7 @@ (defun org-macro-expand (macro templates)
                                (org-element-property :args macro))
                           ;; No argument: remove place-holder.
                           ""))
-                    template)))
+                    template nil 'literal)))
         ;; VALUE starts with "(eval": it is a s-exp, `eval' it.
         (when (string-match "\\`(eval\\>" value)
           (setq value (eval (read value))))
-- 
1.8.1.4


[-- Attachment #3: Type: text/plain, Size: 226 bytes --]


I hope Nicolas will apply it or make a better patch.  The quoting used
unfortunately is not idempotent, so you need to know how many times your
macro will get expanded, but it is a start.  You should now be able to
do this:


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #4: 0001-put-back-macros-with-problematic-characters.patch --]
[-- Type: text/x-patch, Size: 2827 bytes --]

From 042ad50bd06cbc5ab97c753cb5cfce8908121242 Mon Sep 17 00:00:00 2001
From: Achim Gratz <Stromeko@Stromeko.DE>
Date: Sun, 17 Mar 2013 11:27:21 +0100
Subject: [PATCH] put back macros with "problematic characters"

---
 orgmanual.org | 30 +++++++++---------------------
 1 file changed, 9 insertions(+), 21 deletions(-)

diff --git a/orgmanual.org b/orgmanual.org
index 6aa6b81..d50b20b 100644
--- a/orgmanual.org
+++ b/orgmanual.org
@@ -4254,13 +4254,10 @@ Priorities can be attached to any outline node; they do not need to be TODO
 items.
 
 #+attr_texinfo: :table-type table :indic @asis
-  - {{{kbd(C-c XXX)}}} ::
+  - {{{kbd(C-c \\,)}}} ::
 
-    {{{kindex(C-c ,)}}}
-    #+comment:    {{{kindex(@kbd{C-c ,})}}}
-    #+comment: Preceding line won't export to pdf
-    {{{findex(org-priority)}}}
-    #+comment: Should be C-c ,
+       {{{kindex(C-c \\,)}}}
+       {{{findex(org-priority)}}}
 
     Set the priority of the current headline (~org-priority~). The
     command prompts for a priority character {{{samp(A)}}}, {{{samp(B)}}}
@@ -9226,9 +9223,7 @@ the other commands, the cursor needs to be in the desired line.
 
 - ~\~ ~org-agenda-filter-by-tag-refine~ ::
 
-  #+comment: {{{kindex(XXX)}}}
-  #+comment: Should be \
-  @@info:kindex \\@@
+  {{{kindex(\)}}}
 
   Narrow the current agenda filter by an additional condition. When
   called with prefix arg, remove the entries that /do/ have the tag, or
@@ -9350,11 +9345,9 @@ the other commands, the cursor needs to be in the desired line.
   Set tags for the current headline. If there is an active region in the
   agenda, change a tag for all headings in the region.
 
-- {{{kbd(,)}}} ::
+- {{{kbd(\\,)}}} ::
 
-  #+comment: {{{kindex(XXX)}}}
-  #+comment: Should be a comma
-  @@info:kindex ,@@
+  {{{kindex(\\,)}}}
 
   Set the priority for the current item (~org-agenda-priority~). Org
   mode prompts for the priority character. If you reply with
@@ -10691,12 +10684,9 @@ If you would like to see entities displayed as UTF8 characters, use the
 following command:[fn:108]
 
 #+attr_texinfo: :table-type table :indic @asis
-- @@info:@kbd{C-c C-x XXX}@@ ::
-
-  #+comment: {{{kindex(C-c C-x XXX)}}}
-  #+comment: Should be \
+- {{{kbd(C-c C-x \)}}} ::
   
-  @@info:kindex C-c C-x \\
+  {{{kindex(C-c C-x \)}}}
 
   Toggle display of entities as UTF-8 characters. This does not change
   the buffer content which remains plain ASCII, but it overlays the
@@ -10740,9 +10730,7 @@ subscript, but ~a_{b}~ will.
 #+attr_texinfo: :table-type table :indic @asis
 - @@info:@kbd{C-c C-x \\}@@ ::
 
-  #+comment: {{{kindex(C-c C-x XXX)}}}
-  #+comment: Should be \
-  @@info:@kindex C-c C-x \@@
+  {{{kindex(C-c C-x \)}}}
 
   In addition to showing entities as UTF-8 characters, this command will
   also format sub- and superscripts in a WYSIWYM way.
-- 
1.8.1.4


[-- Attachment #5: Type: text/plain, Size: 200 bytes --]




Regards,
Achim.
-- 
+<[Q+ Matrix-12 WAVE#46+305 Neuron microQkb Andromeda XTk Blofeld]>+

SD adaptations for Waldorf Q V3.00R3 and Q+ V3.54R2:
http://Synth.Stromeko.net/Downloads.html#WaldorfSDada

Re: [RFC] Org version of the Org manual

[Nicolas Goaziou]

Hello,

Achim Gratz <Stromeko@nexgo.de> writes:

> Hi Tom,
>
> I have a patch that should fix your problems with some characters in
> macro expansions:
>
>
> From 27b22d17f629a50bd485a0320dac45616d7ceb7f Mon Sep 17 00:00:00 2001
> From: Achim Gratz <Stromeko@Stromeko.DE>
> Date: Sun, 17 Mar 2013 10:20:10 +0100
> Subject: [PATCH] fix macro expansion with separators and backslashes
>
> * lisp/org-element.el (org-element-macro-parser): Do not try to
>   "repair bad splits", only split at the correct places.
> * lisp/org-macro.el (org-macro-expand): Do not try to interpret the
>   macro replacement text as a regex.
>
> Allow to write macros like {{{kbd(\\)}}} and {{{kbd(\\,)}}} and expand
> them correctly.  A backslash at the end of an argument was incorrectly
> trying to cons the next argument (which may not exist).
> ---
>  lisp/org-element.el | 16 +++++-----------
>  lisp/org-macro.el   |  2 +-
>  2 files changed, 6 insertions(+), 12 deletions(-)
>
> diff --git a/lisp/org-element.el b/lisp/org-element.el
> index ba2461a..337cad0 100644
> --- a/lisp/org-element.el
> +++ b/lisp/org-element.el
> @@ -3117,20 +3117,14 @@ (defun org-element-macro-parser ()
>  	  (post-blank (progn (goto-char (match-end 0))
>  			     (skip-chars-forward " \t")))
>  	  (end (point))
> -	  (args (let ((args (org-match-string-no-properties 3)) args2)
> +	  (args (let ((args (org-match-string-no-properties 3)))
>  		  (when args
>  		    ;; Do not use `org-split-string' since empty
>  		    ;; strings are meaningful here.
> -		    (setq args (split-string args ","))
> -		    (while args
> -		      (while (string-match "\\\\\\'" (car args))
> -			;; Repair bad splits, when comma is protected,
> -                        ;; and thus not a real separator.
> -			(setcar (cdr args) (concat (substring (car args) 0 -1)
> -						   "," (nth 1 args)))
> -			(pop args))
> -		      (push (pop args) args2))
> -		    (mapcar 'org-trim (nreverse args2))))))
> +		    (setq args (replace-regexp-in-string "," "\000" args))
> +		    (setq args (replace-regexp-in-string "\\\\\000" "," args))
> +		    (setq args (split-string args "\000"))
> +		    (mapcar 'org-trim args)))))

I suggest the following code instead, which allows to escape the
escaping backslash so the comma is not escaped:

  (args (mapcar 'org-trim
                (split-string
                 (replace-regexp-in-string
                  "\\(\\\\+\\)?\\(,\\)"
                  (lambda (str)
                    (let ((slashes (match-string 1 str)))
                      (if (or (not slashes) (evenp (length slashes))) "\\1\000"
                        (concat (make-string (1- (length slashes)) ?\\) ","))))
                  (org-match-string-no-properties 3))
                 "\000")))

What do you think about it?

> @@ -137,7 +137,7 @@ (defun org-macro-expand (macro templates)
>                                 (org-element-property :args macro))
>                            ;; No argument: remove place-holder.
>                            ""))
> -                    template)))
> +                    template nil 'literal)))

I agree on that part.

Also, a test or two should be added to "test-org-element/macro-parser"
in test-org-element.el.


Regards,

-- 
Nicolas Goaziou

Re: [RFC] Org version of the Org manual

[Achim Gratz]

Am 21.03.2013 22:02, schrieb Nicolas Goaziou:
> I suggest the following code instead, which allows to escape the
> escaping backslash so the comma is not escaped:
>
>    (args (mapcar 'org-trim
>                  (split-string
>                   (replace-regexp-in-string
>                    "\\(\\\\+\\)?\\(,\\)"
>                    (lambda (str)
>                      (let ((slashes (match-string 1 str)))
>                        (if (or (not slashes) (evenp (length slashes))) "\\1\000"
>                          (concat (make-string (1- (length slashes)) ?\\) ","))))
>                    (org-match-string-no-properties 3))
>                   "\000")))
>
> What do you think about it?

I think this is a lot harder to understand and I would guess it is also 
quite a bit slower.  Also I'm not sure why you are trying to match 
multiple backslashes.  The original implementation and the description 
of the syntax says that the only character that can be escaped is a 
comma, so the new implementation changes behaviour in that regard (maybe 
intentionally, I can't tell).


Regards,
-- 
Achim.

(on the road :-)

Re: [RFC] Org version of the Org manual

[Nicolas Goaziou]

Hello,

Achim Gratz <Stromeko@Nexgo.DE> writes:

> Am 21.03.2013 22:02, schrieb Nicolas Goaziou:
>> I suggest the following code instead, which allows to escape the
>> escaping backslash so the comma is not escaped:
>>
>>    (args (mapcar 'org-trim
>>                  (split-string
>>                   (replace-regexp-in-string
>>                    "\\(\\\\+\\)?\\(,\\)"
>>                    (lambda (str)
>>                      (let ((slashes (match-string 1 str)))
>>                        (if (or (not slashes) (evenp (length slashes))) "\\1\000"
>>                          (concat (make-string (1- (length slashes)) ?\\) ","))))
>>                    (org-match-string-no-properties 3))
>>                   "\000")))
>>
>> What do you think about it?
>
> I think this is a lot harder to understand

Actually it is harder to understand because it doesn't make any sense:
the code is wrong. More on this below.

> and I would guess it is also quite a bit slower.

Speed difference is not significant here.

> Also I'm not sure why you are trying to match multiple backslashes.
> The original implementation and the description of the syntax says
> that the only character that can be escaped is a comma, so the new
> implementation changes behaviour in that regard (maybe intentionally,
> I can't tell).

Actually the code I pasted is wrong, I meant:

  (split-string
   (replace-regexp-in-string
    "\\(\\\\+\\)?\\(,\\)"
    (lambda (str)
      (let ((len (length (match-string 1 str))))
        (if (evenp len) (concat (make-string (/ len 2) ?\\) "\000")
          (concat (make-string (/ (1- len) 2) ?\\) ","))))
    (org-matcĥ-string-no-properties 3) nil t)
   "\000")

With the current implementation (and in your refactoring), it is
impossible to have '("a\,b").

If you allow to escape a character, you should also be able to escape
the escaping character. With this patch:

  "a,b"     -> '("a" "b")
  "a\,b"    -> '("a,b")
  "a\\,b"   -> '("a\" "b")
  "a\\\,b"  -> '("a\,b")
  "a\\\\,b" -> '(a"\\" "b")

Note that with the patch, you only need to escape backslashes before
a comma:

  "a\\b\,c" -> '("a\\b,c")

If consistency is a matter, a slightly different patch can require to
escape every backslash character. Though, I don't think it is necessary.


Regards,

-- 
Nicolas Goaziou

Re: [RFC] Org version of the Org manual

[Achim Gratz]

Am 22.03.2013 15:22, schrieb Nicolas Goaziou:
> Actually the code I pasted is wrong, I meant:
>
>    (split-string
>     (replace-regexp-in-string
>      "\\(\\\\+\\)?\\(,\\)"
>      (lambda (str)
>        (let ((len (length (match-string 1 str))))
>          (if (evenp len) (concat (make-string (/ len 2) ?\\) "\000")
>            (concat (make-string (/ (1- len) 2) ?\\) ","))))
>      (org-matcĥ-string-no-properties 3) nil t)
>     "\000")

That is starting to make sense... :-)
You might want to use the (function (lambda ...)) form.

> With the current implementation (and in your refactoring), it is
> impossible to have '("a\,b").

Yes, you'd need a second form of quoting to properly support that.  It's 
also impossible to have leading or trailing whitespace, so I don't know 
how important it is to support this edge case of "\,".

> If consistency is a matter, a slightly different patch can require to
> escape every backslash character. Though, I don't think it is necessary.

I'm not sure it is worth the trouble.


Regards,
-- 
Achim.

(on the road :-)

Re: [RFC] Org version of the Org manual

[Nicolas Goaziou]

Achim Gratz <Stromeko@Nexgo.DE> writes:

> Am 22.03.2013 15:22, schrieb Nicolas Goaziou:
>> Actually the code I pasted is wrong, I meant:
>>
>>    (split-string
>>     (replace-regexp-in-string
>>      "\\(\\\\+\\)?\\(,\\)"
>>      (lambda (str)
>>        (let ((len (length (match-string 1 str))))
>>          (if (evenp len) (concat (make-string (/ len 2) ?\\) "\000")
>>            (concat (make-string (/ (1- len) 2) ?\\) ","))))
>>      (org-matcĥ-string-no-properties 3) nil t)
>>     "\000")
>
> That is starting to make sense... :-)
> You might want to use the (function (lambda ...)) form.

Why? (function (lambda ...)) is equivalent to (lambda ...). In fact,
`lambda' uses `function' as a subroutine.

>> With the current implementation (and in your refactoring), it is
>> impossible to have '("a\,b").
>
> Yes, you'd need a second form of quoting to properly support that.
> It's also impossible to have leading or trailing whitespace, 

That's why I also suggest to drop the (mapcar 'org-trim ...) wrapper.
There's no reason to remove blanks if the user inserted them in the
first place.

> so I don't know how important it is to support this edge case of "\,".

Since that's not costly to achieve, there no real reason not to support
it.

So, any objection (or further comment) for that change?


Regards,

-- 
Nicolas Goaziou

Re: [RFC] Org version of the Org manual

[Achim Gratz]

Am 22.03.2013 19:17, schrieb Nicolas Goaziou:
>> You might want to use the (function (lambda ...)) form.
>
> Why? (function (lambda ...)) is equivalent to (lambda ...). In fact,
> `lambda' uses `function' as a subroutine.

http://www.gnu.org/software/emacs/manual/html_node/elisp/Anonymous-Functions.html

http://stackoverflow.com/questions/1852844/emacs-lisp-difference-between-function-lambda-and-lambda

With Emacs 24.3 and eager macro expoansion the need to do this has 
diminished, but for the benefit of older Emacsen we should probably 
continue the practise.

> That's why I also suggest to drop the (mapcar 'org-trim ...) wrapper.
> There's no reason to remove blanks if the user inserted them in the
> first place.

Oh, OK - I did not realise that.

> Since that's not costly to achieve, there no real reason not to support
> it.

I'll trust you on the "not costly" part, I can't do any experiment right 
now.

> So, any objection (or further comment) for that change?

Please go ahead.


Regards,
-- 
Achim.

(on the road :-)

Re: [RFC] Org version of the Org manual

[Nicolas Goaziou]

Hello,

Achim Gratz <Stromeko@Nexgo.DE> writes:

> Am 22.03.2013 19:17, schrieb Nicolas Goaziou:
>>> You might want to use the (function (lambda ...)) form.
>>
>> Why? (function (lambda ...)) is equivalent to (lambda ...). In fact,
>> `lambda' uses `function' as a subroutine.
>
> http://www.gnu.org/software/emacs/manual/html_node/elisp/Anonymous-Functions.html
>
> http://stackoverflow.com/questions/1852844/emacs-lisp-difference-between-function-lambda-and-lambda
>
> With Emacs 24.3 and eager macro expoansion the need to do this has
> diminished, but for the benefit of older Emacsen we should probably
> continue the practise.

According to the link you pasted:

  The lambda form has one other effect: it tells the Emacs evaluator and
  byte-compiler that its argument is a function, by using function as
  a subroutine (see below).

I don't think this is a recent addition (recent as in "since Emacs
24.1").

>> So, any objection (or further comment) for that change?
>
> Please go ahead.

Done. Could you commit the other part of your patch?


Regards,

-- 
Nicolas Goaziou

Re: [RFC] Org version of the Org manual

[Achim Gratz]

Hi Nicolas,

the change in org-macro.el to use a literal replacement string is still
missing in mainline Org.  Is there something wrong with that patch or
should I apply it?


Regards,
Achim.
-- 
+<[Q+ Matrix-12 WAVE#46+305 Neuron microQkb Andromeda XTk Blofeld]>+

SD adaptations for KORG EX-800 and Poly-800MkII V0.9:
http://Synth.Stromeko.net/Downloads.html#KorgSDada

Re: [RFC] Org version of the Org manual

[Nicolas Goaziou]

Hello,

Achim Gratz <Stromeko@nexgo.de> writes:

> the change in org-macro.el to use a literal replacement string is still
> missing in mainline Org.  Is there something wrong with that patch or
> should I apply it?

IIRC I had told you that you could apply it. If that's not the case, you
can apply it anyway ;)

Thanks.


Regards,

-- 
Nicolas Goaziou

Re: [RFC] Org version of the Org manual

[Achim Gratz]

Nicolas Goaziou writes:
> IIRC I had told you that you could apply it. If that's not the case,
> you can apply it anyway ;)

Done in maint and master.


Regards,
Achim.
-- 
+<[Q+ Matrix-12 WAVE#46+305 Neuron microQkb Andromeda XTk Blofeld]>+

Wavetables for the Waldorf Blofeld:
http://Synth.Stromeko.net/Downloads.html#BlofeldUserWavetables