* Re: iOrg
2016-09-14 11:34 iOrg (was: Programmatically handling org files) Philip Hudson
@ 2016-09-14 22:06 ` Thorsten Jolitz
2016-09-15 17:36 ` iOrg Adam Porter
0 siblings, 1 reply; 5+ messages in thread
From: Thorsten Jolitz @ 2016-09-14 22:06 UTC (permalink / raw)
To: emacs-orgmode
Philip Hudson <phil.hudson@iname.com> writes:
[WARNING: this is an extremely long post with lots of boxquotes that
might turn out unreadable, you might want to consider this as spam
and just ignore it]
Hi Philip,
> On 12 September 2016 at 23:10, Nick Dokos <ndokos@gmail.com> wrote:
>> Thorsten Jolitz <tjolitz@gmail.com> writes:
>>
>>> ...
>>>
>>> If you feel you don't need the whole parse tree, but rather want to act
>>> locally on the Org element at point, you might want to look at
>>> org-dp.el
>>> ...
>>
>> Just a note for the benefit of newcomers to the list: Thorsten's
>> org-dp.el (and many other goodies) are on his github pages:
>>
>> https://github.com/tj64/org-dp
>> https://github.com/tj64?tab=repositories
>
> Just had a look, and I think his "iOrg" looks like the Holy Grail of
> Org: round-trip web-enabled org-mode. Is that right? Has anyone used
> it? Is there a demo site somewhere? Very exciting!
I still think its a splendid idea to bring my two favorite software
systems together (Emacs(Org-mode) and PicoLisp), and iOrg is actually
very close to being usable (and, I would guess, pretty useful).
Events in my life kept me away from Emacs for more than a year, and
recently I used it only occasionally, but now winter comes and the long,
dark and cold evenings ...
I'm (slowly slowly) converting to impressive and exciting Spacemacs, I
would like to create a Spacemacs layer for the outshine-libraries - and
I would like to start again working on iOrg, its such a promising
project.
There is a lot of functionality already there, I don't even remember all
the stuff I wanted and implemented. I'm on Archlinux with tmux and no X,
so I only tried it with emacs-w3m, an it does not look too bad there.
What I can do here locally:
- start the iOrg Server (in Production or Debug mode, the latter with
+), this starts an Emacs daemon too
,----
| [tj@arch old-picoLisp]$ ./pil iorg/main.l -main -go +
| # url> +Role redefined
| # +User em redefined
| # url> +User redefined
|
| Warning: due to a long standing Gtk+ bug
| http://bugzilla.gnome.org/show_bug.cgi?id=85715
| Emacs might crash when run in daemon mode and the X11 connection is
| unexpectedly lost.
| Using an Emacs configured with --with-x-toolkit=lucid does not have
| this problem.
| Starting Emacs daemon.
`----
- open http://localhost:5001 in emacs-w3m and login at the iOrg Wiki
startpage
,----
| Location: http://localhost:5001/
| …20:15 UTC
|
| * iOrg
| * Home
| * logout
| * ----------------------------------------------------------------
| * Documents
| * Org Data
| * Org Headlines
| * Role Administration
| * User Administration
| * Upload
| * Help
|
| iOrg Logo
|
| Name [ ]
| Password [ ]
|
| [login]
`----
- login
,----
| Name [admin ]
| Password [ ]
|
| [logout] 'admin' logged in
`----
- read and edit the wiki pages under node iOrg.
This is all based on the PicoLisp wiki from amazing PicoLisp creator
Alexander Burger, I replaced the PicoLisp wiki syntax (which isn't
bad at all and has its own Emacs mode written by me) with Org-mode.
read:
,----
| Location: http://localhost:43051/19189021633385603~!iorg?home?*Menu=+3&*Tab=+1&*ID=
| …00:14 UTCiOrg-Wiki Logo
| [ ] [Search]
|
| Log out
| My account
|
| * Start Page
|
| * FAQ
|
| * Tutorial
|
| + Assumptions
| + Web UI
| + Emacs UI
| + PicoLisp Database
| + Markup for DHTML
| * Org-mode Home
|
| * Org-mode Wiki
|
| * PicoLisp Wiki
|
| * Help
|
| Edit Reply
|
| High Level Introduction to iOrg
|
| iOrg Application Framework
|
| Merging two of the most exciting Lisp applications around
| into one powerful Web Application Framework
|
| iOrg is based on the PicoLisp Wiki, a rather small but highly
| functional web-application developed with the incredibly expressive
| PicoLisp Application Framework. By replacing the original PicoLisp
| Wiki markup with Org-mode markup, significant synergy effects are
| produced. [...]
`----
press Edit button, and you see Org syntax!
,----
| * PicoLisp Wiki
|
| * Help
|
| home
|
| [#+OPTIONS: toc:nil num:nil ]
| [#+DESCRIPTION: start page ]
| [#+TITLE: High Level Introduction to iOrg ]
| [ ]
| [/{{{title}}}/ ]
| [ ]
| [* iOrg Application Framework ]
| [ ]
| [ ]
| [#+begin_center ]
| [#+begin_verse ]
| [/Merging two of the most exciting Lisp applications around/ ]
| [/into one powerful Web Application Framework/ ]
| [#+end_verse ]
| [#+end_center ]
| [ ]
| [*iOrg* is based on the [[<nobr>http://picolisp.com/5000/!wiki?home</nobr>][PicoL]
| [web-application developed with the incredibly expressive [[<nobr>http://software]
| [Framework]]. By replacing the original PicoLisp Wiki markup with [[<nobr>http://]
| [markup, significant synergy effects are produced. ]
|
| Summary [ ] This is a
| minor edit [ ]
| --------------------------------------------------------------------
|
| Media
| [x] [^]
| [x] [^] Specify a name [ ]
| [x] [^] [Fetch from page] [ ]
| [|<] [<<] [Upload directly] [ ]
| [<] [>]
| [>>] [>|]
`----
Due to a cooperation with the very helpful Japanese emacs-w3m
maintainer, there are some emacs-w3m goodies for editing this
HTML-Form with Org content:
,----[ C-h f w3m-form-textarea-toggle-major-mode RET ]
| w3m-form-textarea-toggle-major-mode is an interactive compiled Lisp
| function in `w3m-form.el'.
|
| (w3m-form-textarea-toggle-major-mode &optional ARG)
|
| Toggle editing textareas in Org-mode.
| When off, textareas are edited in text-mode, otherwise in
| org-mode. With prefix argument ARG, use org-mode if ARG is
| positive, otherwise text-mode.
`----
so, when this is off, HTML forms are edited in small text-mode buffer
at the button of the w3m buffer, when its on, a full new buffer in
org-mode is offered for editing the form, with its own w3m minor-mode:
,----
| W3m-Form-Input-Textarea minor mode (indicator w3m form textarea):
|
| Minor mode to edit form textareas of w3m.
| key binding
| --- -------
| ESC Prefix Command
| M-# Prefix Command
| M-# c w3m-form-input-textarea-set
| M-# k w3m-form-input-textarea-exit
| M-# q w3m-form-input-textarea-exit
| M-# s w3m-form-input-textarea-save
|
| M-# c => Set the value and exit from this textarea.
| M-# k/q => Exit from this textarea without setting the value.
| M-# s => Save editing data in this textarea.
`----
i.e. you edit the HTML form in full org-mode, in the buffer opened by
emacs-w3m, and then call w3m-form-input-textarea-set to close this
buffer and send its content to the form in the w3m buffer, where you
can preview, save or cancel your edits.
Up to this point you already have a fully functional wiki with user
and document management, authentication, version management, and you
can write and edit the wiki docs in a normal Org-mode buffer opened
from emacs-w3m (a bit like editing a source block in a dedicated
window in a programming major mode from org-mode).
There are already a handful of (Org) wiki pages describing iOrg (a bit
overambitious ;-)
This works, because PicoLisp uses Org-mode as an export backend,
i.e. if you create a new Wiki page or edit one in Org-mode, it sends
the Org text via Emacsclient to the started Emacs daemon and outputs
the returned HTML from the Org exporter.
The wiki pages are just BLOBS in the PicoLisp DB.
You have full text search in the wiki already implemented:
,----
| [synergy ] [Search]
| Search results:
| home
| referer
`----
the word synergy appears in two wiki pages, home and referer, and the
results are links to these pages.
Quite nice, but this was the easy part.
- Below the line 1 (Standard PicoLisp Wiki functionality)
| * iOrg
| * Home
| * logout
| * --------------------
| * Documents
| * Org Data
| * Org Headlines
| * Role Administration
| * User Administration
| * Upload
| * Help
here you find the wiki management and help pages from the PicoLisp
wiki, adapted for iOrg, where you can manage your own role, documents,
and upload multimedia files (as user), or manage roles and other user
as well as their documents (as admin).
e.g. Role Administration
,----
| Role: Administrator
|
| [Edit] [Delete] [Select] <<< -- >>>
|
| Name [Administrator ]
|
| Edit Org Element Objects [*]
| Edit Iorg Wiki Documents [*]
| Create Iorg Forms [ ]
| Edit System Data [*]
| File upload [*]
| Role Administration [*]
| User Administration [*]
| Password [*]
| Delete [*]
|
| User
| admin
`----
Documents
,----
| Select Document
|
| Name [ ] [Search]
|
| 11 Documents
| Name Changed by Summary
| @ emacsui 2016-09-07 21:50 admin
| @ referer 2014-08-30 19:54 admin How did I learn about iOrg?
| @ dhtmlMup 2014-08-30 19:54 admin Markup for DHTML
| @ picoDB 2014-08-30 19:54 admin PicoLisp DB
| @ webUI 2014-08-30 19:54 admin Web UI
| @ assumpt 2014-08-30 19:54 admin Assumptions
| @ tutorial 2014-08-30 19:54 admin Tutorial
| @ faq 2014-08-30 19:54 admin FAQ
| @ help 2014-08-30 19:54 admin Help text
| @ menu 2014-08-30 19:54 admin Initial menu
| @ home 2014-08-30 19:54 admin Initial start page
`----
User Administration
,----
| User: admin
|
| [Edit] [Delete] [Select] <<< -- >>>
|
| Login Name [admin ]
| Password [**** ]
| Role [Administrator] obj
| Full Name [ ]
| EMail [ ]
| Picture [ ] [Install]
| Picture
| Created [2014-08-30]
| New [*]
|
| Postings
| Date Time Summary
| 2016-09-07 21:50
| 2014-08-30 19:54 How did I learn about iOrg?
| 2014-08-30 19:54 Markup for DHTML
| 2014-08-30 19:54 PicoLisp DB
| 2014-08-30 19:54 Emacs UI
| 2014-08-30 19:54 Web UI
`----
- Below the line 2 (iOrg functionality)
Here comes the really challenging part of iOrg:
| * --------------------
| * Org Data
| * Org Headlines
I mapped the structure of an Org Document to an E/R model for the
PicoLisp database. There are countless ways to do this, and I decided:
its all about headlines, so thats at the core of iOrg: a PicoLisp class
,----
| (class +OrgHeadline +orgElement)
`----
Every new parsed headline becomes an object (instance) of this class
and is stored as such in the database. I don't even remember exactly
how I did it, but I think there are two ways to create objects:
1. via the new button in the wiki
2. uploading an existing org file, that is parsed and stored in the DB
While in Org, the file/buffer that contains the headlines is central,
in iOrg the headlines themsselves a central.
There is one more concrete class
,----
| (class +OrgData +orgBaseElement)
`----
that represents the buffer/file parsed (if any). When a file is parsed,
each headline object has a link to its OrgData object (i.e. the file)
and to its predecessor and successor headline, so one can easily
restore the original file.
As an example case, I parsed Bernt Hansen's famous Org tutorial written
in Org:
Here is the OrgData object with all the file header information and
links to all nodes (=headlines). There are much more nodes than you
see, but these lists are scrollable:
,----
| OrgData: ID 174 Obj{66}
|
| [Edit] [Delete] [Select] <<< -- >>>
|
| Input Buffer [ ]
| Input File [/home/tj/git/iorg/picoLisp/iorg/sandbox/orgmode-no]
| Creation-Date [2014-09-25]
| Creation-Time [13:14]
|
| Author [ ]
| Email [bernt@norang.ca ] url
| Creator [Emacs 24.3.1 (Org mode 8.3beta) ]
| [A description of how I currently use org]
| Description [ ]
| [ ]
|
| Contents
| [#+LANGUAGE: en ]
| [#+SEQ_TODO: FIXME FIXED ]
| [#+INFOJS_OPT: view:NIL toc:T ltoc:T mouse:underline buttons:]
| [#+EXPORT_SELECT_TAGS: export ]
| [#+EXPORT_EXCLUDE_TAGS: noexport ]
| [ ]
| [ ]
| [ ]
| [ ]
| [ ]
|
| Direct Descendants
| * [#] Todo Category Title Tags
| 1 orgmode-noeval License [x] [^]
| 1 orgmode-noeval Change History - What's new [x] [^]
| Using Git for Automatic
| 1 orgmode-noeval History, Backups, and [x] [^]
| Synchronization
| 1 orgmode-noeval Things I Don't Use (Anymore) [x] [^]
| 1 orgmode-noeval Productivity Tools [x] [^]
| 1 orgmode-noeval Reminders [x] [^]
|
| [<<] [<] [>] [>>]
|
| All Children
| * [#] Todo Category Title Tags
| 2 orgmode-noeval Org Task structure and presentation
| 3 orgmode-noeval Controlling display of leading stars
| on headlines
`----
I scrolled a bit and jumped to an arbitrary 3rd level headline, which is
presented like this. Note, the object has ID 114, its Ancestor is ID
108, and ID 174 is the containing Org file, see above.
,----
| OrgHeadline: ID 114 Obj{421} [<2014-09-25 13:14:59>]
|
| [Edit] [Delete] [Select] <<< -- >>>
|
| OrgData-ID 174 Ancestor-ID 108
|
| Level [3 ] Priority [- ] Todo [--- ]
|
| Category [orgmode-noeval ] Author [ ]
| Title [Searching and showing results ]
| Alt-Title [ ]
| Tags [ ]
|
| Planning
| Titlestamp
| [- ] [ ] [ ] [ ] [ ]
| [- ] [ ] [- ]
| Scheduled
| [- ] [ ] [ ] [ ] [ ]
| [- ] [ ] [- ]
| Deadline
| [- ] [ ] [ ] [ ] [ ]
| [- ] [ ] [- ]
| Closed
| [- ] [ ] [ ] [ ] [ ]
| [- ] [ ] [- ]
|
| Properties
| Key Value
| [ ] [ ] [x] [^]
| [ ] [ ] [x] [^]
| [ ] [ ] [x] [^]
| [ ] [ ] [x] [^]
| [ ] [ ] [x] [^]
| [ ] [ ] [x] [^]
|
| [<<] [<] [>] [>>] [Edit]
|
| Contents
| [Org-mode's searching capabilities are really effective at fi]
| [in your org files. =C-c / /= does a regular expression sear]
| [current file and shows matching results in a collapsed view ]
| [org-file. ]
| [ ]
| [I have org-mode show the hierarchy of tasks above the matche]
| [and also the immediately following sibling task (but not all]
| [with the following settings: ]
| [ ]
| [#+BEGIN_SRC emacs-lisp ]
| [ (setq org-show-following-heading T) ]
| [ (setq org-show-hierarchy-above T) ]
| [ (setq org-show-siblings (quote ((default)))) ]
| [#+END_SRC ]
| [ ]
| [This keeps the results of the search relatively compact and ]
| [accidental errors by cutting too much data from your org fil]
| [=C-k=. Cutting folded data (including the ...) can be reall]
| [dangerous since it cuts text (including following subtrees) ]
| [can't see. For this reason I always show the following head]
| [displaying search results. ]
| [ ]
| [ ]
| [ ]
| [ ]
| [ ]
| [ ]
| [ ]
| [ ]
| [ ]
|
| Archived? [ ]
| Footnote-Section? [ ]
|
| Direct Descendants
| * [#] Todo Category Title Tags
| [x] [^]
| [x] [^]
| [x] [^]
| [x] [^]
| [x] [^]
| [x] [^]
|
| [<<] [<] [>] [>>]
|
| All Children
| * [#] Todo Category Title Tags
|
| [<<] [<] [>] [>>]
|
| Creation-Date 2014-09-25 13:14
`----
if you press on the Search button, you get a headline query form (there
is another one for OrgData files too):
,----
| [Search] [Reset]
|
| Category [ ]
| Level [-] [ ] Priority [-] [ ] Todo [---- ]
| Tags [ ]
|
| Planning
| Start End
| Titlestamp [ ] [ ] [ ]
| Scheduled [ ] [ ] [ ]
| Deadline [ ] [ ] [ ]
| Closed [ ] [ ] [ ]
|
| Properties
| Key Value
| [ ] [ ] [x] [^]
| [ ] [ ] [x] [^]
| [ ] [ ] [x] [^]
| [ ] [ ] [x] [^]
| [ ] [ ] [x] [^]
| [ ] [ ] [x] [^]
|
| [<<] [<] [>] [>>]
|
| Author [ ]
| Creation-Date [ ] [ ] [ ]
| Archived? [ ]
| Footnote-Section? [ ]
|
| [Search] [Reset]
`----
Here you can search headlines by all possible attributes. Note that the
PicoLisp database is scalable for big data, it is even designed as
distributed object-oriented database, and is superfast.
If I search for this:
,----
| [Search] [Reset]
|
| Category [ ]
| Level [2] [ ] Priority [A] [ ] Todo [---- ]
`----
I get
,----
| Headlines
| * [#] Todo Category Title Tags
| @ 2 A TODO tmp2 B1 :ARCHIVED:
| @ 2 C tmp2 B2 :office:noiorg:
`----
i.e. there are two level 2 headlines in the database with priority A or
lower, all other attributes don't matter in this query.
This turns textfile oriented Org-mode into an object-oriented database
application that scales.
- the picolisp repl
when you start the server in debug mode (with +), you can debug your
application and the database from the picolisp REPL, which is kind of
cool (see www.picolisp.com for more info about this programming language
gem).
I even implemented several functions that let you ask for Information
from the running Emacs daemon, e.g.:
: (emacsServerBacktrace)
*ERROR*: No buffer named *Backtrace*
-> T
: (emacsServerToggleDebugOnError)
: (emacsServerMessages)
You can even start an emacsclient for the running Daemon
used by iOrg, so you can debug any errors of the org-exporter backend
too (but I forgot how ...)
- the *picolisp* repl in Emacs
I forgot how to do it too, but it worked:
I implemented a PicoLisp scrape server for iOrg, that makes it possible
to run and control the application from the comint *picolisp* repl from
picolisp-mode, i.e. you don't even need a web browser for querying and
modifying the database, you can do it from the inferior Picolisp buffer:
,----
| *picolisp* 300 Inferior Pico... (picolisp run)
`----
Although the emacs-w3m is more comfortable of course.
CONCLUSION:
You made me spend the whole evening to write this post, it seems I'm
still interested in iOrg. I created all the examples live this evening,
so it basically works and runs already.
But it needs a bit more work:
1. Fix the (tremendously complicated) +OrgHeadline and +OrgData queries
(they work somehow, but not really without surprises)
2. Clean up the code base (a lot of trial and error left a bit of a mess)
3. Bring the whole thing in Beta state
4. Define a download, install and run process and document it
I want have time to do anything before october, and then it will be slow
too, but maybe - what lasts (very) long, turns out good in the end?
--
cheers,
Thorsten
^ permalink raw reply [flat|nested] 5+ messages in thread