literate programming, development log -- ideas?

literate programming, development log -- ideas?

[Greg Minshall]

hi, all.

i write most of my code in a (per-project) .org file, which is typically
tangled into source or script files.  i have a question about how people
structure their .org files for this sort of use.

some of the non-source bits surrounding my source blocks are for
"classical" literate programming text, i.e., introductions to sections
of code, maybe with links to external pages for libraries or packages i
use in the associated code block.

but i also feel a need for something that might be called a lab
notebook, a development log, of ideas, including dead ends, i pursue
during the development process, with links, etc..  but, i'm not really
sure how to structure this bit, how to integrate it in the rest of the
.org file -- i.e., as a separate heading, or related to the code section
that (originally) was under development when the notes were created.
or...?  etc.

i'm wondering if people do this, especially the development log, and if
there are any hints or practices people might feel would be of interest
to share.

(sorry for the non-specificness of the question.)

cheers, Greg

Re: literate programming, development log -- ideas?

[Juan Manuel Macías]

Hi Greg,

Greg Minshall writes:

> but i also feel a need for something that might be called a lab
> notebook, a development log, of ideas, including dead ends, i pursue
> during the development process, with links, etc..  but, i'm not really
> sure how to structure this bit, how to integrate it in the rest of the
> .org file -- i.e., as a separate heading, or related to the code section
> that (originally) was under development when the notes were created.
> or...?  etc.

I use `org-add-note' a lot, in a wide variety of scenarios. According to their
docstring:

"Add a note to the current entry.
This is done in the same way as adding a state change note."

And I've set (setq org-log-into-drawer t)

On the other hand, maybe you can find interesting the org-marginalia package:
(https://github.com/nobiot/org-marginalia).

Best regards,

Juan Manuel 

Re: literate programming, development log -- ideas? (ominbus reply)

[Greg Minshall]

Juan Manuel, Eric, Jack, Arne,

thank you all very much for your thoughts.

i've sort of come to grips with Jack's + Arne's, solution, and defined a
"capture template" that adds something to a pre-named headline.  (note
and capture are among the org features of which i have maintained to
date a studied ignorance; alas, or a-luck, no more.)

i realized i would like to have headlines (mostly to have the
convenience of =consult-org-heading=), which i think rules out putting
my logs inside property drawers.

Eric, when you use something RCS-like as your version control system, i
assume that makes grepping to find some old note easy enough.  but,
these days i tend to use git.  when (assuming) you use git, do you have
some easy way to say "well, i had this code that looked sort of like
this... where was it?"?  (sorry, that's really a git question, but ...)

cheers, Greg

ps -- for completeness, or code review...
----
("l"
 "add a logbook entry"
 entry
 (file+olp+datetree buffer-file-name "Logbook")
 "* lbe: %?logbook entry\n  :PROPERTIES:\n    creation-date: %t\n
  :END:"
 :empty-lines 1
 :tree-type month)  
----

Re: literate programming, development log -- ideas? (ominbus reply)

[Samuel Banya]

[-- Attachment #1: Type: text/plain, Size: 1923 bytes --]

Not sure if it counts as off-topic for this thread, but does everyone use Git to manage their Org docs and notes?

I ask because of Greg's previous post.

I've noticed that some times after git merge events across a few machines (ex: I forgot I had already pushed notes for my private notes on one machine, and had to merge the results from another machine), I'll get weird "HEAD" and "END" statements inserted by Git.

Also, combined with some tasks duplicating as a result was annoying.

Was debating if this is just something I'd have to deal with, or if there might be a better versioning workflow (ex: just using rsync, etc)

Would be curious on everyone's thoughts.

~ Sam

On Tue, Jun 8, 2021, at 1:15 PM, Greg Minshall wrote:
> Juan Manuel, Eric, Jack, Arne,
> 
> thank you all very much for your thoughts.
> 
> i've sort of come to grips with Jack's + Arne's, solution, and defined a
> "capture template" that adds something to a pre-named headline.  (note
> and capture are among the org features of which i have maintained to
> date a studied ignorance; alas, or a-luck, no more.)
> 
> i realized i would like to have headlines (mostly to have the
> convenience of =consult-org-heading=), which i think rules out putting
> my logs inside property drawers.
> 
> Eric, when you use something RCS-like as your version control system, i
> assume that makes grepping to find some old note easy enough.  but,
> these days i tend to use git.  when (assuming) you use git, do you have
> some easy way to say "well, i had this code that looked sort of like
> this... where was it?"?  (sorry, that's really a git question, but ...)
> 
> cheers, Greg
> 
> ps -- for completeness, or code review...
> ----
> ("l"
> "add a logbook entry"
> entry
> (file+olp+datetree buffer-file-name "Logbook")
> "* lbe: %?logbook entry\n  :PROPERTIES:\n    creation-date: %t\n
>   :END:"
> :empty-lines 1
> :tree-type month)  
> ----
> 
> 

[-- Attachment #2: Type: text/html, Size: 2801 bytes --]

Re: literate programming, development log -- ideas? (ominbus reply)

[Eric S Fraga]

> Not sure if it counts as off-topic for this thread, but does everyone
> use Git to manage their Org docs and notes?

I use a variety of version control systems but for multiple computers I
use unison to keep them all synchronised.

-- 
: Eric S Fraga via Emacs 28.0.50, Org release_9.4.6-549-ga0a87d

Re: literate programming, development log -- ideas? (ominbus reply)

[Dr. Arne Babenhauserheide]

[-- Attachment #1: Type: text/plain, Size: 433 bytes --]


Eric S Fraga <e.fraga@ucl.ac.uk> writes:

>> Not sure if it counts as off-topic for this thread, but does everyone
>> use Git to manage their Org docs and notes?
>
> I use a variety of version control systems but for multiple computers I
> use unison to keep them all synchronised.

I use Mercurial for all my private versiontracking.

Best wishes,
Arne
-- 
Unpolitisch sein
heißt politisch sein
ohne es zu merken

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 1125 bytes --]

Re: literate programming, development log -- ideas? (ominbus reply)

[Samuel Wales]

undo [or equivalent] in buffer.  no saving to disk.

emacs auto save to an auto-expiring directory

emacs backups every save to an auto-expiring directory

git snapshots when i feel like it.  search as needed.

branching in the text itself, by use of comments above if it is
incoming/new and below if it is old/reference.

many use git branch but i wouldn't remember the branch was in git and
i don't want to have to rely on git for such things unless it disrupts
the text without it.  for me it is merely an archive.  it is not an
active repository of future possibilities and current ongoing
activities.  even with magit, git ui is complex to me. its storage
model is not merely an endofuctor of dags.

so i use only basic features when possible.  many years later i still
fear merges/rebases and even stashes and find the merge/rebase
presentation to be quite bizarre and can't believe anybody finds it
the most useful.  certain sequences of staging, unstaging, and
snaphotting can actually cause buffer corruption.  git notes feature
is another complexity which while small i'd rather have git recognize
all file permissions by default.

incidentally, rsync wrapper called rsnapshot works well.  i wouldn't
use it for branching.  you can use unix commands = win.


On 6/9/21, Dr. Arne Babenhauserheide <arne_bab@web.de> wrote:
>
> Eric S Fraga <e.fraga@ucl.ac.uk> writes:
>
>>> Not sure if it counts as off-topic for this thread, but does everyone
>>> use Git to manage their Org docs and notes?
>>
>> I use a variety of version control systems but for multiple computers I
>> use unison to keep them all synchronised.
>
> I use Mercurial for all my private versiontracking.
>
> Best wishes,
> Arne
> --
> Unpolitisch sein
> heißt politisch sein
> ohne es zu merken
>


-- 
The Kafka Pandemic

Please learn what misopathy is.
https://thekafkapandemic.blogspot.com/2013/10/why-some-diseases-are-wronged.html

Re: literate programming, development log -- ideas? (ominbus reply)

[Samuel Banya]

[-- Attachment #1: Type: text/plain, Size: 3011 bytes --]

I actually nuke Emacs's ability to do backups with this in my config:

** Nuke Emacs' ability to make backups and autosaves since its annoying and too bloated
   #+BEGIN_SRC emacs-lisp
     (setq make-backup-files nil)
     (setq auto-save-default nil)
   #+END_SRC

Reason being is that its one of the worst automatic features of any editor. I hate having to deal with ~ files all over the place. However, I've done this for other editors like Vim as well since swapfiles just are a pain to deal with.

I didn't know they had an rsync variant for Emacs. Now that is nice.

I'm okay with git repos for dot files or some kind of programming project but yeah, I've been debating something else for an org based repo cause I too have almost had my 'life.org' be completely destroyed with a merge event.

On Thu, Jun 10, 2021, at 10:07 PM, Samuel Wales wrote:
> undo [or equivalent] in buffer.  no saving to disk.
> 
> emacs auto save to an auto-expiring directory
> 
> emacs backups every save to an auto-expiring directory
> 
> git snapshots when i feel like it.  search as needed.
> 
> branching in the text itself, by use of comments above if it is
> incoming/new and below if it is old/reference.
> 
> many use git branch but i wouldn't remember the branch was in git and
> i don't want to have to rely on git for such things unless it disrupts
> the text without it.  for me it is merely an archive.  it is not an
> active repository of future possibilities and current ongoing
> activities.  even with magit, git ui is complex to me. its storage
> model is not merely an endofuctor of dags.
> 
> so i use only basic features when possible.  many years later i still
> fear merges/rebases and even stashes and find the merge/rebase
> presentation to be quite bizarre and can't believe anybody finds it
> the most useful.  certain sequences of staging, unstaging, and
> snaphotting can actually cause buffer corruption.  git notes feature
> is another complexity which while small i'd rather have git recognize
> all file permissions by default.
> 
> incidentally, rsync wrapper called rsnapshot works well.  i wouldn't
> use it for branching.  you can use unix commands = win.
> 
> 
> On 6/9/21, Dr. Arne Babenhauserheide <arne_bab@web.de <mailto:arne_bab%40web.de>> wrote:
> >
> > Eric S Fraga <e.fraga@ucl.ac.uk <mailto:e.fraga%40ucl.ac.uk>> writes:
> >
> >>> Not sure if it counts as off-topic for this thread, but does everyone
> >>> use Git to manage their Org docs and notes?
> >>
> >> I use a variety of version control systems but for multiple computers I
> >> use unison to keep them all synchronised.
> >
> > I use Mercurial for all my private versiontracking.
> >
> > Best wishes,
> > Arne
> > --
> > Unpolitisch sein
> > heißt politisch sein
> > ohne es zu merken
> >
> 
> 
> -- 
> The Kafka Pandemic
> 
> Please learn what misopathy is.
> https://thekafkapandemic.blogspot.com/2013/10/why-some-diseases-are-wronged.html
> 

[-- Attachment #2: Type: text/html, Size: 4355 bytes --]

Re: literate programming, development log -- ideas? (ominbus reply)

[Juan Manuel Macías]

Samuel Banya writes:

> I'm okay with git repos for dot files or some kind of programming
> project but yeah, I've been debating something else for an org based
> repo cause I too have almost had my 'life.org' be completely destroyed
> with a merge event.

I have all my everyday Org documents in a Nextcloud folder, but only
because I work between the desktop PC and the laptop. I sync using a
script with nextcloud-cmd, not via the Nextcloud app, which I find it
somewhat intrusive. And with another script I do every week a backup of
that folder to a git repo. But this repo is secondary and is only for
keep some backups (I also keep a weekly backup of Elpa folders ---last
versions--- there, in case some update breaks something).

Best regards,

Juan Manuel 

Re: literate programming, development log -- ideas? (ominbus reply)

[Samuel Banya]

[-- Attachment #1: Type: text/plain, Size: 1396 bytes --]

Now this idea I like!

Thanks for this, I didn't think about using the Git repo as a SECONDARY backup source.

Also, using Nextcloud sounds like a neat idea.

I'm into Self-Hosting stuff myself nowadays but am starting out small (ex: Just bought an old Dell Optiplex from eBay, put 2 HDDs into it, planning on using it for HDD backups, and as a Git server).

However, this sounds like an awesome workflow, as I did not consider to host Nextcloud. Awesome idea, thank you for this, Juan!

On Fri, Jun 11, 2021, at 10:30 AM, Juan Manuel Macías wrote:
> Samuel Banya writes:
> 
> > I'm okay with git repos for dot files or some kind of programming
> > project but yeah, I've been debating something else for an org based
> > repo cause I too have almost had my 'life.org' be completely destroyed
> > with a merge event.
> 
> I have all my everyday Org documents in a Nextcloud folder, but only
> because I work between the desktop PC and the laptop. I sync using a
> script with nextcloud-cmd, not via the Nextcloud app, which I find it
> somewhat intrusive. And with another script I do every week a backup of
> that folder to a git repo. But this repo is secondary and is only for
> keep some backups (I also keep a weekly backup of Elpa folders ---last
> versions--- there, in case some update breaks something).
> 
> Best regards,
> 
> Juan Manuel 
> 
> 
> 
> 

[-- Attachment #2: Type: text/html, Size: 2003 bytes --]

Re: literate programming, development log -- ideas? (ominbus reply)

[Maxim Nikulin]

On 09/06/2021 00:21, Samuel Banya wrote:
> 
> I've noticed that some times after git merge events across a few 
> machines (ex: I forgot I had already pushed notes for my private notes 
> on one machine, and had to merge the results from another machine), I'll 
> get weird "HEAD" and "END" statements inserted by Git.

Likely you are writing about merge conflicts. They are not weird, it is 
just necessary to resolve them manually and commit clean variant. There 
is no general way to guess what particular user mean when some file in 
different branches has been modified at the same places. See e.g.
https://git-scm.com/book/en/v2/Git-Branching-Basic-Branching-and-Merging#_basic_merge_conflicts

I remember one case when *clean* merge resulted in JS file that was 
syntactically correct but actually had broken code. It is extremely rare 
accident.

> On Tue, Jun 8, 2021, at 1:15 PM, Greg Minshall wrote:
>> Eric, when you use something RCS-like as your version control system, i
>> assume that makes grepping to find some old note easy enough.  but,
>> these days i tend to use git.  when (assuming) you use git, do you have
>> some easy way to say "well, i had this code that looked sort of like
>> this... where was it?"?  (sorry, that's really a git question, but ...)

There is "git log -Gpattern" option that allows to search for code 
changes. Do not confuse it with "--grep" that is for commit messages. 
"git gui blame file" can show file state before some line was changed. 
Probably magit has a similar command.

Re: literate programming, development log -- ideas? (ominbus reply)

[Greg Minshall]

Maxim,

thanks for =git log -G...=.

cheers, Greg

Re: literate programming, development log -- ideas? (ominbus reply)

[Christian Barthel]

On Tue, Jun 08 2021, Samuel Banya wrote:

> Not sure if it counts as off-topic for this thread, but does
> everyone use Git to manage their Org docs and notes? 

I store my "main" org file in git and commit daily (*).  I have
activated magit-wip-mode for keeping some kind of a backup in
case I execute an unwanted command.  I also let Emacs save
numbered backup files in ~/.emacs.d/backups.  This could be
helpful when I delete the git repository and my org file in
~/doc/org accidentally.

I also store the org files that I create with org-ref
(literature/AuthorYYYY.org). 
I am undecided how to do this with my org-attach folder because
there are some larger files that I do not want to store in git.
git-lfs seemed to be an option but as far as I have seen, it is
not possible to do a local clone [1].
Would love to hear how others manage this.

(*) Previously, I did this monthly with RCS but switched a few
months.

[1] https://github.com/git-lfs/git-lfs/issues/3073
-- 
Christian Barthel

Re: literate programming, development log -- ideas? (ominbus reply)

[Tim Cross]

"Samuel Banya" <sbanya@fastmail.com> writes:

> Not sure if it counts as off-topic for this thread, but does everyone use Git to manage their Org docs and notes?
>
> I ask because of Greg's previous post.
>
> I've noticed that some times after git merge events across a few machines (ex: I forgot I had already pushed notes for my private notes on one machine,
> and had to merge the results from another machine), I'll get weird "HEAD" and "END" statements inserted by Git.
>
> Also, combined with some tasks duplicating as a result was annoying.
>
> Was debating if this is just something I'd have to deal with, or if there might be a better versioning workflow (ex: just using rsync, etc)
>
> Would be curious on everyone's thoughts.
>
> ~ Sam
>

I use git as the master and then checkout to whatever machine I'm working
on. I tend to have at least 3 different machines I'm working on (home
Linux, work Linux and Macbook).

On each machine, I will checkout from master and then create a 'local'
branch where I make any local changes. When I'm finished working
locally, I will commit to the local branch, switch to the master branch,
do a pull. If no changes are pulled, then I will merge in the local
branch and push up to the master repository. If changes are pulled, then
I will make a decision whether to use rebase to add those changes to my
local branch or just merge. Deciding on which depends on the types of
things changed, size of what has changed etc.

I find rebasing and merging is often the best approach to keeping commit
logs fairly clean and linear. However, that will depend on what is being
changed and the amount of changes. Frequent pulling and either merging
and rebasing is useful.

Creating new branches (both just locally and within the master
repository) is a very lightweight operation. I use lots of branches and
will regularly go back through and get rid of old branches when no
longer needed (i.e. changes in the branch have been merged into master
or the branch topic is no longer relevant/needed etc). Understanding the
difference between a rebase and a fast-forward merge is important.
Likewise, using branches effectively is critical. My master branch tends
to be quite clean - I almost never make changes directly in the master
branch. Everything happens in another branch and later merged into the
master when ready. 

Re: literate programming, development log -- ideas? (ominbus reply)

[Samuel Banya]

[-- Attachment #1: Type: text/plain, Size: 2995 bytes --]

Hey Tim,

Wow that's an awesome idea too aka to create local branches on each machine, and then switch over the master branch to pull them in.

Do you have any video references for this kind of thing?

I like videos first, and then documentation.

It would really help me out since I'd love to try to figure this kind of thing out, since I am pretty close to just using rsync for my Git notes going forward, but needed that kind of convincing otherwise.

On Sun, Jun 13, 2021, at 12:46 AM, Tim Cross wrote:
> 
> "Samuel Banya" <sbanya@fastmail.com <mailto:sbanya%40fastmail.com>> writes:
> 
> > Not sure if it counts as off-topic for this thread, but does everyone use Git to manage their Org docs and notes?
> >
> > I ask because of Greg's previous post.
> >
> > I've noticed that some times after git merge events across a few machines (ex: I forgot I had already pushed notes for my private notes on one machine,
> > and had to merge the results from another machine), I'll get weird "HEAD" and "END" statements inserted by Git.
> >
> > Also, combined with some tasks duplicating as a result was annoying.
> >
> > Was debating if this is just something I'd have to deal with, or if there might be a better versioning workflow (ex: just using rsync, etc)
> >
> > Would be curious on everyone's thoughts.
> >
> > ~ Sam
> >
> 
> I use git as the master and then checkout to whatever machine I'm working
> on. I tend to have at least 3 different machines I'm working on (home
> Linux, work Linux and Macbook).
> 
> On each machine, I will checkout from master and then create a 'local'
> branch where I make any local changes. When I'm finished working
> locally, I will commit to the local branch, switch to the master branch,
> do a pull. If no changes are pulled, then I will merge in the local
> branch and push up to the master repository. If changes are pulled, then
> I will make a decision whether to use rebase to add those changes to my
> local branch or just merge. Deciding on which depends on the types of
> things changed, size of what has changed etc.
> 
> I find rebasing and merging is often the best approach to keeping commit
> logs fairly clean and linear. However, that will depend on what is being
> changed and the amount of changes. Frequent pulling and either merging
> and rebasing is useful.
> 
> Creating new branches (both just locally and within the master
> repository) is a very lightweight operation. I use lots of branches and
> will regularly go back through and get rid of old branches when no
> longer needed (i.e. changes in the branch have been merged into master
> or the branch topic is no longer relevant/needed etc). Understanding the
> difference between a rebase and a fast-forward merge is important.
> Likewise, using branches effectively is critical. My master branch tends
> to be quite clean - I almost never make changes directly in the master
> branch. Everything happens in another branch and later merged into the
> master when ready. 
> 
> 

[-- Attachment #2: Type: text/html, Size: 4010 bytes --]

Re: literate programming, development log -- ideas? (ominbus reply)

[Tim Cross]

"Samuel Banya" <sbanya@fastmail.com> writes:

> Hey Tim,
>
> Wow that's an awesome idea too aka to create local branches on each machine, and then switch over the master branch to pull them in.
>
> Do you have any video references for this kind of thing?
>
> I like videos first, and then documentation.
>
> It would really help me out since I'd love to try to figure this kind of thing out, since I am pretty close to just using rsync for my Git notes going forward,
> but needed that kind of convincing otherwise.
>

Sorry, no video links/references. There are lots out there, I just don't
watch them. I think one of the best ways to understand git is to just
use it at the command line and experiment. Git can appear daunting, but
it is actually fairly easy. It has a huge number of commands and
options, but 90% of the time, you really only need a very small handful.

The nice thing about learning a VCS is that you can clone a repository,
experiment as much as you like and when you make a mistake, try to use
git to undo it, but if all else fails, blow away your copy and clone and
try again.

Probably the biggest complaint I see about git (or almost any VCS) is
merge hell. However, this can be largely avoided with a few simple rules
and some discipline in your workflow. People often get into this type of
mess because they make lots of changes before committing them and
because they make changes on the master branch rather than a topic
branch.

I almost never make changes directly to the master branch. I will create
a new branch for even small changes and then merge it back into master
once the changes are complete. Sometimes, I forget and realise I've made
changes which should have been done in a branch. When this happens, I
use the 'stash' command to stash my changes, create a topic branch and
then apply the stash to that branch.

If a topic branch has existed for a while and you know the master branch
has diverged significantly since you made your topic branch, it can be
useful to rebase your topic branch against the current master branch.
This will effectively apply all the changes from master to your topic
branch *before* then applying your changes. It is like you have just
forked your topic branch from master. You may still get some conflicts
when re-applying your branch changes, but there will typically be less
and they will be easier to resolve. 

If your working on a project with lots of contributors, where master
changes quite quickly, you may want to do frequent pulls and either
merge or rebase into your topic branch. Merge conflict really only
becomes a pain when there are lots of changes to all apply at once. If
I'm working on a large project with lots of contributors, I might do a
pull and merge or rebase into my topic branch every day (maybe multiple
times per day). On a private repository, I will do this far less often
because I know what has changed and when it is a good idea to merge or
rebase my current topic branch.

Provided some discipline is applied, I find when it comes time to merge
my topic branch back into master, it is a straight forward merge with no
conflicts and changes in master remain clean. 

It is sometimes a good idea to poke around a bit in the .git directory
and get to understand what is in there and how it works. It is
surprisingly simple once you understand the structure. One of the things
I like about git is that it makes simple things simple and hard things
possible.

Sometimes, using some sort of sync software is the right solution. I
regularly use rsync for a number of tasks. However, I use git and remote
git services like gitlab or github for all of my projects and org notes.
Some people prefer Mercurial (hg) over git. I think it is a fine VCS as
well and often use it too. It has many of the same concepts and can be
used in a similar manner. For some people, it sits better with the way
they work/think than git. However, the core concepts are the same.  

Re: literate programming, development log -- ideas? (ominbus reply)

[Eric S Fraga]

On Tuesday,  8 Jun 2021 at 20:15, Greg Minshall wrote:
> Eric, when you use something RCS-like as your version control system, i
> assume that makes grepping to find some old note easy enough.  but,
> these days i tend to use git.  when (assuming) you use git, do you have
> some easy way to say "well, i had this code that looked sort of like
> this... where was it?"?  (sorry, that's really a git question, but ...)

Yes, for SRC (RCS-like), it's straightforward to go into the ,v file and
search.  For git, I try to make sure the commit message is as
informative as necessary and then search on the log.  The alternative is
git-timemachine which allows me to go back and forward in time looking
at the buffer.

Having said this, I don't often find myself looking for anything in the
history.  For major alternatives that might come up during the project,
I use branches.  But only with git as I have never figured out how to
get a branch set up in src.

-- 
: Eric S Fraga via Emacs 28.0.50, Org release_9.4.6-549-ga0a87d

Re: literate programming, development log -- ideas? (ominbus reply)

[Tim Cross]

Eric S Fraga <e.fraga@ucl.ac.uk> writes:

> On Tuesday,  8 Jun 2021 at 20:15, Greg Minshall wrote:
>> Eric, when you use something RCS-like as your version control system, i
>> assume that makes grepping to find some old note easy enough.  but,
>> these days i tend to use git.  when (assuming) you use git, do you have
>> some easy way to say "well, i had this code that looked sort of like
>> this... where was it?"?  (sorry, that's really a git question, but ...)
>
> Yes, for SRC (RCS-like), it's straightforward to go into the ,v file and
> search.  For git, I try to make sure the commit message is as
> informative as necessary and then search on the log.  The alternative is
> git-timemachine which allows me to go back and forward in time looking
> at the buffer.
>
> Having said this, I don't often find myself looking for anything in the
> history.  For major alternatives that might come up during the project,
> I use branches.  But only with git as I have never figured out how to
> get a branch set up in src.

Similar here as well. There are some good git tools, like timemachine
and git blame and having good commit messages is crucial. However, it
really depends on what your use case is. Other suggestions, like
maintaining a 'date tree' with sections under that can be really useful
for some scenarios. Marking some sections as archived can also be
useful.

A lot depends on whether what you want is an org file which documents
the current state of play or one which is more similar to a lab book
which contains a more chronological type evolution of ideas and
experiments. I often setup completely separate org 'projects' which will
consist of multiple org files and will move things between different
files as the project evolves. In some extreme cases, I may even have
multiple git branches and will often use git tags to mark specific
'milestones'.

How I decide whether to use a date tree with notes, branches, tags,
archive sections/files, separate org files etc is typically determined
by how likely I might be to want to review or go back through previous
work/experiments/decisions. Working this out can take a bit of time and
experimentation, but in general, I rarely need to checkout old versions
or even open archive trees/files. I do have a journal file for each
major project which has lots of ideas, random thoughts and even small
experiments (with source blokcs) and I tend ot have a large 'reference'
file which contains notes and links to external references and then a
'main' org file, which reflects the current state. 

-- 
Tim Cross

Re: literate programming, development log -- ideas? (ominbus reply)

[Greg Minshall]

Tim,

thanks for your comments.

> A lot depends on whether what you want is an org file which documents
> the current state of play or one which is more similar to a lab book
> which contains a more chronological type evolution of ideas and
> experiments. I often setup completely separate org 'projects' which will
> consist of multiple org files and will move things between different
> files as the project evolves. In some extreme cases, I may even have
> multiple git branches and will often use git tags to mark specific
> 'milestones'.
>
> How I decide whether to use a date tree with notes, branches, tags,
> archive sections/files, separate org files etc is typically determined
> by how likely I might be to want to review or go back through previous
> work/experiments/decisions. Working this out can take a bit of time and
> experimentation, but in general, I rarely need to checkout old versions
> or even open archive trees/files. I do have a journal file for each
> major project which has lots of ideas, random thoughts and even small
> experiments (with source blokcs) and I tend ot have a large 'reference'
> file which contains notes and links to external references and then a
> 'main' org file, which reflects the current state. 

my current curiosity is in how to integrate lab book, brain storming,
functionality into my projects.  it seems as if, in an extreme case, you
might possibly have
- a lab book sort of file (i.e., date order, minimal "going back and
  correcting entries")
- a journal file, unstructured, not-infrequently updated, notes, URLs,
  etc.
- the "main" org file -- that which "ships".
- an archive file (one per each of the other?)

for any given project, is the evolution from =foo.org= to this larger
number sort of organic, in the sense that you start with one file, then,
at some point, say, "okay, i need to put these bits in a new journal
file"?

and, trying to leverage the brain cells of others, have you tended to
settle on any sort of consistent naming scheme for the different files?
=-log=, =-notes=, etc.?

i suspect that if my brain were more git-shaped, i'd find the idea of
separate files easier.  i.e., my instinct is to think of each of these
files as having a version-path independent of the others.  rather than
the git-view, which is that the version-path is the commit-path, and
each commit includes the (mostly-temporally) related versions of each of
the files.  more [C-x v d], less [C-x v v].

again, thanks!  Greg

Re: literate programming, development log -- ideas? (ominbus reply)

[Tim Cross]

Greg Minshall <minshall@umich.edu> writes:

> Tim,
>
> thanks for your comments.
>
>> A lot depends on whether what you want is an org file which documents
>> the current state of play or one which is more similar to a lab book
>> which contains a more chronological type evolution of ideas and
>> experiments. I often setup completely separate org 'projects' which will
>> consist of multiple org files and will move things between different
>> files as the project evolves. In some extreme cases, I may even have
>> multiple git branches and will often use git tags to mark specific
>> 'milestones'.
>>
>> How I decide whether to use a date tree with notes, branches, tags,
>> archive sections/files, separate org files etc is typically determined
>> by how likely I might be to want to review or go back through previous
>> work/experiments/decisions. Working this out can take a bit of time and
>> experimentation, but in general, I rarely need to checkout old versions
>> or even open archive trees/files. I do have a journal file for each
>> major project which has lots of ideas, random thoughts and even small
>> experiments (with source blokcs) and I tend ot have a large 'reference'
>> file which contains notes and links to external references and then a
>> 'main' org file, which reflects the current state. 
>
> my current curiosity is in how to integrate lab book, brain storming,
> functionality into my projects.  it seems as if, in an extreme case, you
> might possibly have
> - a lab book sort of file (i.e., date order, minimal "going back and
>   correcting entries")
> - a journal file, unstructured, not-infrequently updated, notes, URLs,
>   etc.
> - the "main" org file -- that which "ships".
> - an archive file (one per each of the other?)
>
> for any given project, is the evolution from =foo.org= to this larger
> number sort of organic, in the sense that you start with one file, then,
> at some point, say, "okay, i need to put these bits in a new journal
> file"?
>
> and, trying to leverage the brain cells of others, have you tended to
> settle on any sort of consistent naming scheme for the different files?
> =-log=, =-notes=, etc.?
>
> i suspect that if my brain were more git-shaped, i'd find the idea of
> separate files easier.  i.e., my instinct is to think of each of these
> files as having a version-path independent of the others.  rather than
> the git-view, which is that the version-path is the commit-path, and
> each commit includes the (mostly-temporally) related versions of each of
> again, thanks!  Greg


For me, the growth was quite organic. I still have quite a few single
file 'projects' where everything is in one file (with top level headings
for the different 'bits').

In fact, nearly all my org usage has followed an organic approach. When
I first started with org, I made the common mistake of getting in and
defining lots of todo states, tags, templates for different structures,
lots of capture templates etc etc. I later realised this was a big
mistake and ended up being a fine example of over engineering.

I ended up going back to a stock standard org setup with next to no
configuration. I then used this 'out of the box' setup for a while and
worked out how to do what I wanted using what org already had. I avoided
installing any additional org related packages. Once I was familiar with
what org has and how to use it, I then began to refine my workflow bit
by bit. I slowly started tweaking my configuration and added some simple
elisp functions to make my life easier. I then systematically looked at
various org packages I had heard about and installed a couple which I
found useful.

I now have a very workable and nice workflow where org is integrated
into most of what I do. There is a bit in my config, but most of it is
just setting various org variables, a few capture templates, a custom
agenda view and some tweaking to some of the export stuff. While I use
org pretty much daily, there is a lot of org I just don't use. For
example, I only use tags very sparingly. I use babel to generate my
init.el, numerous config files (my zsh shell setup, my mbsync and lots
of other config files) and I have some language 'logbook' projects where
I experiment and learn different languages, frameworks etc. I tend not
to use babel/literate programming with full on coding projects. I will
use org for documentation, issue tracking, notes etc. However, for
'real' code projects, I tend to keep the code in 'normal' language
dependent files rather than adopt a full literate/noweb style of
development. I do often use org to document and manage deployment
scripts and some 'devops' type stuff though. 
-- 
Tim Cross

Re: literate programming, development log -- ideas? (ominbus reply)

[Greg Minshall]

Tim,, again thanks for your workflow/org-growth, and also for the git
tutorial in your separate reply to S Banya.  cheers, Greg

Re: literate programming, development log -- ideas?

[Eric S Fraga]

On Monday,  7 Jun 2021 at 14:43, Greg Minshall wrote:
> i write most of my code in a (per-project) .org file, which is typically
> tangled into source or script files.  

I do the same.

> i'm wondering if people do this, especially the development log, and if
> there are any hints or practices people might feel would be of interest
> to share.

I use version control for this aspect, liberally adding/deleting
text/code and relying on the version control system to keep the log for
me.  I used to try to keep the log, as you call it, within the org file
but that seemed eventually to be both difficult and pointless when there
are decent version control tools out there.

I use src mostly [1] when everything is going to be in one file.

The "current" version of the document will have the code and results
that match the text.

YMMV, of course.

HTH,
eric


Footnotes:
[1]  https://gitlab.com/esr/src

-- 
: Eric S Fraga via Emacs 28.0.50, Org release_9.4.6-549-ga0a87d

Re: literate programming, development log -- ideas?

[Tim Cross]

Eric S Fraga <e.fraga@ucl.ac.uk> writes:

> On Monday,  7 Jun 2021 at 14:43, Greg Minshall wrote:
>> i write most of my code in a (per-project) .org file, which is typically
>> tangled into source or script files.  
>
> I do the same.
>
>> i'm wondering if people do this, especially the development log, and if
>> there are any hints or practices people might feel would be of interest
>> to share.
>
> I use version control for this aspect, liberally adding/deleting
> text/code and relying on the version control system to keep the log for
> me.  I used to try to keep the log, as you call it, within the org file
> but that seemed eventually to be both difficult and pointless when there
> are decent version control tools out there.
>
> I use src mostly [1] when everything is going to be in one file.
>
> The "current" version of the document will have the code and results
> that match the text.
>
> YMMV, of course.
>
> Footnotes:
> [1]  https://gitlab.com/esr/src

I do something very similar. I will use org's archive facility as well,
but git with good commit logs seems to meet most of my needs. The
current 'master' HEAD is the current 'state' of the code, documentation,
notes etc.


-- 
Tim Cross

Re: literate programming, development log -- ideas?

[Samuel Banya]

[-- Attachment #1: Type: text/plain, Size: 1801 bytes --]

Hey everyone, 

I just wanted to put this here in the Literate Programming thread, DistroTube just did a video on why Emacs rules, but the following timestamps are pretty dang useful for literate programming for still-always learning newbies like myself, just wanted to mention them here for reference:
Timestamps:
7:20
11:46
15:35
17:36

What Are The Benefits Of Emacs Over Vim? (https://www.youtube.com/watch?v=kRkp-uJTK7s)

On Sun, Jun 13, 2021, at 12:24 AM, Tim Cross wrote:
> 
> Eric S Fraga <e.fraga@ucl.ac.uk <mailto:e.fraga%40ucl.ac.uk>> writes:
> 
> > On Monday,  7 Jun 2021 at 14:43, Greg Minshall wrote:
> >> i write most of my code in a (per-project) .org file, which is typically
> >> tangled into source or script files.  
> >
> > I do the same.
> >
> >> i'm wondering if people do this, especially the development log, and if
> >> there are any hints or practices people might feel would be of interest
> >> to share.
> >
> > I use version control for this aspect, liberally adding/deleting
> > text/code and relying on the version control system to keep the log for
> > me.  I used to try to keep the log, as you call it, within the org file
> > but that seemed eventually to be both difficult and pointless when there
> > are decent version control tools out there.
> >
> > I use src mostly [1] when everything is going to be in one file.
> >
> > The "current" version of the document will have the code and results
> > that match the text.
> >
> > YMMV, of course.
> >
> > Footnotes:
> > [1]  https://gitlab.com/esr/src
> 
> I do something very similar. I will use org's archive facility as well,
> but git with good commit logs seems to meet most of my needs. The
> current 'master' HEAD is the current 'state' of the code, documentation,
> notes etc.
> 
> 
> -- 
> Tim Cross
> 
> 

[-- Attachment #2: Type: text/html, Size: 2880 bytes --]

Re: literate programming, development log -- ideas?

[Greg Minshall]

Samuel,

> I just wanted to put this here in the Literate Programming thread, DistroTube just did a video on why Emacs rules, but the following timestamps are pretty dang useful for literate programming for still-always learning newbies like myself, just wanted to mention them here for reference:
> Timestamps:
> 7:20
> 11:46
> 15:35
> 17:36

thanks, that's very nice.

Re: literate programming, development log -- ideas?

[Dr. Arne Babenhauserheide]

[-- Attachment #1: Type: text/plain, Size: 760 bytes --]


Greg Minshall <minshall@umich.edu> writes:
> but i also feel a need for something that might be called a lab
> notebook, a development log, of ideas, including dead ends, i pursue
> during the development process, with links, etc..  but, i'm not really
> sure how to structure this bit, how to integrate it in the rest of the
> .org file -- i.e., as a separate heading, or related to the code section
> that (originally) was under development when the notes were created.
> or...?  etc.

I just add more sections/headings, sometimes tagged as :noexport:

If need be, I add code-snippets with examples there.

I feel that that’s the simplest way to do it.

Best wishes,
Arne
-- 
Unpolitisch sein
heißt politisch sein
ohne es zu merken

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 1125 bytes --]

Re: literate programming, development log -- ideas?

[briangpowell]

[-- Attachment #1: Type: text/plain, Size: 1466 bytes --]

* Donald Knuth created much for us, including TeX and a Literate
Programming system called CWeb which helped to make C code documented in
what he envisioned for Literate Programming

** A more generalized system that is based on CWeb is NoWeb--useful not
just for C/C++ code but for every language: Recommend using NoWeb for
Literate Programming: "NoWeb — A Simple, Extensible Tool for Literate
Programming":

https://www.cs.tufts.edu/~nr/noweb/

"As of 28 June 2018, the current supported version is version 2.12."
{Author recommends against NOWEB 3.x}





On Mon, Jun 7, 2021 at 9:54 AM Dr. Arne Babenhauserheide <arne_bab@web.de>
wrote:

>
> Greg Minshall <minshall@umich.edu> writes:
> > but i also feel a need for something that might be called a lab
> > notebook, a development log, of ideas, including dead ends, i pursue
> > during the development process, with links, etc..  but, i'm not really
> > sure how to structure this bit, how to integrate it in the rest of the
> > .org file -- i.e., as a separate heading, or related to the code section
> > that (originally) was under development when the notes were created.
> > or...?  etc.
>
> I just add more sections/headings, sometimes tagged as :noexport:
>
> If need be, I add code-snippets with examples there.
>
> I feel that that’s the simplest way to do it.
>
> Best wishes,
> Arne
> --
> Unpolitisch sein
> heißt politisch sein
> ohne es zu merken
>

[-- Attachment #2: Type: text/html, Size: 2062 bytes --]

Re: literate programming, development log -- ideas?

[Dr. Arne Babenhauserheide]

[-- Attachment #1: Type: text/plain, Size: 1451 bytes --]


briangpowell <briangpowellms@gmail.com> writes:

> * Donald Knuth created much for us, including TeX and a Literate
> Programming system called CWeb which helped to make C code documented in
> what he envisioned for Literate Programming
>
> ** A more generalized system that is based on CWeb is NoWeb--useful not
> just for C/C++ code but for every language: Recommend using NoWeb for
> Literate Programming: "NoWeb — A Simple, Extensible Tool for Literate
> Programming":
>
> https://www.cs.tufts.edu/~nr/noweb/

I use :noweb-ref quite a bit, and most of my org-files end in

# Local Variables:
# org-confirm-babel-evaluate: nil
# org-export-default-language: en
# org-babel-noweb-wrap-start: "{{{"
# org-babel-noweb-wrap-end: "}}}"
# End:

A recent wonderful discovery I made is M-x org-babel-detangle. I used it
extensively with mathe-ass:
- https://hg.sr.ht/~arnebab/draketo/browse/anderes/mathe-ass.org?rev=e9c77a19ed5f#L344
- https://www.draketo.de/anderes/mathe-ass.js

mathe-ass.org:
* code                                                             :noexport:
#+begin_src javascript :tangle "mathe-ass.js" :exports none :comments link
…
#+end_src

mathe-ass.js:
// [[file:mathe-ass.org::*code][code:1]]
…
// code:1 ends here

With this I can program in the tangled file and then re-import the code
to org.

Best wishes,
Arne
-- 
Unpolitisch sein
heißt politisch sein
ohne es zu merken

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 1125 bytes --]

Re: literate programming, development log -- ideas?

[Samuel Banya]

[-- Attachment #1: Type: text/plain, Size: 2497 bytes --]

I did a ton of notes in Org Mode with src blocks all of the time in my previous support job, and planning to do the same in my new one.

Can anyone post any video links to this kind of style of programming being used directly in Emacs?

I know there's the awesome one on YouTube by Harry Schwartz, but I feel like there's gotta be some more content on how to do this, and how to integrate this effectively in a workflow.

I know some people had mentioned in this thread that they use a .org file for their given project, but can anyone link to any repos where this is actually being used?

Would be curious on how people have implemented this.

Also, would be interested in other Org packages as well (since I was debating org-roam, but didn't feel I was smart enough to handle it yet).

On Mon, Jun 7, 2021, at 11:17 PM, Dr. Arne Babenhauserheide wrote:
> 
> briangpowell <briangpowellms@gmail.com <mailto:briangpowellms%40gmail.com>> writes:
> 
> > * Donald Knuth created much for us, including TeX and a Literate
> > Programming system called CWeb which helped to make C code documented in
> > what he envisioned for Literate Programming
> >
> > ** A more generalized system that is based on CWeb is NoWeb--useful not
> > just for C/C++ code but for every language: Recommend using NoWeb for
> > Literate Programming: "NoWeb — A Simple, Extensible Tool for Literate
> > Programming":
> >
> > https://www.cs.tufts.edu/~nr/noweb/
> 
> I use :noweb-ref quite a bit, and most of my org-files end in
> 
> # Local Variables:
> # org-confirm-babel-evaluate: nil
> # org-export-default-language: en
> # org-babel-noweb-wrap-start: "{{{"
> # org-babel-noweb-wrap-end: "}}}"
> # End:
> 
> A recent wonderful discovery I made is M-x org-babel-detangle. I used it
> extensively with mathe-ass:
> - https://hg.sr.ht/~arnebab/draketo/browse/anderes/mathe-ass.org?rev=e9c77a19ed5f#L344
> - https://www.draketo.de/anderes/mathe-ass.js
> 
> mathe-ass.org:
> * code                                                             :noexport:
> #+begin_src javascript :tangle "mathe-ass.js" :exports none :comments link
> …
> #+end_src
> 
> mathe-ass.js:
> // [[file:mathe-ass.org::*code][code:1]]
> …
> // code:1 ends here
> 
> With this I can program in the tangled file and then re-import the code
> to org.
> 
> Best wishes,
> Arne
> -- 
> Unpolitisch sein
> heißt politisch sein
> ohne es zu merken
> 
> 
> *Attachments:*
>  * signature.asc

[-- Attachment #2: Type: text/html, Size: 4027 bytes --]

Re: literate programming, development log -- ideas?

[Greg Minshall]

Samuel,

> Can anyone post any video links to this kind of style of programming
> being used directly in Emacs?

there was an effort recently, that i'm tardy on reporting on here, to do
an introduction to "emacs for R programmers".
----
https://ess-intro.github.io/
----

as part of that, i did a (either too short, or too long) video on Org
mode, which, in reality, is pretty neutral w.r.t. programming language.
you might find that of interest.  it's linked at the above ESS intro
site.

i hadn't known of Harry Schwartz's efforts -- thanks for that.

> I know some people had mentioned in this thread that they use a .org
> file for their given project, but can anyone link to any repos where
> this is actually being used?

many people here probably have repos with software.  two of mine are
----
https://gitlab.com/minshall/covid-19
https://gitlab.com/minshall/photoTagger
----

cheers, Greg

Re: literate programming, development log -- ideas?

[Samuel Banya]

[-- Attachment #1: Type: text/plain, Size: 1059 bytes --]

Neat stuff, will bookmark this, thanks Greg!

On Tue, Jun 8, 2021, at 3:23 AM, Greg Minshall wrote:
> Samuel,
> 
> > Can anyone post any video links to this kind of style of programming
> > being used directly in Emacs?
> 
> there was an effort recently, that i'm tardy on reporting on here, to do
> an introduction to "emacs for R programmers".
> ----
> https://ess-intro.github.io/
> ----
> 
> as part of that, i did a (either too short, or too long) video on Org
> mode, which, in reality, is pretty neutral w.r.t. programming language.
> you might find that of interest.  it's linked at the above ESS intro
> site.
> 
> i hadn't known of Harry Schwartz's efforts -- thanks for that.
> 
> > I know some people had mentioned in this thread that they use a .org
> > file for their given project, but can anyone link to any repos where
> > this is actually being used?
> 
> many people here probably have repos with software.  two of mine are
> ----
> https://gitlab.com/minshall/covid-19
> https://gitlab.com/minshall/photoTagger
> ----
> 
> cheers, Greg
> 

[-- Attachment #2: Type: text/html, Size: 1839 bytes --]

Re: literate programming, development log -- ideas?

[Greg Minshall]

Samuel,

> Neat stuff, will bookmark this, thanks Greg!

you're welcome.  i should mention that i don't recommend the .org files
in my gitlab repositories as exemplars of structure/style.  but, they
might give an idea.  the .org file related to the ESS video is probably
more manageable.

cheers, Greg

Re: literate programming, development log -- ideas?

[Greg Minshall]

Brian,

> ** A more generalized system that is based on CWeb is NoWeb--useful
> not just for C/C++ code but for every language: Recommend using NoWeb
> for Literate Programming: "NoWeb — A Simple, Extensible Tool for
> Literate Programming":

thanks.  in fact, i also make extensive use of =<<noweb>>= in my
programming.  and, also, when i found it, =<<noweb()>>= (as it were), to
include the *results* of the execution of the named block -- very cool.

cheers, Greg

Re: literate programming, development log -- ideas?

[Jack Kamm]

Hi Greg,

Greg Minshall <minshall@umich.edu> writes:

> but i also feel a need for something that might be called a lab
> notebook, a development log, of ideas, including dead ends, i pursue
> during the development process, with links, etc..  but, i'm not really
> sure how to structure this bit, how to integrate it in the rest of the
> .org file -- i.e., as a separate heading, or related to the code section
> that (originally) was under development when the notes were created.
> or...?  etc.

This is how I have been using org-babel recently. In particulary, I use
it as a journal for my exploratory data analyses (EDA), mainly in Python
and R.

I find that using a month-tree format, similar to org-capture with
:tree-type month [1], works well for organizing my EDA sections/notes.

Figures I create are named like so:

fig/eda/<6-digit-date>-<short-description>.png

e.g., "fig/eda/210607-celltype-heatmap.png".

I found this workflow keeps things fairly neat, without having to think
too much about how I'm going to organize and name everything. Perfect
for trying out new ideas, iterating quickly, and having an easy to refer
to log of what I've already tried.

[1] https://orgmode.org/manual/Template-elements.html