emacs-orgmode@gnu.org archives
 help / color / mirror / code / Atom feed
* Exploring data that is in org-mode format
@ 2013-09-27  8:18 Alan Schmitt
  2013-09-27  8:25 ` Marcin Borkowski
                   ` (3 more replies)
  0 siblings, 4 replies; 6+ messages in thread
From: Alan Schmitt @ 2013-09-27  8:18 UTC (permalink / raw)
  To: emacs-org list

Hello,

This question is slightly off-topic, but it may be of interest to people
who have a lot of data entered in org-mode.

The short version: what tools are available to explore data, typically
stored in org-mode tables?

The long version: I've tried an interesting website
(https://tictrac.com/) whose goal is to gain some insight about
ourselves by exploring some data we collect (think quantified self). I'm
not happy with this site for three reasons:
- I need to send it the data;
- it focuses on health / activity data whereas there is much more that
interests me (I for instance have weekly records of natural gas use in my
gas-heated house and daily record of temperature average outside which I
would love to compare);
- it won't let you input arbitrary data (I asked about importing a CSV
of my daily coffee consumption, they answered they require an external
service to integrate the data).

So I collect all this data because it's something I enjoy doing, and I
would really like to explore it, from the comfortable position of my own
computer. All of this data is in org-mode tables (or can be easily
converted to org-mode table). Hence my questions: are there tools you
would recommend? I'm not afraid of programming (I suspect an answer will
be 'R'), but I would like pointers to tutorials to do these kind of
things. The kind of things I would like to do are:
- extract weekly or monthly tallies or estimation from data collected at
irregular intervals;
- compare data sources against each other;
- estimate future trends based on past data (how much will my gas bill be?);
- display the result in some kind of dashboard.

Thanks a lot,

Alan

^ permalink raw reply	[flat|nested] 6+ messages in thread

* Re: Exploring data that is in org-mode format
  2013-09-27  8:18 Exploring data that is in org-mode format Alan Schmitt
@ 2013-09-27  8:25 ` Marcin Borkowski
  2013-09-27  8:29 ` Karl Voit
                   ` (2 subsequent siblings)
  3 siblings, 0 replies; 6+ messages in thread
From: Marcin Borkowski @ 2013-09-27  8:25 UTC (permalink / raw)
  To: emacs-orgmode

Dnia 2013-09-27, o godz. 10:18:15
Alan Schmitt <alan.schmitt@polytechnique.org> napisał(a):

> Hello,
> 
> This question is slightly off-topic, but it may be of interest to
> people who have a lot of data entered in org-mode.
> 
> The short version: what tools are available to explore data, typically
> stored in org-mode tables?
> 
> The long version: I've tried an interesting website
> (https://tictrac.com/) whose goal is to gain some insight about
> ourselves by exploring some data we collect (think quantified self).
> I'm not happy with this site for three reasons:
> - I need to send it the data;
> - it focuses on health / activity data whereas there is much more that
> interests me (I for instance have weekly records of natural gas use
> in my gas-heated house and daily record of temperature average
> outside which I would love to compare);
> - it won't let you input arbitrary data (I asked about importing a CSV
> of my daily coffee consumption, they answered they require an external
> service to integrate the data).
> 
> So I collect all this data because it's something I enjoy doing, and I
> would really like to explore it, from the comfortable position of my
> own computer. All of this data is in org-mode tables (or can be easily
> converted to org-mode table). Hence my questions: are there tools you
> would recommend? I'm not afraid of programming (I suspect an answer
> will be 'R'), but I would like pointers to tutorials to do these kind
> of things. The kind of things I would like to do are:
> - extract weekly or monthly tallies or estimation from data collected
> at irregular intervals;
> - compare data sources against each other;
> - estimate future trends based on past data (how much will my gas
> bill be?);
> - display the result in some kind of dashboard.
> 
> Thanks a lot,

Interesting question.  And although it is probably of no use for me,
I'd love to see an Emacs-based tool to do that...

> Alan

Best,

-- 
Marcin Borkowski
http://octd.wmi.amu.edu.pl/en/Marcin_Borkowski
Adam Mickiewicz University

^ permalink raw reply	[flat|nested] 6+ messages in thread

* Re: Exploring data that is in org-mode format
  2013-09-27  8:18 Exploring data that is in org-mode format Alan Schmitt
  2013-09-27  8:25 ` Marcin Borkowski
@ 2013-09-27  8:29 ` Karl Voit
  2013-09-27 12:42 ` Nick Dokos
  2013-09-27 18:42 ` John Hendy
  3 siblings, 0 replies; 6+ messages in thread
From: Karl Voit @ 2013-09-27  8:29 UTC (permalink / raw)
  To: emacs-orgmode

* Alan Schmitt <alan.schmitt@polytechnique.org> wrote:
> Hello,

Hi Alan!

> The short version: what tools are available to explore data, typically
> stored in org-mode tables?

Great question!

I did some very basic R-scripts to derive boxplots by myself.

> The long version: I've tried an interesting website
> (https://tictrac.com/) whose goal is to gain some insight about
> ourselves by exploring some data we collect (think quantified self). I'm
> not happy with this site for three reasons:
> - I need to send it the data;

I dislike cloud-based services which are not under my control.

> - it focuses on health / activity data whereas there is much more that
> interests me (I for instance have weekly records of natural gas use in my
> gas-heated house and daily record of temperature average outside which I
> would love to compare);

Absolutely!

> - it won't let you input arbitrary data (I asked about importing a CSV
> of my daily coffee consumption, they answered they require an external
> service to integrate the data).

I understand.

> So I collect all this data because it's something I enjoy doing, and I
> would really like to explore it, from the comfortable position of my own
> computer. All of this data is in org-mode tables (or can be easily
> converted to org-mode table). Hence my questions: are there tools you
> would recommend? I'm not afraid of programming (I suspect an answer will
> be 'R'), but I would like pointers to tutorials to do these kind of
> things. The kind of things I would like to do are:
> - extract weekly or monthly tallies or estimation from data collected at
> irregular intervals;
> - compare data sources against each other;
> - estimate future trends based on past data (how much will my gas bill be?);
> - display the result in some kind of dashboard.

I collect much data about myself using Memacs (see signature).
However, I did not derive anything else but some boxplots from it
(visualizing time-differences between periodic events).

Something generic which I can feed Org-mode files (most preferably
in Memacs-like Org-mode format) and derive some standard graphics
(where I pick those that make sense) would be very great!

-- 
mail|git|SVN|photos|postings|SMS|phonecalls|RSS|CSV|XML to Org-mode:
       > get Memacs from https://github.com/novoid/Memacs <

https://github.com/novoid/extract_pdf_annotations_to_orgmode + more on github

^ permalink raw reply	[flat|nested] 6+ messages in thread

* Re: Exploring data that is in org-mode format
  2013-09-27  8:18 Exploring data that is in org-mode format Alan Schmitt
  2013-09-27  8:25 ` Marcin Borkowski
  2013-09-27  8:29 ` Karl Voit
@ 2013-09-27 12:42 ` Nick Dokos
  2013-09-27 18:42 ` John Hendy
  3 siblings, 0 replies; 6+ messages in thread
From: Nick Dokos @ 2013-09-27 12:42 UTC (permalink / raw)
  To: emacs-orgmode

Alan Schmitt <alan.schmitt@polytechnique.org> writes:

> Hello,
>
> This question is slightly off-topic, but it may be of interest to people
> who have a lot of data entered in org-mode.
>
> The short version: what tools are available to explore data, typically
> stored in org-mode tables?
>
> The long version: I've tried an interesting website
> (https://tictrac.com/) whose goal is to gain some insight about
> ourselves by exploring some data we collect (think quantified self). I'm
> not happy with this site for three reasons:
> - I need to send it the data;
> - it focuses on health / activity data whereas there is much more that
> interests me (I for instance have weekly records of natural gas use in my
> gas-heated house and daily record of temperature average outside which I
> would love to compare);
> - it won't let you input arbitrary data (I asked about importing a CSV
> of my daily coffee consumption, they answered they require an external
> service to integrate the data).
>
> So I collect all this data because it's something I enjoy doing, and I
> would really like to explore it, from the comfortable position of my own
> computer. All of this data is in org-mode tables (or can be easily
> converted to org-mode table). Hence my questions: are there tools you
> would recommend? I'm not afraid of programming (I suspect an answer will
> be 'R'), but I would like pointers to tutorials to do these kind of
> things. The kind of things I would like to do are:
> - extract weekly or monthly tallies or estimation from data collected at
> irregular intervals;
> - compare data sources against each other;
> - estimate future trends based on past data (how much will my gas bill be?);
> - display the result in some kind of dashboard.
>

Not org-related and not even emacs-related (sorry Marcin!) but
applicable to the question:

Apart from R and Matlab, there is also ... Python: I'm currently reading 
a very nice book that uses Python, Numpy, Pandas and Matplotlib for
data exploration. It is called "Python for Data Analysis", by Wes
McKinney (the original developer of Pandas). I'm about a third of the
way through it and I can recommend it. You can find a link to the book
at the Pandas site:

   http://pandas.pydata.org/

Just in case the question arises: no, I'm not a paid endorser - just a
satisfied customer :-)

-- 
Nick

^ permalink raw reply	[flat|nested] 6+ messages in thread

* Re: Exploring data that is in org-mode format
  2013-09-27  8:18 Exploring data that is in org-mode format Alan Schmitt
                   ` (2 preceding siblings ...)
  2013-09-27 12:42 ` Nick Dokos
@ 2013-09-27 18:42 ` John Hendy
  2013-09-30  7:56   ` Alan Schmitt
  3 siblings, 1 reply; 6+ messages in thread
From: John Hendy @ 2013-09-27 18:42 UTC (permalink / raw)
  To: Alan Schmitt; +Cc: emacs-org list

On Fri, Sep 27, 2013 at 3:18 AM, Alan Schmitt
<alan.schmitt@polytechnique.org> wrote:
> Hello,
>
> This question is slightly off-topic, but it may be of interest to people
> who have a lot of data entered in org-mode.
>
> The short version: what tools are available to explore data, typically
> stored in org-mode tables?

[snip]

> So I collect all this data because it's something I enjoy doing, and I
> would really like to explore it, from the comfortable position of my own
> computer. All of this data is in org-mode tables (or can be easily
> converted to org-mode table). Hence my questions: are there tools you
> would recommend? I'm not afraid of programming (I suspect an answer will
> be 'R'), but I would like pointers to tutorials to do these kind of
> things. The kind of things I would like to do are:
> - extract weekly or monthly tallies or estimation from data collected at
> irregular intervals;
> - compare data sources against each other;
> - estimate future trends based on past data (how much will my gas bill be?);
> - display the result in some kind of dashboard.
>

For simple exploration, you might have a look at ggobi? [1] It allows
you to do some really quick/easy exploration by plotting and being
able to check which variables to use for X and Y, coloring, filtering,
changing plot type, and so on. There's an R package which allows you
to call ggobi on an R data object, which you could easily create with
babel and your existing org tables with the #+name option for the
table and :var specification in the babel block header.

I'd also highly recommend taking a look at shiny via R-Studio.[2] Not
sure if you can call it from Org-mode, but even if you can't... not
*everything* has to be done with Org. You could use the file to do
some data munging/summarization/etc., save it as a new data set (.csv
or similar), and then read that into Shiny. It could be *awesome* for
something like this. I only recently started playing with it but it's
just fantastic and would make for the ability to subset, change
scales/time ranges, and much more in an interactive web app.

I applied for a public server account with RStudio and got it so that
as I learn and do more, I can make them public. As a result of an SO
question, I repaid the answerer by creating an app from his reply:
- Post: http://stackoverflow.com/questions/17958730/faceting-a-set-of-contour-plots-in-ggplot-r
- Shiny app: http://spark.rstudio.com/jwhendy/interactive-contour/


Good luck!
John

[1] http://www.ggobi.org/
[2] http://www.rstudio.com/shiny/


> Thanks a lot,
>
> Alan
>

^ permalink raw reply	[flat|nested] 6+ messages in thread

* Re: Exploring data that is in org-mode format
  2013-09-27 18:42 ` John Hendy
@ 2013-09-30  7:56   ` Alan Schmitt
  0 siblings, 0 replies; 6+ messages in thread
From: Alan Schmitt @ 2013-09-30  7:56 UTC (permalink / raw)
  To: John Hendy; +Cc: emacs-org list

jw.hendy@gmail.com writes:

> For simple exploration, you might have a look at ggobi? [1] It allows
> you to do some really quick/easy exploration by plotting and being
> able to check which variables to use for X and Y, coloring, filtering,
> changing plot type, and so on. There's an R package which allows you
> to call ggobi on an R data object, which you could easily create with
> babel and your existing org tables with the #+name option for the
> table and :var specification in the babel block header.
>
> I'd also highly recommend taking a look at shiny via R-Studio.[2] Not
> sure if you can call it from Org-mode, but even if you can't... not
> *everything* has to be done with Org. You could use the file to do
> some data munging/summarization/etc., save it as a new data set (.csv
> or similar), and then read that into Shiny. It could be *awesome* for
> something like this. I only recently started playing with it but it's
> just fantastic and would make for the ability to subset, change
> scales/time ranges, and much more in an interactive web app.

Thank you for these suggestions, they will definitely keep me occupied
for a while.

Alan

^ permalink raw reply	[flat|nested] 6+ messages in thread

end of thread, other threads:[~2013-09-30  7:56 UTC | newest]

Thread overview: 6+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2013-09-27  8:18 Exploring data that is in org-mode format Alan Schmitt
2013-09-27  8:25 ` Marcin Borkowski
2013-09-27  8:29 ` Karl Voit
2013-09-27 12:42 ` Nick Dokos
2013-09-27 18:42 ` John Hendy
2013-09-30  7:56   ` Alan Schmitt

Code repositories for project(s) associated with this public inbox

	https://git.savannah.gnu.org/cgit/emacs/org-mode.git

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).