* Exploring data that is in org-mode format
@ 2013-09-27 8:18 Alan Schmitt
2013-09-27 8:25 ` Marcin Borkowski
` (3 more replies)
0 siblings, 4 replies; 6+ messages in thread
From: Alan Schmitt @ 2013-09-27 8:18 UTC (permalink / raw)
To: emacs-org list
Hello,
This question is slightly off-topic, but it may be of interest to people
who have a lot of data entered in org-mode.
The short version: what tools are available to explore data, typically
stored in org-mode tables?
The long version: I've tried an interesting website
(https://tictrac.com/) whose goal is to gain some insight about
ourselves by exploring some data we collect (think quantified self). I'm
not happy with this site for three reasons:
- I need to send it the data;
- it focuses on health / activity data whereas there is much more that
interests me (I for instance have weekly records of natural gas use in my
gas-heated house and daily record of temperature average outside which I
would love to compare);
- it won't let you input arbitrary data (I asked about importing a CSV
of my daily coffee consumption, they answered they require an external
service to integrate the data).
So I collect all this data because it's something I enjoy doing, and I
would really like to explore it, from the comfortable position of my own
computer. All of this data is in org-mode tables (or can be easily
converted to org-mode table). Hence my questions: are there tools you
would recommend? I'm not afraid of programming (I suspect an answer will
be 'R'), but I would like pointers to tutorials to do these kind of
things. The kind of things I would like to do are:
- extract weekly or monthly tallies or estimation from data collected at
irregular intervals;
- compare data sources against each other;
- estimate future trends based on past data (how much will my gas bill be?);
- display the result in some kind of dashboard.
Thanks a lot,
Alan
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: Exploring data that is in org-mode format
2013-09-27 8:18 Exploring data that is in org-mode format Alan Schmitt
@ 2013-09-27 8:25 ` Marcin Borkowski
2013-09-27 8:29 ` Karl Voit
` (2 subsequent siblings)
3 siblings, 0 replies; 6+ messages in thread
From: Marcin Borkowski @ 2013-09-27 8:25 UTC (permalink / raw)
To: emacs-orgmode
Dnia 2013-09-27, o godz. 10:18:15
Alan Schmitt <alan.schmitt@polytechnique.org> napisał(a):
> Hello,
>
> This question is slightly off-topic, but it may be of interest to
> people who have a lot of data entered in org-mode.
>
> The short version: what tools are available to explore data, typically
> stored in org-mode tables?
>
> The long version: I've tried an interesting website
> (https://tictrac.com/) whose goal is to gain some insight about
> ourselves by exploring some data we collect (think quantified self).
> I'm not happy with this site for three reasons:
> - I need to send it the data;
> - it focuses on health / activity data whereas there is much more that
> interests me (I for instance have weekly records of natural gas use
> in my gas-heated house and daily record of temperature average
> outside which I would love to compare);
> - it won't let you input arbitrary data (I asked about importing a CSV
> of my daily coffee consumption, they answered they require an external
> service to integrate the data).
>
> So I collect all this data because it's something I enjoy doing, and I
> would really like to explore it, from the comfortable position of my
> own computer. All of this data is in org-mode tables (or can be easily
> converted to org-mode table). Hence my questions: are there tools you
> would recommend? I'm not afraid of programming (I suspect an answer
> will be 'R'), but I would like pointers to tutorials to do these kind
> of things. The kind of things I would like to do are:
> - extract weekly or monthly tallies or estimation from data collected
> at irregular intervals;
> - compare data sources against each other;
> - estimate future trends based on past data (how much will my gas
> bill be?);
> - display the result in some kind of dashboard.
>
> Thanks a lot,
Interesting question. And although it is probably of no use for me,
I'd love to see an Emacs-based tool to do that...
> Alan
Best,
--
Marcin Borkowski
http://octd.wmi.amu.edu.pl/en/Marcin_Borkowski
Adam Mickiewicz University
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: Exploring data that is in org-mode format
2013-09-27 8:18 Exploring data that is in org-mode format Alan Schmitt
2013-09-27 8:25 ` Marcin Borkowski
@ 2013-09-27 8:29 ` Karl Voit
2013-09-27 12:42 ` Nick Dokos
2013-09-27 18:42 ` John Hendy
3 siblings, 0 replies; 6+ messages in thread
From: Karl Voit @ 2013-09-27 8:29 UTC (permalink / raw)
To: emacs-orgmode
* Alan Schmitt <alan.schmitt@polytechnique.org> wrote:
> Hello,
Hi Alan!
> The short version: what tools are available to explore data, typically
> stored in org-mode tables?
Great question!
I did some very basic R-scripts to derive boxplots by myself.
> The long version: I've tried an interesting website
> (https://tictrac.com/) whose goal is to gain some insight about
> ourselves by exploring some data we collect (think quantified self). I'm
> not happy with this site for three reasons:
> - I need to send it the data;
I dislike cloud-based services which are not under my control.
> - it focuses on health / activity data whereas there is much more that
> interests me (I for instance have weekly records of natural gas use in my
> gas-heated house and daily record of temperature average outside which I
> would love to compare);
Absolutely!
> - it won't let you input arbitrary data (I asked about importing a CSV
> of my daily coffee consumption, they answered they require an external
> service to integrate the data).
I understand.
> So I collect all this data because it's something I enjoy doing, and I
> would really like to explore it, from the comfortable position of my own
> computer. All of this data is in org-mode tables (or can be easily
> converted to org-mode table). Hence my questions: are there tools you
> would recommend? I'm not afraid of programming (I suspect an answer will
> be 'R'), but I would like pointers to tutorials to do these kind of
> things. The kind of things I would like to do are:
> - extract weekly or monthly tallies or estimation from data collected at
> irregular intervals;
> - compare data sources against each other;
> - estimate future trends based on past data (how much will my gas bill be?);
> - display the result in some kind of dashboard.
I collect much data about myself using Memacs (see signature).
However, I did not derive anything else but some boxplots from it
(visualizing time-differences between periodic events).
Something generic which I can feed Org-mode files (most preferably
in Memacs-like Org-mode format) and derive some standard graphics
(where I pick those that make sense) would be very great!
--
mail|git|SVN|photos|postings|SMS|phonecalls|RSS|CSV|XML to Org-mode:
> get Memacs from https://github.com/novoid/Memacs <
https://github.com/novoid/extract_pdf_annotations_to_orgmode + more on github
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: Exploring data that is in org-mode format
2013-09-27 8:18 Exploring data that is in org-mode format Alan Schmitt
2013-09-27 8:25 ` Marcin Borkowski
2013-09-27 8:29 ` Karl Voit
@ 2013-09-27 12:42 ` Nick Dokos
2013-09-27 18:42 ` John Hendy
3 siblings, 0 replies; 6+ messages in thread
From: Nick Dokos @ 2013-09-27 12:42 UTC (permalink / raw)
To: emacs-orgmode
Alan Schmitt <alan.schmitt@polytechnique.org> writes:
> Hello,
>
> This question is slightly off-topic, but it may be of interest to people
> who have a lot of data entered in org-mode.
>
> The short version: what tools are available to explore data, typically
> stored in org-mode tables?
>
> The long version: I've tried an interesting website
> (https://tictrac.com/) whose goal is to gain some insight about
> ourselves by exploring some data we collect (think quantified self). I'm
> not happy with this site for three reasons:
> - I need to send it the data;
> - it focuses on health / activity data whereas there is much more that
> interests me (I for instance have weekly records of natural gas use in my
> gas-heated house and daily record of temperature average outside which I
> would love to compare);
> - it won't let you input arbitrary data (I asked about importing a CSV
> of my daily coffee consumption, they answered they require an external
> service to integrate the data).
>
> So I collect all this data because it's something I enjoy doing, and I
> would really like to explore it, from the comfortable position of my own
> computer. All of this data is in org-mode tables (or can be easily
> converted to org-mode table). Hence my questions: are there tools you
> would recommend? I'm not afraid of programming (I suspect an answer will
> be 'R'), but I would like pointers to tutorials to do these kind of
> things. The kind of things I would like to do are:
> - extract weekly or monthly tallies or estimation from data collected at
> irregular intervals;
> - compare data sources against each other;
> - estimate future trends based on past data (how much will my gas bill be?);
> - display the result in some kind of dashboard.
>
Not org-related and not even emacs-related (sorry Marcin!) but
applicable to the question:
Apart from R and Matlab, there is also ... Python: I'm currently reading
a very nice book that uses Python, Numpy, Pandas and Matplotlib for
data exploration. It is called "Python for Data Analysis", by Wes
McKinney (the original developer of Pandas). I'm about a third of the
way through it and I can recommend it. You can find a link to the book
at the Pandas site:
http://pandas.pydata.org/
Just in case the question arises: no, I'm not a paid endorser - just a
satisfied customer :-)
--
Nick
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: Exploring data that is in org-mode format
2013-09-27 8:18 Exploring data that is in org-mode format Alan Schmitt
` (2 preceding siblings ...)
2013-09-27 12:42 ` Nick Dokos
@ 2013-09-27 18:42 ` John Hendy
2013-09-30 7:56 ` Alan Schmitt
3 siblings, 1 reply; 6+ messages in thread
From: John Hendy @ 2013-09-27 18:42 UTC (permalink / raw)
To: Alan Schmitt; +Cc: emacs-org list
On Fri, Sep 27, 2013 at 3:18 AM, Alan Schmitt
<alan.schmitt@polytechnique.org> wrote:
> Hello,
>
> This question is slightly off-topic, but it may be of interest to people
> who have a lot of data entered in org-mode.
>
> The short version: what tools are available to explore data, typically
> stored in org-mode tables?
[snip]
> So I collect all this data because it's something I enjoy doing, and I
> would really like to explore it, from the comfortable position of my own
> computer. All of this data is in org-mode tables (or can be easily
> converted to org-mode table). Hence my questions: are there tools you
> would recommend? I'm not afraid of programming (I suspect an answer will
> be 'R'), but I would like pointers to tutorials to do these kind of
> things. The kind of things I would like to do are:
> - extract weekly or monthly tallies or estimation from data collected at
> irregular intervals;
> - compare data sources against each other;
> - estimate future trends based on past data (how much will my gas bill be?);
> - display the result in some kind of dashboard.
>
For simple exploration, you might have a look at ggobi? [1] It allows
you to do some really quick/easy exploration by plotting and being
able to check which variables to use for X and Y, coloring, filtering,
changing plot type, and so on. There's an R package which allows you
to call ggobi on an R data object, which you could easily create with
babel and your existing org tables with the #+name option for the
table and :var specification in the babel block header.
I'd also highly recommend taking a look at shiny via R-Studio.[2] Not
sure if you can call it from Org-mode, but even if you can't... not
*everything* has to be done with Org. You could use the file to do
some data munging/summarization/etc., save it as a new data set (.csv
or similar), and then read that into Shiny. It could be *awesome* for
something like this. I only recently started playing with it but it's
just fantastic and would make for the ability to subset, change
scales/time ranges, and much more in an interactive web app.
I applied for a public server account with RStudio and got it so that
as I learn and do more, I can make them public. As a result of an SO
question, I repaid the answerer by creating an app from his reply:
- Post: http://stackoverflow.com/questions/17958730/faceting-a-set-of-contour-plots-in-ggplot-r
- Shiny app: http://spark.rstudio.com/jwhendy/interactive-contour/
Good luck!
John
[1] http://www.ggobi.org/
[2] http://www.rstudio.com/shiny/
> Thanks a lot,
>
> Alan
>
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: Exploring data that is in org-mode format
2013-09-27 18:42 ` John Hendy
@ 2013-09-30 7:56 ` Alan Schmitt
0 siblings, 0 replies; 6+ messages in thread
From: Alan Schmitt @ 2013-09-30 7:56 UTC (permalink / raw)
To: John Hendy; +Cc: emacs-org list
jw.hendy@gmail.com writes:
> For simple exploration, you might have a look at ggobi? [1] It allows
> you to do some really quick/easy exploration by plotting and being
> able to check which variables to use for X and Y, coloring, filtering,
> changing plot type, and so on. There's an R package which allows you
> to call ggobi on an R data object, which you could easily create with
> babel and your existing org tables with the #+name option for the
> table and :var specification in the babel block header.
>
> I'd also highly recommend taking a look at shiny via R-Studio.[2] Not
> sure if you can call it from Org-mode, but even if you can't... not
> *everything* has to be done with Org. You could use the file to do
> some data munging/summarization/etc., save it as a new data set (.csv
> or similar), and then read that into Shiny. It could be *awesome* for
> something like this. I only recently started playing with it but it's
> just fantastic and would make for the ability to subset, change
> scales/time ranges, and much more in an interactive web app.
Thank you for these suggestions, they will definitely keep me occupied
for a while.
Alan
^ permalink raw reply [flat|nested] 6+ messages in thread
end of thread, other threads:[~2013-09-30 7:56 UTC | newest]
Thread overview: 6+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2013-09-27 8:18 Exploring data that is in org-mode format Alan Schmitt
2013-09-27 8:25 ` Marcin Borkowski
2013-09-27 8:29 ` Karl Voit
2013-09-27 12:42 ` Nick Dokos
2013-09-27 18:42 ` John Hendy
2013-09-30 7:56 ` Alan Schmitt
Code repositories for project(s) associated with this public inbox
https://git.savannah.gnu.org/cgit/emacs/org-mode.git
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).