Re: Problem with Google Calendar Synchronization

[Guido Van Hoecke <guivho@gmail.com>]

[-- Attachment #1: Type: text/plain, Size: 3668 bytes --]

Hi Eric,

>>> Hang on, I am still looking into the UTC aspect.
>>>
>>> Right now the offset is dependent upon the execution time rather than
>>> upon the date being converted.
>>
>> Yes, if by execution time you mean by the time zone of the computer
>> running the script.  This is definitely a problem and one which I gave
>> up trying to solve.  Mind you, it did cause me no end of problems last
>> year and this year again when I moved from the UK to Australia and then
>> back again...
>
> What I meant is that running the script during winter time applies the
> winter time offset to all times, and running during summer time would
> apply the summer time offset to all times.
>
> As far as handling the tz difference between th file and the computer, I
> suggest to use a remote shell to a computer that has the same time zone
> as the file. The offsets will be different, and probably the switchover
> times too. Not something to handle in an awk script, I am afraid.
>
>> Okay.  No rush from my point of view.  I would love to see a solution to
>> this problem!
>
> The attached patch provides a solution. UTC times are corrected with the
> offset appropriate to that specific time. This will work for full hour
> offsets as well as offsets with any number of minutes (e.g. -0145,
> +0230). Still, this conversion uses the computer's time zone...
>
>
> I added some additional configuration options that allow for easy
> tailoring of the script's behaviour:
>
>     # set to 1 or 0 to yes or not output a header block with TITLE,
>     # AUTHOR, EMAIL etc...
>     header = 1;
>
>     # set to 1 or 0 to yes or not output the original ical preamble as
>     # comment
>     preamble = 1;
>
>     # set to 1 to output time and summary as one line starting with
>     # the time (value 1) or to 0 to output the summary as first line
>     # and the date and time info as a second line
>     condense = 0;
>
>     # set to 1 or 0 to yes or not output the original ical entry as a
>     # comment (mostly useful for debugging purposes)
>     original = 1;
>
>     # google truncates long subjects with ... which is misleading in
>     # an org file: it gives the unfortunate impression that an
>     # expanded entry is still collapsed; value 1 will trim those
>     # ... and value 0 doesn't touch them
>     trimdots = 0;
>
>     # change this to your name
>     author = "Eric S Fraga"
>
>     # and to your email address
>     emailaddress = "e.fraga@ucl.ac.uk"
>
>
> I left the default values to produce the output as you initially
> intended.
>
> I prefer to toggle preamble, condense, original and trimdots, but that
> is very subjective.
>
> With kind regards,
>

The attached patch replaces the one of my previous mail. It is to be
applied to the version that is available at
http://orgmode.org/worg/org-tutorials/org-google-sync.html

Until now the script generated a date string for both the start and end
times found in the ics file. This resulted in redundant data.

If the start and end time fall belong to the same day, the script will
now output one single date with a hyphen-separated time span, rather
than two individual (double-hyphen-separated) dates with resp. the start
date/time and the end date/time.

In the same spirit, when start and end date are 1 day apart and no time
info is present (birthdays, anniversaries etc.), only the start date
will be output. So it would output <2013-05-11 Sat> rather than
<2013-05-11 Sat 00:00>--<2013-05-12 Sun 00:00>.

For dates before the epoch this would be <1913-05-11> rather than
<1913-05-11 00:00>--<1913-05-12 00:00>.

Have a nice weekend,


Guido

--
Computer programmers do it byte by byte.



[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: ical2org.patch --]
[-- Type: text/x-patch, Size: 13992 bytes --]

--- ical2org.awk.orig	2013-05-09 14:15:14.000000000 +0200
+++ ical2org.awk	2013-05-11 16:46:37.000000000 +0200
@@ -9,9 +9,22 @@
 # Note: change org meta information generated below for author and
 # email entries!
 #
-# Known bugs:
-# - not so much a bug as a possible assumption: date entries with no time
-#   specified are assumed to be independent of the time zone.
+# Caveats:
+#
+# - date entries with no time specified are assumed to be local time zone;
+#   same remark for date entries that do have a time but do not end with Z
+#   e.g.: 20130101T123456 is local and will be kept as 2013-01-01 12:34
+#   where 20130223T123422Z is UTC and will be corrected appropriately
+#
+# - UTC times are changed into local times, using the time zone of the
+#   computer that runs the script; it would be very hard in an awk script
+#   to respect the time zone of a file belonging to another time zone:
+#   the offsets will be different as well as the switchover time(s);
+#   (consider a remote shell to a computer with the file's time zone)
+#
+# - the UTC conversion entirely relies on the built-in strftime method;
+#   the author is not responsible for any erroneous conversions nor the
+#   consequence of such conversions
 #
 # Eric S Fraga
 # 20100629 - initial version
@@ -27,101 +40,105 @@
 # no further revision log after this as the file was moved into a git
 # repository...
 #
-# Last change: 2011.01.28 16:08:03
+# Last change: 2013.05.11 16:46:37
 #----------------------------------------------------------------------------------

-# a function to take the iCal formatted date+time, convert it into an
-# internal form (seconds since time 0), and adjust according to the
-# local time zone (specified by +-UTC_offset calculated in the BEGIN
-# section)
+BEGIN {
+    ### config section

-function datetimestamp(input)
-{
-    # convert the iCal Date+Time entry to a format that mktime can understand
+    # maximum age in days for entries to be output: set this to -1 to
+    # get all entries or to N>0 to only get enties that start or end
+    # less than N days ago
+    max_age = -1;
+    max_age =  7;
+
+    # set to 1 or 0 to yes or not output a header block with TITLE,
+    # AUTHOR, EMAIL etc...
+    header = 1;
+
+    # set to 1 or 0 to yes or not output the original ical preamble as
+    # comment
+    preamble = 1;
+
+    # set to 1 to output time and summary as one line starting with
+    # the time (value 1) or to 0 to output the summary as first line
+    # and the date and time info as a second line
+    condense = 0;
+
+    # set to 1 or 0 to yes or not output the original ical entry as a
+    # comment (mostly useful for debugging purposes)
+    original = 1;
+
+    # google truncates long subjects with ... which is misleading in
+    # an org file: it gives the unfortunate impression that an
+    # expanded entry is still collapsed; value 1 will trim those
+    # ... and value 0 doesn't touch them
+    trimdots = 0;

-    # datespec in UTC, i.e. ending with Z
-    UTC = "no"
-    UTC = gensub("([0-9][0-9][0-9][0-9])([0-9][0-9])([0-9][0-9])T([0-9][0-9])([0-9][0-9])([0-9][0-9])Z.*[\r]*", "yes", "g", input);
-
-    # parse date and time
-    datespec  = gensub("([0-9][0-9][0-9][0-9])([0-9][0-9])([0-9][0-9])T([0-9][0-9])([0-9][0-9])([0-9][0-9]).*[\r]*", "\\1 \\2 \\3 \\4 \\5 \\6", "g", input);
-
-    # print "date spec : " datespec; convert this date+time into
-    # seconds from the beginning of time and include adjustment for
-    # time zone, as determined in the BEGIN section below.  For time
-    # zone adjustment, I have not tested edge effects, specifically
-    # what happens when UTC time is a different day to local time and
-    # especially when an event with a duration crosses midnight in UTC
-    # time.  It should work but...
-    if(UTC == "yes")
-        timestamp = mktime(datespec) + UTC_offset ;
-    else
-        timestamp = mktime(datespec);
+    # change this to your name
+    author = "Eric S Fraga"

-    # print "adjusted    : " timestamp
-    # print "Time stamp  : " strftime("%Y-%m-%d %H:%M", timestamp);
-    return timestamp;
-}
+    # and to your email address
+    emailaddress = "e.fraga@ucl.ac.uk"

-BEGIN {
-    ### config section
-    max_age =  7; # in days
-                  # set this to -1 to get all entries or to N>0 to only get
-                  # that start or end less than N days ago
     ### end config section

     # use a colon to separate the type of data line from the actual contents
     FS = ":";

-    # determine the number of seconds to use for adjusting for time
-    # zone difference from UTC.  This is used in the function
-    # datetimestamp above.  The time zone information returned by
-    # strftime() is in hours * 100 so we multiply by 36 to get
-    # seconds.  This does not work for time zones that are not an
-    # integral multiple of hours (e.g. Newfoundland)
-    UTC_offset = gensub("([+-])0", "\\1", "", strftime("%z")) * 36;
+    # we only need to preserve the original entry lines if either the
+    # preamble or original options are true
+    preserve = preamble || original

     date = "";
     entry = ""
-    first = 1;                  # true until an event has been found
+    first = 1;      # true until an event has been found
     headline = ""
     icalentry = ""  # the full entry for inspection
     id = ""
     indescription = 0;
-    lasttimestamp=-1;
+    lasttimestamp = -1;

-    print "#+TITLE:     Main Google calendar entries"
-    print "#+AUTHOR:    Eric S Fraga"
-    print "#+EMAIL:     e.fraga@ucl.ac.uk"
-    print "#+DESCRIPTION: converted using the ical2org awk script"
-    print "#+CATEGORY: google"
-    print "#+STARTUP: hidestars"
-    print "#+STARTUP: overview"
-    print " "
+    if (header) {
+        print "#+TITLE:       Main Google calendar entries"
+        print "#+AUTHOR:     ", author
+        print "#+EMAIL:      ", emailaddress
+        print "#+DESCRIPTION: converted using the ical2org awk script"
+        print "#+CATEGORY:    google"
+        print "#+STARTUP:     hidestars"
+        print "#+STARTUP:     overview"
+        print ""
+    }
 }

-# continuation lines (at least from Google) start with two spaces
+# continuation lines (at least from Google) start with a space
 # if the continuation is after a description or a summary, append the entry
 # to the respective variable

-/^[ ]+/ {
+/^[ ]/ {
     if (indescription) {
-        entry = entry gensub("\r", "", "g", gensub("^[ ]+", "", "", $0));
+        entry = entry gensub("\r", "", "g", gensub("^[ ]", "", "", $0));
     } else if (insummary) {
-        summary = summary gensub("\r", "", "g", gensub("^[ ]+", "", "", $0))
+        summary = summary gensub("\r", "", "g", gensub("^[ ]", "", "", $0))
     }
-    icalentry = icalentry "\n" $0
+    if (preserve)
+        icalentry = icalentry "\n" $0
 }

 /^BEGIN:VEVENT/ {
-    # start of an event.  if this is the first, output the preamble from the iCal file
+    # start of an event.
     if (first) {
-        print "* COMMENT original iCal preamble"
-        print gensub("\r", "", "g", icalentry)
-        icalentry = ""
+        # if this is the first event, output the preamble from the iCal file
+        if(preamble) {
+            print "* COMMENT original iCal preamble"
+            print gensub("\r", "", "g", icalentry)
+        }
+        if (preserve)
+            icalentry = ""
     }
     first = false;
 }
+
 # any line that starts at the left with a non-space character is a new data field

 /^[A-Z]/ {
@@ -130,7 +147,9 @@
     # org file as I output the original input.  This change, which is
     # really content free, makes a revision control system update the
     # repository and confuses.
-    if (! index("DTSTAMP", $1)) icalentry = icalentry "\n" $0
+    if (preserve)
+        if (! index("DTSTAMP", $1))
+            icalentry = icalentry "\n" $0
     # this line terminates the collection of description and summary entries
     indescription = 0;
     insummary = 0;
@@ -140,34 +159,37 @@

 /^DTSTART;VALUE=DATE/ {
     datetmp = gensub("([0-9][0-9][0-9][0-9][0-9][0-9][0-9][0-9])(.*[\r])", "\\1T000000\\2", "g", $2)
-    date = strftime("%Y-%m-%d %a %H:%M", datetimestamp(datetmp));
-    if(max_age>0)     lasttimestamp = datetimestamp(datetmp);
+    date = datetimestring(datetmp);
+    starttimestamp = lasttimestamp
 }
 /^DTEND;VALUE=DATE/ {
     datetmp = gensub("([0-9][0-9][0-9][0-9][0-9][0-9][0-9][0-9])(.*[\r])", "\\1T000000\\2", "g", $2)
-    time2 = strftime("%Y-%m-%d %a %H:%M", datetimestamp(datetmp));
-    date = date ">--<" time2;
-    if(max_age>0)     lasttimestamp = datetimestamp(datetmp);
+    time2 = datetimestring(datetmp);
+    if (lasttimestamp == starttimestamp + 86400)
+        # is a full day activity, keep first date without time part
+        date = substr(date, 1, length(date) - 6)
+    else
+        date = date ">--<" time2;
 }

 # this represents a timed entry with date and time stamp YYYYMMDDTHHMMSS
 # we ignore the seconds

 /^DTSTART[:;][^V]/ {
-    date = strftime("%Y-%m-%d %a %H:%M", datetimestamp($2));
-    if(max_age>0)     lasttimestamp = datetimestamp($2);
+    date = datetimestring($2);
     # print date;
 }

-# and the same for the end date; here we extract only the time and append this to the
-# date+time found by the DTSTART entry.  We assume that entry was there, of course.
-# should probably add some error checking here!  In time...
+# and the same for the end date;

 /^DTEND[:;][^V]/ {
-    # print $0
-    time2 = strftime("%Y-%m-%d %a %H:%M", datetimestamp($2));
-    date = date ">--<" time2;
-    if(max_age>0)     lasttimestamp = datetimestamp($2);
+    time2 = datetimestring($2);
+    if (substr(date,1,10) == substr(time2,1,10))
+        # timespan within same date, use one date with a time range
+        date = date "-" substr(time2, length(time2)-4)
+    else
+        # timespan extends over at least two dates
+        date = date ">--<" time2;
 }

 # The description will the contents of the entry in org-mode.
@@ -184,6 +206,10 @@
 /^SUMMARY/ {
     $1 = "";
     summary = gensub("\r", "", "g", $0);
+
+    # trim trailing dots if requested by config option
+    if(trimdots && summary ~ /\.\.\.$/)
+        sub(/\.\.\.$/, "", summary)
     insummary = 1;
 }

@@ -209,23 +235,27 @@
     #output event
     if(max_age<0 || ( lasttimestamp>0 && systime()<lasttimestamp+max_age*24*60*60 ) )
     {
-    # translate \n sequences to actual newlines and unprotect commas (,)
-    print "* " gensub("\\\\,", ",", "g", gensub("\\\\n", " ", "g", summary))
-    print "  <" date ">"
-    print "  :PROPERTIES:"
-    print "  :ID:       " id
-    if(length(location))
-        print "  :LOCATION: " location
-    if(length(status))
-        print "  :STATUS: " status
-    print "  :END:"
-    # for the entry, convert all embedded "\n" strings to actual newlines
-    print ""
-    # translate \n sequences to actual newlines and unprotect commas (,)
-    if(length(entry)>1)
-    print gensub("\\\\,", ",", "g", gensub("\\\\n", "\n", "g", entry));
-    print "** COMMENT original iCal entry"
-    print gensub("\r", "", "g", icalentry)
+        # translate \n sequences to actual newlines and unprotect commas (,)
+        if (condense)
+            print "* <" date "> " gensub("^[ ]+", "", "", gensub("\\\\,", ",", "g", gensub("\\\\n", " ", "g", summary)))
+        else
+            print "* " gensub("^[ ]+", "", "", gensub("\\\\,", ",", "g", gensub("\\\\n", " ", "g", summary))) "\n<" date ">"
+        print ":PROPERTIES:"
+        print     ":ID:       " id
+        if(length(location))
+            print ":LOCATION: " location
+        if(length(status))
+            print ":STATUS:   " status
+        print ":END:"
+
+        print ""
+        # translate \n sequences to actual newlines and unprotect commas (,)
+        if(length(entry)>1)
+            print gensub("^[ ]+", "", "", gensub("\\\\,", ",", "g", gensub("\\\\n", "\n", "g", entry)));
+
+        # output original entry if requested by 'original' config option
+        if (original)
+            print "** COMMENT original iCal entry\n", gensub("\r", "", "g", icalentry)
     }
     summary = ""
     date = ""
@@ -238,6 +268,47 @@
     lasttimestamp = -1
 }

+# funtion to convert an iCal time string 'yyyymmddThhmmss[Z]' into a
+# date time string as used by org, preferably including the short day
+# of week: 'yyyy-mm-dd day hh:mm' or 'yyyy-mm-dd hh:mm' if we cannot
+# define the day of the week
+
+function datetimestring(input)
+{
+    # print "________"
+    # print "input : " input
+    # convert the iCal Date+Time entry to a format that mktime can understand
+    spec  = gensub("([0-9][0-9][0-9][0-9])([0-9][0-9])([0-9][0-9])T([0-9][0-9])([0-9][0-9])([0-9][0-9]).*[\r]*", "\\1 \\2 \\3 \\4 \\5 \\6", "g", input);
+    # print "spec :" spec
+
+    stamp = mktime(spec);
+    lasttimestamp = stamp;
+
+    if (stamp <= 0) {
+        # this is a date before the start of the epoch, so we cannot
+        # use strftime and will deliver a 'yyyy-mm-dd hh:mm' string
+        # without day of week; this assumes local time, and does not
+        # attempt UTC offset correction
+        spec = gensub("([0-9][0-9][0-9][0-9])([0-9][0-9])([0-9][0-9])T([0-9][0-9])([0-9][0-9])([0-9][0-9]).*[\r]*", "\\1-\\2-\\3 \\4:\\5", "g", input);
+        # print "==> spec:" spec;
+        return spec;
+    }
+
+    if (input ~ /[0-9]{8}T[0-9]{6}Z/ ) {
+        # this is an utc time;
+        # we need to correct the timestamp by the utc offset for this time
+        offset = strftime("%z", stamp)
+        pm = substr(offset,1,1) 1 # define multiplier +1 or -1
+        hh = substr(offset,2,2) * 3600 * pm
+        mm = substr(offset,4,2) * 60 * pm
+
+        # adjust the timestamp
+        stamp = stamp + hh + mm
+    }
+
+    return strftime("%Y-%m-%d %a %H:%M", stamp);
+}
+
 # Local Variables:
 # time-stamp-line-limit: 1000
 # time-stamp-format: "%04y.%02m.%02d %02H:%02M:%02S"