From mboxrd@z Thu Jan 1 00:00:00 1970 From: Guido Van Hoecke Subject: Re: Problem with Google Calendar Synchronization Date: Fri, 10 May 2013 23:39:20 +0200 Message-ID: References: <87vc6vxf06.fsf@thinkpad.tsdh.de> <874ned6fhj.fsf@ucl.ac.uk> <87a9o41ns3.fsf@ucl.ac.uk> <87ip2sp7u5.fsf@ucl.ac.uk> <874nebopkv.fsf@ucl.ac.uk> <878v3nvszy.fsf@pinto.chemeng.ucl.ac.uk> Mime-Version: 1.0 Content-Type: multipart/mixed; boundary="=-=-=" Return-path: Received: from eggs.gnu.org ([208.118.235.92]:50485) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1Uav2E-0005Bh-VF for emacs-orgmode@gnu.org; Fri, 10 May 2013 17:39:28 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1Uav2C-0006Xj-R2 for emacs-orgmode@gnu.org; Fri, 10 May 2013 17:39:26 -0400 Received: from mail-we0-x230.google.com ([2a00:1450:400c:c03::230]:65138) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1Uav2C-0006Xa-H0 for emacs-orgmode@gnu.org; Fri, 10 May 2013 17:39:24 -0400 Received: by mail-we0-f176.google.com with SMTP id p60so4310487wes.21 for ; Fri, 10 May 2013 14:39:23 -0700 (PDT) List-Id: "General discussions about Org-mode." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: emacs-orgmode-bounces+geo-emacs-orgmode=m.gmane.org@gnu.org Sender: emacs-orgmode-bounces+geo-emacs-orgmode=m.gmane.org@gnu.org To: orgmode --=-=-= Content-Type: text/plain Hi Eric, > Guido Van Hoecke writes: > > [...] > >> Hang on, I am still looking into the UTC aspect. >> >> Right now the offset is dependent upon the execution time rather than >> upon the date being converted. > > Yes, if by execution time you mean by the time zone of the computer > running the script. This is definitely a problem and one which I gave > up trying to solve. Mind you, it did cause me no end of problems last > year and this year again when I moved from the UK to Australia and then > back again... What I meant is that running the script during winter time applies the winter time offset to all times, and running during summer time would apply the summer time offset to all times. As far as handling the tz difference between th file and the computer, I suggest to use a remote shell to a computer that has the same time zone as the file. The offsets will be different, and probably the switchover times too. Not something to handle in an awk script, I am afraid. > Okay. No rush from my point of view. I would love to see a solution to > this problem! The attached patch provides a solution. UTC times are corrected with the offset appropriate to that specific time. This will work for full hour offsets as well as offsets with any number of minutes (e.g. -0145, +0230). Still, this conversion uses the computer's time zone... I added some additional configuration options that allow for easy tailoring of the script's behaviour: # set to 1 or 0 to yes or not output a header block with TITLE, # AUTHOR, EMAIL etc... header = 1; # set to 1 or 0 to yes or not output the original ical preamble as # comment preamble = 1; # set to 1 to output time and summary as one line starting with # the time (value 1) or to 0 to output the summary as first line # and the date and time info as a second line condense = 0; # set to 1 or 0 to yes or not output the original ical entry as a # comment (mostly useful for debugging purposes) original = 1; # google truncates long subjects with ... which is misleading in # an org file: it gives the unfortunate impression that an # expanded entry is still collapsed; value 1 will trim those # ... and value 0 doesn't touch them trimdots = 0; # change this to your name author = "Eric S Fraga" # and to your email address emailaddress = "e.fraga@ucl.ac.uk" I left the default values to produce the output as you initially intended. I prefer to toggle preamble, condense, original and trimdots, but that is very subjective. With kind regards, --=-=-= Content-Type: text/x-patch Content-Disposition: inline; filename=ical2org.patch --- ical2org.awk.orig 2013-05-09 14:15:14.000000000 +0200 +++ ical2org.awk 2013-05-10 23:33:50.000000000 +0200 @@ -9,9 +9,22 @@ # Note: change org meta information generated below for author and # email entries! # -# Known bugs: -# - not so much a bug as a possible assumption: date entries with no time -# specified are assumed to be independent of the time zone. +# Caveats: +# +# - date entries with no time specified are assumed to be local time zone; +# same remark for date entries that do have a time but do not end with Z +# e.g.: 20130101T123456 is local and will be kept as 2013-01-01 12:34 +# where 20130223T123422Z is UTC and will be corrected appropriately +# +# - UTC times are changed into local times, using the time zone of the +# computer that runs the script; it would be very hard in an awk script +# to respect the time zone of a file belonging to another time zone: +# the offsets will be different as well as the switchover time(s); +# (consider a remote shell to a computer with the file's time zone) +# +# - the UTC conversion entirely relies on the built-in strftime method; +# the author is not responsible for any erroneous conversions nor the +# consequence of such conversions # # Eric S Fraga # 20100629 - initial version @@ -27,101 +40,105 @@ # no further revision log after this as the file was moved into a git # repository... # -# Last change: 2011.01.28 16:08:03 +# Last change: 2013.05.10 23:33:50 #---------------------------------------------------------------------------------- -# a function to take the iCal formatted date+time, convert it into an -# internal form (seconds since time 0), and adjust according to the -# local time zone (specified by +-UTC_offset calculated in the BEGIN -# section) +BEGIN { + ### config section -function datetimestamp(input) -{ - # convert the iCal Date+Time entry to a format that mktime can understand + # maximum age in days for entries to be output: set this to -1 to + # get all entries or to N>0 to only get enties that start or end + # less than N days ago + max_age = -1; + max_age = 7; + + # set to 1 or 0 to yes or not output a header block with TITLE, + # AUTHOR, EMAIL etc... + header = 1; + + # set to 1 or 0 to yes or not output the original ical preamble as + # comment + preamble = 1; + + # set to 1 to output time and summary as one line starting with + # the time (value 1) or to 0 to output the summary as first line + # and the date and time info as a second line + condense = 0; + + # set to 1 or 0 to yes or not output the original ical entry as a + # comment (mostly useful for debugging purposes) + original = 1; + + # google truncates long subjects with ... which is misleading in + # an org file: it gives the unfortunate impression that an + # expanded entry is still collapsed; value 1 will trim those + # ... and value 0 doesn't touch them + trimdots = 0; - # datespec in UTC, i.e. ending with Z - UTC = "no" - UTC = gensub("([0-9][0-9][0-9][0-9])([0-9][0-9])([0-9][0-9])T([0-9][0-9])([0-9][0-9])([0-9][0-9])Z.*[\r]*", "yes", "g", input); - - # parse date and time - datespec = gensub("([0-9][0-9][0-9][0-9])([0-9][0-9])([0-9][0-9])T([0-9][0-9])([0-9][0-9])([0-9][0-9]).*[\r]*", "\\1 \\2 \\3 \\4 \\5 \\6", "g", input); - - # print "date spec : " datespec; convert this date+time into - # seconds from the beginning of time and include adjustment for - # time zone, as determined in the BEGIN section below. For time - # zone adjustment, I have not tested edge effects, specifically - # what happens when UTC time is a different day to local time and - # especially when an event with a duration crosses midnight in UTC - # time. It should work but... - if(UTC == "yes") - timestamp = mktime(datespec) + UTC_offset ; - else - timestamp = mktime(datespec); - - # print "adjusted : " timestamp - # print "Time stamp : " strftime("%Y-%m-%d %H:%M", timestamp); - return timestamp; -} + # change this to your name + author = "Eric S Fraga" + + # and to your email address + emailaddress = "e.fraga@ucl.ac.uk" -BEGIN { - ### config section - max_age = 7; # in days - # set this to -1 to get all entries or to N>0 to only get - # that start or end less than N days ago ### end config section # use a colon to separate the type of data line from the actual contents FS = ":"; - # determine the number of seconds to use for adjusting for time - # zone difference from UTC. This is used in the function - # datetimestamp above. The time zone information returned by - # strftime() is in hours * 100 so we multiply by 36 to get - # seconds. This does not work for time zones that are not an - # integral multiple of hours (e.g. Newfoundland) - UTC_offset = gensub("([+-])0", "\\1", "", strftime("%z")) * 36; + # we only need to preserve the original entry lines if either the + # preamble or original options are true + preserve = preamble || original date = ""; entry = "" - first = 1; # true until an event has been found + first = 1; # true until an event has been found headline = "" icalentry = "" # the full entry for inspection id = "" indescription = 0; - lasttimestamp=-1; + lasttimestamp = -1; - print "#+TITLE: Main Google calendar entries" - print "#+AUTHOR: Eric S Fraga" - print "#+EMAIL: e.fraga@ucl.ac.uk" - print "#+DESCRIPTION: converted using the ical2org awk script" - print "#+CATEGORY: google" - print "#+STARTUP: hidestars" - print "#+STARTUP: overview" - print " " + if (header) { + print "#+TITLE: Main Google calendar entries" + print "#+AUTHOR: ", author + print "#+EMAIL: ", emailaddress + print "#+DESCRIPTION: converted using the ical2org awk script" + print "#+CATEGORY: google" + print "#+STARTUP: hidestars" + print "#+STARTUP: overview" + print "" + } } -# continuation lines (at least from Google) start with two spaces +# continuation lines (at least from Google) start with a space # if the continuation is after a description or a summary, append the entry # to the respective variable -/^[ ]+/ { +/^[ ]/ { if (indescription) { - entry = entry gensub("\r", "", "g", gensub("^[ ]+", "", "", $0)); + entry = entry gensub("\r", "", "g", gensub("^[ ]", "", "", $0)); } else if (insummary) { - summary = summary gensub("\r", "", "g", gensub("^[ ]+", "", "", $0)) + summary = summary gensub("\r", "", "g", gensub("^[ ]", "", "", $0)) } - icalentry = icalentry "\n" $0 + if (preserve) + icalentry = icalentry "\n" $0 } /^BEGIN:VEVENT/ { - # start of an event. if this is the first, output the preamble from the iCal file + # start of an event. if (first) { - print "* COMMENT original iCal preamble" - print gensub("\r", "", "g", icalentry) - icalentry = "" + # if this is the first event, output the preamble from the iCal file + if(preamble) { + print "* COMMENT original iCal preamble" + print gensub("\r", "", "g", icalentry) + } + if (preserve) + icalentry = "" } first = false; } + # any line that starts at the left with a non-space character is a new data field /^[A-Z]/ { @@ -130,7 +147,9 @@ # org file as I output the original input. This change, which is # really content free, makes a revision control system update the # repository and confuses. - if (! index("DTSTAMP", $1)) icalentry = icalentry "\n" $0 + if (preserve) + if (! index("DTSTAMP", $1)) + icalentry = icalentry "\n" $0 # this line terminates the collection of description and summary entries indescription = 0; insummary = 0; @@ -140,22 +159,19 @@ /^DTSTART;VALUE=DATE/ { datetmp = gensub("([0-9][0-9][0-9][0-9][0-9][0-9][0-9][0-9])(.*[\r])", "\\1T000000\\2", "g", $2) - date = strftime("%Y-%m-%d %a %H:%M", datetimestamp(datetmp)); - if(max_age>0) lasttimestamp = datetimestamp(datetmp); + date = datetimestring(datetmp); } /^DTEND;VALUE=DATE/ { datetmp = gensub("([0-9][0-9][0-9][0-9][0-9][0-9][0-9][0-9])(.*[\r])", "\\1T000000\\2", "g", $2) - time2 = strftime("%Y-%m-%d %a %H:%M", datetimestamp(datetmp)); + time2 = datetimestring(datetmp); date = date ">--<" time2; - if(max_age>0) lasttimestamp = datetimestamp(datetmp); } # this represents a timed entry with date and time stamp YYYYMMDDTHHMMSS # we ignore the seconds /^DTSTART[:;][^V]/ { - date = strftime("%Y-%m-%d %a %H:%M", datetimestamp($2)); - if(max_age>0) lasttimestamp = datetimestamp($2); + date = datetimestring($2); # print date; } @@ -165,9 +181,8 @@ /^DTEND[:;][^V]/ { # print $0 - time2 = strftime("%Y-%m-%d %a %H:%M", datetimestamp($2)); + time2 = datetimestring($2); date = date ">--<" time2; - if(max_age>0) lasttimestamp = datetimestamp($2); } # The description will the contents of the entry in org-mode. @@ -184,6 +199,10 @@ /^SUMMARY/ { $1 = ""; summary = gensub("\r", "", "g", $0); + + # trim trailing dots if requested by config option + if(trimdots && summary ~ /\.\.\.$/) + sub(/\.\.\.$/, "", summary) insummary = 1; } @@ -209,23 +228,27 @@ #output event if(max_age<0 || ( lasttimestamp>0 && systime()" - print " :PROPERTIES:" - print " :ID: " id - if(length(location)) - print " :LOCATION: " location - if(length(status)) - print " :STATUS: " status - print " :END:" - # for the entry, convert all embedded "\n" strings to actual newlines - print "" - # translate \n sequences to actual newlines and unprotect commas (,) - if(length(entry)>1) - print gensub("\\\\,", ",", "g", gensub("\\\\n", "\n", "g", entry)); - print "** COMMENT original iCal entry" - print gensub("\r", "", "g", icalentry) + # translate \n sequences to actual newlines and unprotect commas (,) + if (condense) + print "* <" date "> " gensub("^[ ]+", "", "", gensub("\\\\,", ",", "g", gensub("\\\\n", " ", "g", summary))) + else + print "* " gensub("^[ ]+", "", "", gensub("\\\\,", ",", "g", gensub("\\\\n", " ", "g", summary))) "\n<" date ">" + print ":PROPERTIES:" + print ":ID: " id + if(length(location)) + print ":LOCATION: " location + if(length(status)) + print ":STATUS: " status + print ":END:" + + print "" + # translate \n sequences to actual newlines and unprotect commas (,) + if(length(entry)>1) + print gensub("^[ ]+", "", "", gensub("\\\\,", ",", "g", gensub("\\\\n", "\n", "g", entry))); + + # output original entry if requested by 'original' config option + if (original) + print "** COMMENT original iCal entry\n", gensub("\r", "", "g", icalentry) } summary = "" date = "" @@ -238,6 +261,48 @@ lasttimestamp = -1 } +# funtion to convert an iCal time string 'yyyymmddThhmmss[Z]' into a +# date time string as used by org, preferably including the short day +# of week: 'yyyy-mm-dd day hh:mm' or 'yyyy-mm-dd hh:mm' if we cannot +# define the day of the week + +function datetimestring(input) +{ + # print "________" + # print "input : " input + # convert the iCal Date+Time entry to a format that mktime can understand + spec = gensub("([0-9][0-9][0-9][0-9])([0-9][0-9])([0-9][0-9])T([0-9][0-9])([0-9][0-9])([0-9][0-9]).*[\r]*", "\\1 \\2 \\3 \\4 \\5 \\6", "g", input); + # print "spec :" spec + + stamp = mktime(spec); + if (max_age>0) + lasttimestamp = stamp; + + if (stamp <= 0) { + # this is a date before the start of the epoch, so we cannot + # use strftime and will deliver a 'yyyy-mm-dd hh:mm' string + # without day of week; this assumes local time, and does not + # attempt UTC offset correction + spec = gensub("([0-9][0-9][0-9][0-9])([0-9][0-9])([0-9][0-9])T([0-9][0-9])([0-9][0-9])([0-9][0-9]).*[\r]*", "\\1-\\2-\\3 \\4:\\5", "g", input); + # print "==> spec:" spec; + return spec; + } + + if (input ~ /[0-9]{8}T[0-9]{6}Z/ ) { + # this is an utc time; + # we need to correct the timestamp by the utc offset for this time + offset = strftime("%z", stamp) + pm = substr(offset,1,1) 1 # define multiplier +1 or -1 + hh = substr(offset,2,2) * 3600 * pm + mm = substr(offset,4,2) * 60 * pm + + # adjust the timestamp + stamp = stamp + hh + mm + } + + return strftime("%Y-%m-%d %a %H:%M", stamp); +} + # Local Variables: # time-stamp-line-limit: 1000 # time-stamp-format: "%04y.%02m.%02d %02H:%02M:%02S" --=-=-= Content-Type: text/plain Guido -- All of the animals except man know that the principal business of life is to enjoy it. --=-=-=--