From: Nicolas Goaziou <mail@nicolasgoaziou.fr> To: Utkarsh Singh <utkarsh190601@gmail.com> Cc: 47885@debbugs.gnu.org, emacs-orgmode@gnu.org Subject: Re: [PATCH] org-table-import: Make it more smarter for interactive use Date: Tue, 20 Apr 2021 15:40:12 +0200 [thread overview] Message-ID: <87im4h9irn.fsf@nicolasgoaziou.fr> (raw) In-Reply-To: <87k0oyfj4y.fsf@gmail.com> (Utkarsh Singh's message of "Mon, 19 Apr 2021 19:53:25 +0530") Hello, Utkarsh Singh <utkarsh190601@gmail.com> writes: > At first I was also reluctant in creating a new function but decided to > do so because: > > + org-table-convert-region is currently doing two thing 'guessing the > separator' and 'converting the region'. I thought it was a good idea to > separate out function into it's atomic operations. I understand, but there is sometimes a (difficult) line to draw between "separating concerns" and "function proliferation". Anyway, that's fine here. > + Current guessing technique is quite basic as it assumes that data > (file that has to be imported) has no error/inconsistency in it. I > would like to show you the doc string of Python's CSV library > implementation to guess separator (region inside """): > > """ > Looks for text enclosed between two identical quotes > (the probable quotechar) which are preceded and followed > by the same character (the probable delimiter). > For example: > ,'some text', > The quote with the most wins, same with the delimiter. > If there is no quotechar the delimiter can't be determined > this way. > """ > > And if this functions fails then we have: > > """ > The delimiter /should/ occur the same number of times on > each row. However, due to malformed data, it may not. We don't want > an all or nothing approach, so we allow for small variations in this > number. > 1) build a table of the frequency of each character on every line. > 2) build a table of frequencies of this frequency (meta-frequency?), > e.g. 'x occurred 5 times in 10 rows, 6 times in 1000 rows, > 7 times in 2 rows' > 3) use the mode of the meta-frequency to determine the /expected/ > frequency for that character > 4) find out how often the character actually meets that goal > 5) the character that best meets its goal is the delimiter > For performance reasons, the data is evaluated in chunks, so it can > try and evaluate the smallest portion of the data possible, evaluating > additional chunks as necessary. > """ For the problem we're trying to solve, this sounds like over-engineering to me. Do we want so badly to guess a separator? > I tried to do similar in Elisp but currently facing some issues due to > my inexperience in functional programming. Also moving the 'guessing' > part out the function may lead to development of even better algorithm > than Python counterpart. > > Modified version of concerned function: > > (defun org-table-guess-separator (beg0 end0) > "Guess separator for `org-table-convert-region' for region BEG0 to END0. > > List of preferred separator: > comma, TAB, semicolon, colon or SPACE. > > If region contains a line which doesn't contain the required > separator then discard the separator and search again using next > separator." > (let* ((beg (save-excursion > (goto-char (min beg0 end0)) > (line-beginning-position))) > (end (save-excursion > (goto-char (max beg0 end0)) > (line-end-position))) Thinking again about it, this needs extra care, as end0 might end up on an empty line. You tried to avoid this in your first function, but I think this was not sufficient either. Actually, beg0 could also start on an empty line. This needs to be tested extensively, but as a first approximation, I think `beg' needs to be defined as: (save-excursion (goto-char (min beg0 end0)) (skip-chars-forward " \t\n") (if (eobp) (point) (line-beginning-position))) and `end' as (save-excursion (goto-char (max beg end0)) (skip-chars-backward " \t\n" beg) (if (= beg (point)) (point) (line-end-position))) Then you need to bail out if beg = end. > (sep-rexp '(("," "^[^\n,]+$") sep-rexp -> sep-regexp > ("\t" "^[^\n\t]+$") > (";" "^[^\n;]+$") > (":" "^[^\n:]+$") > (" " "^\\([^'\"][^\n\s][^'\"]\\)+$"))) At this point, I suggest to use `rx' macro instead. > (tmp (car sep-rexp)) > sep) > (save-excursion > (goto-char beg) > (while (and (not sep) > (if (save-excursion > (not (re-search-forward (nth 1 tmp) end t))) > (setq sep (nth 0 tmp)) > (setq sep-rexp (cdr sep-rexp)) > (setq tmp (car sep-rexp))))) I suggest this (yes, I like pattern-matching, `car' and `cdr' are so 80's) instead: (save-excursion (goto-char beg) (catch :found (pcase-dolist (`(,sep ,regexp) sep-regexp) (save-excursion (unless (re-search-forward regexp end t) (throw :found sep)))) nil)) Again all this needs to extensively tested, as there are a lot of dangers lurking around. Regards, -- Nicolas Goaziou
next prev parent reply other threads:[~2021-04-20 13:41 UTC|newest] Thread overview: 33+ messages / expand[flat|nested] mbox.gz Atom feed top 2021-04-19 4:43 Utkarsh Singh 2021-04-19 8:19 ` Nicolas Goaziou 2021-04-19 14:23 ` Utkarsh Singh 2021-04-20 13:40 ` Nicolas Goaziou [this message] 2021-04-20 17:15 ` Utkarsh Singh 2021-04-23 4:58 ` Utkarsh Singh 2021-04-27 20:21 ` bug#47885: " Nicolas Goaziou 2021-04-28 8:37 ` Utkarsh Singh 2021-04-28 16:38 ` Maxim Nikulin 2021-05-10 18:36 ` Utkarsh Singh 2021-05-12 17:08 ` Maxim Nikulin 2021-05-14 14:54 ` Utkarsh Singh 2021-05-15 9:13 ` Bastien 2021-05-15 10:10 ` Utkarsh Singh 2021-05-15 10:30 ` Bastien 2021-05-15 11:09 ` Utkarsh Singh 2021-05-17 5:29 ` Bastien 2021-05-17 16:27 ` Utkarsh Singh 2021-06-01 16:23 ` Maxim Nikulin 2021-06-01 17:46 ` Utkarsh Singh 2021-06-02 12:06 ` Maxim Nikulin 2021-06-02 15:08 ` Utkarsh Singh 2021-06-02 16:44 ` Maxim Nikulin 2021-06-04 4:04 ` Utkarsh Singh 2021-06-05 12:40 ` Maxim Nikulin 2021-06-05 17:50 ` Utkarsh Singh 2021-06-09 12:15 ` Maxim Nikulin 2021-09-26 8:40 ` Bastien 2021-05-16 16:24 ` Maxim Nikulin 2021-05-17 16:30 ` Utkarsh Singh 2021-05-18 10:24 ` Utkarsh Singh 2021-05-18 12:31 ` Maxim Nikulin 2021-05-18 15:05 ` Utkarsh Singh
Reply instructions: You may reply publicly to this message via plain-text email using any one of the following methods: * Save the following mbox file, import it into your mail client, and reply-to-all from there: mbox Avoid top-posting and favor interleaved quoting: https://en.wikipedia.org/wiki/Posting_style#Interleaved_style List information: https://www.orgmode.org/ * Reply using the --to, --cc, and --in-reply-to switches of git-send-email(1): git send-email \ --in-reply-to=87im4h9irn.fsf@nicolasgoaziou.fr \ --to=mail@nicolasgoaziou.fr \ --cc=47885@debbugs.gnu.org \ --cc=emacs-orgmode@gnu.org \ --cc=utkarsh190601@gmail.com \ --subject='Re: [PATCH] org-table-import: Make it more smarter for interactive use' \ /path/to/YOUR_REPLY https://kernel.org/pub/software/scm/git/docs/git-send-email.html * If your mail client supports setting the In-Reply-To header via mailto: links, try the mailto: link
Code repositories for project(s) associated with this inbox: https://git.savannah.gnu.org/cgit/emacs/org-mode.git This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).