Back referenced regular expressions in R

I always found the use of regular expressions in R a bit clumsy and difficult. Probably it's just because of lack of exhaustive documentation, but I find the logic of regular expressions as used in Unix's grep and javascript way more clear and useful. A particular thing that is lacking in all the (somewhat cryptic) documentation on R regular expressions is the use of back references in substitutions. Some documentation speaks of back references, but you have to make a distinction between a back reference inside a regular expression and a back reference in a replacement string: In javascript you can back reference inside a regular expression like /(\w+)\s+\1/g which matches all doubled words (e.g., "the" and "of" in "I always found the the use of of regular expressions..."), or you can refer to a (stored) match in the replacement (e.g., "I always found the the use of of regular expressions...".replace(/(\w+)\s+\1/g,"$1") will replace "the the" by "the" and "of of" by "of"). This is obviously quite useful. The available documentation on R regular expressions doesn't seem to be making this distinction. An additional awkwardness in R is the use of backslashes and escape sequences in strings, which makes it quite hard to understand what patterns will match a certain string and how a pattern should be formed to match a certain character sequence in a string. For instance backslashes (\) should always be escaped in R. Double quotes (") in strings don't have to be escaped if you use single quotes to construct a string, but if you print the string on the console, R will print the double quotes with an escaping backslash in front of them. Furthermore some escape sequences have special meaning (aside from \t, \n, \v, \f, \r that are familiar from C) and are printed on the console very differently from what the average user would expect. For instance typing "\1" on the command line prints "\001" to the console and typing "\8" results in "\b" on the console, whereas "\9" prints "9" with an additional warning that the escape sequence '\9' is not recognized. In R routines that use regular expressions you have to provide a string in which the matching pattern is encoded. One problem arises if you want to match a double quote: should you match " or should you match \"? The corresponding patterns would be \" or \\\". The former turns out to be correct. Another problem is back referencing inside the regular expression; should it be "(\w+)\s+\1", "(\\w+)\\s+\1", or "(\\w+)\\s+\\1"? The documentation indicates that the latter back reference is correct, but it doesn't explain the use of \w or \s (which respectively match any 'word-character' (a-z, A-Z, 0-9 and _) and any 'white space' (spaces, tabs, newlines, and certain other non-printable characters) in most regexp implementations). It turns out the "(\\w+)\\s+\\1" is correct. Now how do you match a tab? Should it be "\t" or "\\t"? It turns out to be "\t". The same thing holds for newlines.

Why these differences? You should note that there are two levels of escape sequences: One escape sequence at the level of R strings, and one at the level of the regular expressions engine. Escape sequences that are always required in R strings are clearly necessary (R doesn't know the pattern string encodes a regular expression), and don't get fooled by what is printed on the console! (try typing "this (\\\t) is a double escape sequence with a mistake\\n\n", and then try cat("this (\\\t) is a double escape sequence with a mistake\\n\n")). At the second level, the level of the regular expressions engine, there also exist escape sequences that are like strings in R indicated by starting with a backslash. These sequences however should be handed down unmodified, so '\w' in the example above should consist of a backslash '\' and a 'w' when it is provided to the engine. R however always assumes that a '\' indicates the start of an escape sequence and converts this internally to the corresponding character, just like \t and \n are converted to a tab- and newline-character respectively. If it doesn't recognize the escaped sequence, it prints a warning message. So to hand down escape sequences intended for the regexp engine, you need a double backslash, as '\\' is converted to '\' by R. Special characters like \t and \n do not have to be handed down as escape sequences to the regexp engine, as they are matched by their character, which is a bit unintuitive as you may think that they are treated in the same special way by the engine as other escape sequences.

It always helps to see some examples that should work if you copy and paste them.

# a piece of JSON
json = '{ name:\"v0\",value:0.1,fixed:false}'
# according to the fromJSON function in the rjson package key names should also be quoted (which is not quite right)
# therefore we'd like to substitute 'name' by '"names"', 'value' by '"value"', and 'fixed' by '"fixed"' gsub("([[:alnum:]]+)[[:space:]]*:",'"\\1":',js);
# what also works is
gsub("(\\w+)\\s*:",'"\\1":',js);
# or
gsub("([^\"a-zA-Z])([a-zA-Z]\\w+)([^\"])",'\\1"\\2"\\3',js)