Regular Expressions in Groovy

From NGDCWiki

(Difference between revisions)
Jump to: navigation, search
m (Include a sample of text to match)
m (Example usage of Match Flags)
Line 308: Line 308:
This is how the Match Flags may be used in '''Groovy'''. Below is a pattern that will grab the disk space value from the output of the unix "df" (disk freespace) command, but only on lines containing "/nfs/data", the "(?i)" indicates a case insensitive match:
This is how the Match Flags may be used in '''Groovy'''. Below is a pattern that will grab the disk space value from the output of the unix "df" (disk freespace) command, but only on lines containing "/nfs/data", the "(?i)" indicates a case insensitive match:
-
  pattern = ~/'''<font color="red">(?i)</font>'''(\d+)\s+\d+%\s+(\/nfs\/data[^\/]*)/
+
  pattern = ~/'''<font color="red">(?i)</font>'''(\d+)\s+\d+%\s+(\/nfs\/data.*)/
Note that the "(?i)" match flag comes at the beginning and is wrapped in parentheses. The "?" indicates this is a non-capturing group. Normally anything in a regular expression you wish to capture is wrapped in parentheses. Assuming the match over all succeeded, this "captured" group is available for later use as a substring that matched the portion of the regular expression in the parentheses.
Note that the "(?i)" match flag comes at the beginning and is wrapped in parentheses. The "?" indicates this is a non-capturing group. Normally anything in a regular expression you wish to capture is wrapped in parentheses. Assuming the match over all succeeded, this "captured" group is available for later use as a substring that matched the portion of the regular expression in the parentheses.
Line 314: Line 314:
For comparison, here is the equivalent pattern in '''Perl''':
For comparison, here is the equivalent pattern in '''Perl''':
-
  $dfkOutput =~ /(\d+)\s+\d+%\s+(\/nfs\/data[^\/]*)/'''<font color="red">i</font>'''
+
  $dfkOutput =~ /(\d+)\s+\d+%\s+(\/nfs\/data.*)/'''<font color="red">i</font>'''
Notice that the "i" indicating a case-insensitive match is appended at the end of the pattern.
Notice that the "i" indicating a case-insensitive match is appended at the end of the pattern.

Revision as of 12:46, 4 May 2010

Here are some suggestions for using regular expressions in Groovy. This page mainly focuses on documenting regular expressions in Groovy, however, those suggestions are applicable to any programming language that supports regular expressions such as Perl and Java.

Contents

Reference Links

Here are some useful reference links that you may want to open up along side this page:


Documenting the RegEx

It's important to document any regular expression, or "regex" for short, that is more than a trivial match. Documenting regexes is the key to making them understandable so they can be debugged and modified either by someone else or by you after you've had time to forget the details.

Overview

  • Include a sample of text to match
    • give a plain English description of your goal
    • omit excess lines of sample if long
  • Use extended patterns with comments
    • mark capturing groups by number
    • include "landmark" keys in the pattern
  • Include debugging feedback
    • use debugging lines

Include a sample of text to match

Having a sample of the input that the regular expression is being applied to look at right on screen is a great help in deciphering what the pattern is trying to match. This is best done in a block comment before the regular expression pattern is defined.

  • give a plain English description of your goal
    • note "landmark" keys in the pattern that you rely on to reliably parse the data
    • list any sub-parts (captured groups) of the pattern you wish to use after the match
  • omit excess lines of sample if long

For example, on a system that has remotely mounted disk space with names like "/nfs/data" or "/nfs/DATA" we wish to gather the space free in kilobytes and the name on which the space is mounted. The output from the "df -k" (disk free space in kilobytes, on linux/mac/unix systems) could be parsed by this pattern:

pattern = ~/(?i)(\d+)\s+\d+%\s+(\/nfs\/data.*)/

The "(?i)" is a match flag that means the pattern is case insensitive.

To summarize the parts (regular expression constructs) here:

  • (\d+) - One or more digits, captured for later use, "+" means 1 or more repetitions, see the Pattern API
  • \s+ - One or more whitespace characters
  • \d+% - One or more digits followed by "%", the percentage of disk used
  • (\/nfs\/data.*) - look for a partition name that starts out with "/nfs/data"
    • \/ - A literal "/", escaped by "\" since a slash by itself starts or ends the pattern
    • .* - Matches 0 or more characters, "." is a wildcard, "*" means 0 or more repetitions

Following the suggestions above, a header comment is added:

/*
    The input string to parse is from the output of the 'df -k' command.
    Example input:
 
        Filesystem           1K-blocks      Used Available Use% Mounted on
        /dev/sda1              4185956   1206996   2762888  31% /
        /dev/sda11            30679784  28324040    772140  98% /extra
        <..lines omitted..>
        fas3050c-1-2.b:/vol/canis
                              10485760   6599936   3885824  63% /nfs/data_d2/dog_data
        fas6070-1-1.b:/vol/felis
                             314572800  54889600 259683200  18% /nfs/DATA-1/cat_data
  
    We want the available disk space in KB, for /nfs/data remote
    disk partitions, which is "3885824" and "259683200" in the sample
    above. Note that partitions that start out with "/nfs/data" may be
    either upper or lower case. Capture the partition name for debugging.
  
    Capture the space available in KB as the number before the
    percentage number (digits followed by '%'). The pattern match
    is relying on there being just one instance of a numeric
    percentage (\d+%) occurring in the each usable output line.

    The header line contains a "%", but the characters preceding it
    are non-numeric, "Use%", so it is ignored. Also, many of the lines
    we want are split across two lines, but it is only the second line
    that contains the information we want.

    The "(?i)" match flag indicates the pattern is case insensitive.
*/
pattern = ~/(?i)(\d+)\s+\d+%\s+(\/nfs\/data.*)/

Use extended patterns with comments

The extended match mode is enabled by a pattern match flag which allows white space and comments to be embedded into the pattern. You can then describe, piece by piece, the parts of the regular expression without dumping those details into the already large header comment suggested above. Pattern match flags are discussed in more detail later in this document.

In Groovy, this match flag is "(?x)" and can be combined with other flags you wish to turn on such as "(?ix)" for both extended and case-insensitive modes. This is done in conjunction with Groovy "here" documents (triple quoting), which is handled somewhat differently than the "slashy" quoting used for regular expression patterns. The three examples below are equivalent, but I've highlighted in red what is removed from the first, and colored green the new text in the second and third examples.

// slashy regex

pattern = ~/(?i)(\d+)\s+\d+%\s+(\/nfs\/data.*)/
// string converted to a regex

regex = "(?i)(\\d+)\\s+\\d+%\\s+(/nfs/data.*)"
pattern = ~regex
// here document string converted to a regex

regex = '''(?ix)(\\d+)\\s+\\d+%\\s+(/nfs/data.*)'''
pattern = ~regex

Essentially:

  • Forward slashes don't need to be escaped by back slashes so "\/" becomes "/"
  • Double the remaining back slashes. Back slashes need to be escaped by back slashes when quoting strings (either normal or here documents)
  • If you want to match whitespace, then you must use "\\s"
  • You can match "#" with "\\#" so that it's not interpreted as the beginning of a comment

What does the third example buy you? Now newlines and comments can be included. The third example (here document) above can also be written:

// here document string converted to a regex

regex =  '''(?ix)    # comments are now allowed!
            (\\d+)   # disk space
            \\s+
            \\d+%    # one or more numbers followed by "%"
            \\s+
            (/nfs/data.*)  # partition name'''
pattern = ~regex

This allows you to

  • mark capturing groups by number
    • I mark these with a numbered comment like "# 1: The disk space we want"
  • explain "landmark" keys in the pattern
    • For example "\\d% # a number followed by %". Not every line needs a comment, but don't leave out any important key matches.

Expanding on the example above:

/*
    The input string to parse is from the output of the 'df -k' command.
    Example input:
  
        Filesystem           1K-blocks      Used Available Use% Mounted on
        /dev/sda1              4185956   1206996   2762888  31% /
        /dev/sda11            30679784  28324040    772140  98% /extra
        <..lines omitted..>
        fas3050c-1-2.b:/vol/canis
                              10485760   6599936   3885824  63% /nfs/data_d2/dog_data
        fas6070-1-1.b:/vol/felis
                             314572800  54889600 259683200  18% /nfs/DATA-1/cat_data
   
    We want the available disk space in KB, for /nfs/data remote
    disk partitions, which is "3885824" and "259683200" in the sample
    above. Note that partitions that start out with "/nfs/data" may be
    either upper or lower case. (#2:) Capture the partition name for debugging.
   
    (#1:) Capture the space available in KB as the number before the
    percentage number (digits followed by '%'). The pattern match
    is relying on there being just one instance of a numeric
    percentage (\d+%) occuring in the each usable output line.
 
    The header line contains a "%", but the characters preceding it
    are non-numeric, "Use%", so it is ignored. Also, many of the lines
    we want are split across two lines, but it is only the second line
    that contains the information we want.
 
    The "(?i)" match flag indicates the pattern is case insensitive.

    The extended mode (?x) allows whitespace and comments starting with "#"
    to be embedded in the regular expression.
*/
 
regex =  '''(?ix)        # case insensitive, extended format
            (\\d+)       # 1: The disk space we want
            \\s+         # some whitespace
            \\d+%        # a number followed by %
            \\s+         # some more whitespace
            (/nfs/data.*)    # 2: partition name'''

pattern = ~regex

If that's not any easier to understand than what we started out with,

pattern = ~/(?i)(\d+)\s+\d+%\s+(\/nfs\/data.*)/

I'll just assume you're the sort of person who never reads code comments.

Include debugging feedback

  • use debugging lines
    • they are easy to turn on/off with a flag variable
    • they verify the regular expression is working

While developing regular expressions, you will probably want to be able to easily test the result. An easy option is to add debugging lines The debugging lines can be controlled by a boolean flag to turn them on or off. For little development programs and snippets in the groovyConsole, this is easier than setting up logging. The example above can be expanded with a 'debugging' flag and debugging lines like this:

/*
    The input string to parse is from...
    <...the rest of the header comment from above...>
*/
boolean debugging = true

if (debugging) {
    // Test data
    dfkOutput = '''
Filesystem           1K-blocks      Used Available Use% Mounted on
/dev/sda1              4185956   1206996   2762888  31% /
/dev/sda11            30679784  28324040    772140  98% /extra
fas3050c-1-2.b:/vol/canis
                      10485760   6599936   3885824  63% /nfs/data_d2/dog_data
fas6070-1-1.b:/vol/felis
                     314572800  54889600 259683200  18% /nfs/DATA-1/cat_data
'''
} else {
    // Real data
    dfkOutput = 'df -k'.execute().text
}

long kbAvail = 0

regex =  '''(?ix)      # enable case-insensitive matches, extended patterns
            (\\d+)     # 1: The disk space we want
            \\s+       # some whitespace
            \\d+%      # a number followed by %
            \\s+       # some more whitespace
            (/nfs/data.*)  # 2: partition name'''

pattern = ~regex
matcher = pattern.matcher(dfkOutput)

if (debugging) {
    println """matcher pattern: 
/---------------------------------\\
${matcher.pattern()}
\\---------------------------------/"""

    println "match count=${matcher.getCount()}"
}

for (i=0; i < matcher.getCount(); i++) {
    if (debugging) {
        println "    text matched in matcher[${i}]: '" + matcher[i][0] + "'"
        println "        free space in  (group 1): '" + matcher[i][1] + "'"
        println "        partition name (group 2): '" + matcher[i][2] + "'"
    }
    kbAvail += matcher[i][1].toLong()
}

println "KB available=${kbAvail}"

With 'debugging = true', this prints some information to show the regular expression is working on the test data:

matcher pattern: 
/---------------------------------\
(?ix)      # enable case-insensitive matches, extended patterns
            (\d+)     # 1: The disk space we want
            \s+       # some whitespace
            \d+%      # a number followed by %
            \s+       # some more whitespace
            (/nfs/data.*)  # 2: partition name
\---------------------------------/
match count=2
    text matched in matcher[0]: '3885824  63% /nfs/data_d2/dog_data'
        free space in  (group 1): '3885824'
        partition name (group 2): '/nfs/data_d2/dog_data'
    text matched in matcher[1]: '259683200  18% /nfs/DATA-1/cat_data'
        free space in  (group 1): '259683200'
        partition name (group 2): '/nfs/DATA-1/cat_data'
KB available=263569024

You can see by the output above that most of the input is ignored because it doesn't meet the described pattern. For those entries that are split across two lines, it turns out that all the information we want is in the second line, which still meets the pattern criteria, and the first line is ignored for not matching.

And if you set 'debugging = false', only the result is printed:

KB available=263569024

Pattern Match Flags

The Java regular expression support includes many options modeled after Perl, which is one of the strongest regular expression parsing languages. Since Groovy gets its regular expression capability from Java (which copied Perl), what works in Java applies equally well to Groovy. Looking at the Java java.util.regex.Pattern API we see that there is support for pattern match flags under the section called "Special constructs (non-capturing)." Specifically the line indicating

(?idmsux-idmsux)  	Nothing, but turns match flags on - off

These capture nothing, but activate the match flags "idmsux". These correspond mostly to similarly named flags in Perl:

Match Flags
Flag Java/Groovy Perl Description
i CASE_INSENSITIVE ignore case Do case insensitive pattern matching
d UNIX_LINES not in Perl Enables Unix lines mode, only '\n' line terminator affects ., ^ and $
m MULTILINE multiline Enables multiline mode. In multiline mode the expressions ^ and $ match just after or just before, respectively, a line terminator or the end of the input sequence. By default these expressions only match at the beginning and the end of the entire input sequence.
s DOTALL single line In Perl this is called Single-line mode, treating the input as as single line even if it includes line terminators. Normally the "." wildcard doesn't match line terminators, but in Dotall mode it matches all characters.
u UNICODE_CASE not in Perl Enables Unicode-aware case folding.
x COMMENTS eXtended Extended mode allows whitespace, including newlines, and comments beginning with "#" and ending with a newline. Since whitespace in the pattern is ignored, use '\s' to indicate whitespace you wish to match.
g automatic in Java/Groovy global Global matching keeps track of a current position in the input, so you can step through each place the pattern matches the input. Groovy sets up a result array that is an array of arrays of strings. The first dimension represents the number of matches, the second dimension contains the substrings that represent actual text matched and any captured groups.

Example usage of Match Flags

This is how the Match Flags may be used in Groovy. Below is a pattern that will grab the disk space value from the output of the unix "df" (disk freespace) command, but only on lines containing "/nfs/data", the "(?i)" indicates a case insensitive match:

pattern = ~/(?i)(\d+)\s+\d+%\s+(\/nfs\/data.*)/

Note that the "(?i)" match flag comes at the beginning and is wrapped in parentheses. The "?" indicates this is a non-capturing group. Normally anything in a regular expression you wish to capture is wrapped in parentheses. Assuming the match over all succeeded, this "captured" group is available for later use as a substring that matched the portion of the regular expression in the parentheses.

For comparison, here is the equivalent pattern in Perl:

$dfkOutput =~ /(\d+)\s+\d+%\s+(\/nfs\/data.*)/i

Notice that the "i" indicating a case-insensitive match is appended at the end of the pattern.

The eXtended Pattern Match Flag (x)

The key flag to making regular expressions more readable is the "x" flag, which meant "eXtended" in Perl, and in Java/Groovy refers to "Comments," in a less mnemonic way.

Although Java and Groovy use the java.util.regex.Pattern API, which was modeled after Perl regular expressions, the extended mode wasn't of much use in Java. You could write

// Java regex with extended format
String patternStr = 
               "(?ix)           # case insensitive, extended format\n" +
               "(\\d+)          # 1: The disk space we want\n" +
               "\\s+            # some whitespace\n" +
               "\\d+%           # a number followed by %\n" +
               "\\s+            # some more whitespace\n" +
               "(/nfs/data.*)   # 2: partition name";

Pattern pattern = Pattern.compile(patternStr);
Matcher m = pattern.matcher(dfOutput);

But it would be a little less typing to simulate this with ordinary comments:

// Java regex with comments simulating extended format
String patternStr =
               "(?i)" +         // case insensitive
               "(\\d+)" +       // 1: The disk space we want
               "\\s+" +         // some whitespace
               "\\d+%" +        // a number followed by %
               "\\s+" +         // some more whitespace
               "(/nfs/data.*)"; // 2: partition name

Pattern pattern = Pattern.compile(patternStr);
Matcher m = pattern.matcher(dfOutput);

However, in Groovy a regex combined with a here document, this becomes much cleaner (and more Perl-like):

// Groovy regex
regex =  '''(?ix)          # enable case-insensitive matches, extended patterns
            (\\d+)         # 1: The disk space we want
            \\s+           # some whitespace
            \\d+%          # a number followed by %
            \\s+           # some more whitespace
            (/nfs/data.*)  # 2: partition name'''

pattern = ~regex
matcher = pattern.matcher(dfOutput)

For comparison, here's the Perl equivalent:

# Perl regex
$dfOutput =~ 
    m!              # begin match
        (\d+)          # 1: Disk space we're after
        \s+            # one+ whitespace characters
        \d+%           # digits followed by '%'
        \s+            # one+ whitespace characters
        (/nfs/data.*)  # 2: partition name
    !ix;            # end match with case-(i)nsensitive
                    # and e(x)tended format options

In Perl you can use "m<delim>" to start a match instead of "/", so I could pick "!" as the delimiter and avoid having to escape the "/" characters in the partition name. Also Perl regular expressions use a single backslash, as in Groovy single line slashy regex patterns. Unfortunately, for multi-line Groovy patterns defined as a here document string, you need to double the backslashes.

Other Notes

Test code fragments in the groovyConsole

Groovy takes a couple of seconds to compile and run, and during iterative testing of a regular expression, this time becomes noticeable. Paste your code fragment into groovyConsole, set up some test input and a print statement to show the result. You can then run rapidly there without the compile lag. The print statement can become your debugging line when copying code back to your main project.

Conquer Complex Patterns by Dividing them into Simpler Categories

Beware of trying to deal with complicated patterns with a "one-size-fits-all" monster regular expression. Often it's easier to use an initial pattern to decide the category of input, and then select (if-then-else or switch-case) an appropriate simpler regex that deals with just that sub-category of input. You may need to include messages to warn against unexpected input that doesn't match any of the known categories.

Personal tools