14.6. Transformation Function Usage

14.6.1. Control Functions

14.6.1.1. try

Description: Execute another function - if it fails, instead use a default value

Usage: try($1, $2)

Example: try("1"::int, 0) = 1

Example: try("abcd"::int, 0) = 0

14.6.2. String Functions

14.6.2.1. stripQuotes

Description: Remove double quotes from a string.

Usage: stripQuotes($1)

Example: stripQuotes('fo"o') = foo

14.6.2.2. length

Description: Returns the length of a string.

Usage: length($1)

Example: length('foo') = 3

14.6.2.3. trim

Description: Trim whitespace from around a string.

Usage: trim($1)

Example: trim('  foo ') = foo

14.6.2.4. capitalize

Description: Capitalize a string.

Usage: capitalize($1)

Example: capitalize('foo') = Foo

14.6.2.5. lowercase

Description: Lowercase a string.

Usage: lowercase($1)

Example: lowercase('FOO') = foo

14.6.2.6. uppercase

Description: Uppercase a string.

Usage: uppercase($1)

Example: uppercase('foo') = FOO

14.6.2.7. regexReplace

Description: Replace a given pattern with a target pattern in a string.

Usage: regexReplace($regex, $replacement, $1)

Example: regexReplace('foo'::r, 'bar', 'foobar') = barbar

14.6.2.8. concatenate

Description: Concatenate two strings.

Usage: concatenate($0, $1)

Example: concatenate('foo', 'bar') = foobar

14.6.2.9. substring

Description: Return the substring of a string.

Usage: substring($1, $startIndex, $endIndex)

Example: substring('foobarbaz', 2, 5) = oba

14.6.2.10. toString

Description: Convert another data type to a string.

Usage: toString($0)

Example: concatenate(toString(5), toString(6)) = '56'

14.6.3. Date Functions

14.6.3.1. now

Description: Use the current system time.

Usage: now()

14.6.3.2. date

Description: Custom date parser.

Usage: date($format, $1)

Example: date('YYYY-MM-dd\'T\'HH:mm:ss.SSSSSS', '2015-01-01T00:00:00.000000')

14.6.3.3. dateTime

Description: A strict ISO 8601 Date parser for format yyyy-MM-dd'T'HH:mm:ss.SSSZZ.

Usage: dateTime($1)

Example: dateTime('2015-01-01T00:00:00.000Z')

14.6.3.4. basicDate

Description: A basic date format for yyyyMMdd.

Usage: basicDate($1)

Example: basicDate('20150101')

14.6.3.5. basicDateTime

Description: A basic format that combines a basic date and time for format yyyyMMdd'T'HHmmss.SSSZ.

Usage: basicDateTime($1)

Example: basicDateTime('20150101T000000.000Z')

14.6.3.6. basicDateTimeNoMillis

Description: A basic format that combines a basic date and time with no millis for format yyyyMMdd'T'HHmmssZ.

Usage: basicDateTimeNoMillis($1)

Example: basicDateTimeNoMillis('20150101T000000Z')

14.6.3.7. dateHourMinuteSecondMillis

Description: Formatter for full date, and time keeping the first 3 fractional seconds for format yyyy-MM-dd'T'HH:mm:ss.SSS.

Usage: dateHourMinuteSecondMillis($1)

Example: dateHourMinuteSecondMillis('2015-01-01T00:00:00.000')

14.6.3.8. millisToDate

Description: Create a new date from a long representing milliseconds since January 1, 1970.

Usage: millisToDate($1)

Example: millisToDate('1449675054462'::long)

14.6.3.9. secsToDate

Description: Create a new date from a long representing seconds since January 1, 1970.

Usage: secsToDate($1)

Example: secsToDate(1449675054)

14.6.4. Geometry Functions

14.6.4.1. point

Description: Parse a Point geometry from lon/lat or WKT.

Usage: point($lon, $lat) or point($wkt)

Note: Ordering is important here...GeoMesa defaults to longitude first

Example: Parsing lon/lat from JSON:

Parsing lon/lat

# config
{ name = "lon", json-type="double", path="$.lon" }
{ name = "lat", json-type="double", path="$.lat" }
{ name = "geom", transform="point($lon, $lat)" }

# data
{
    "lat": 23.9,
    "lon": 24.2,
}

Example: Parsing lon/lat from text without creating lon/lat fields:

# config
{ name = "geom", transform="point($2::double, $3::double)"

# data
id,lat,lon,date
identity1,23.9,24.2,2015-02-03

Example: Parsing WKT as a point

# config
{ name = "geom", transform="point($2)" }

# data
ID,wkt,date
1,POINT(2 3),2015-01-02

14.6.4.2. linestring

Description: Parse a linestring from a WKT string.

Usage: linestring($0)

Example: linestring('LINESTRING(102 0, 103 1, 104 0, 105 1)')

14.6.4.3. polygon

Description: Parse a polygon from a WKT string.

Usage: polygon($0)

Example: polygon('polygon((100 0, 101 0, 101 1, 100 1, 100 0))')

14.6.4.4. geometry

Description: Parse a geometry from a WKT string or GeoJson.

Usage: geometry($0)

Example: Parsing WKT as a geometry

# config
{ name = "geom", transform="geometry($2)" }

# data
ID,wkt,date
1,POINT(2 3),2015-01-02

Example: Parsing GeoJson geometry

# config
{ name = "geom", json-type = "geometry", path = "$.geometry" }

# data
{
    id: 1,
    number: 123,
    color: "red",
    "geometry": {"type": "Point", "coordinates": [55, 56]}
}

14.6.5. ID Functions

14.6.5.1. stringToBytes

Description: Converts a string to a UTF-8 byte array (to pass to md5() or base64()).

Usage: stringToBytes($0)

Example: stringToBytes('row,of,data')

14.6.5.2. md5

Description: Creates an MD5 hash from a byte array.

Usage: md5($0)

Example: md5(stringToBytes('row,of,data'))

14.6.5.3. uuid

Description: Generates a random UUID.

Usage: uuid()

14.6.5.4. base64

Description: Encodes a byte array as a base-64 string.

Usage; base64($0)

Example: base64(stringToBytes('foo'))

14.6.6. Type Conversions

14.6.6.1. ::int or ::integer

Description: Converts a string into an integer. Invalid values will cause the record to fail.

Example: '1'::int = 1

14.6.6.2. ::long

Description: Converts a string into a long. Invalid values will cause the record to fail.

Example: '1'::long = 1L

14.6.6.3. ::float

Description: Converts a string into a float. Invalid values will cause the record to fail.

Example: '1.0'::float = 1.0f

14.6.6.4. ::double

Description: Converts a string into a double. Invalid values will cause the record to fail.

Example: '1.0'::double = 1.0d

14.6.6.5. ::boolean

Description: Converts a string into a boolean. Invalid values will cause the record to fail.

Example: 'true'::boolean = true

14.6.6.6. ::r

Description: Converts a string into a Regex object.

Example: 'f.*'::r = f.*: scala.util.matching.Regex

14.6.6.7. stringToInt or stringToInteger

Description: Converts a string into a integer, with a default value if conversion fails.

Usage; stringToInt($1, $2)

Example: stringToInt('1', 0) = 1

Example: stringToInt('', 0) = 0

14.6.6.8. stringToLong

Description: Converts a string into a long, with a default value if conversion fails.

Usage; stringToLong($1, $2)

Example: stringToLong('1', 0L) = 1L

Example: stringToLong('', 0L) = 0L

14.6.6.9. stringToFloat

Description: Converts a string into a float, with a default value if conversion fails.

Usage; stringToFloat($1, $2)

Example: stringToFloat('1.0', 0.0f) = 1.0f

Example: stringToFloat('not a float', 0.0f) = 0.0f

14.6.6.10. stringToDouble

Description: Converts a string into a double, with a default value if conversion fails.

Usage; stringToDouble($1, $2)

Example: stringToDouble('1.0', 0.0) = 1.0d

Example: stringToDouble(null, 0.0) = 0.0d

14.6.6.11. stringToBoolean

Description: Converts a string into a boolean, with a default value if conversion fails.

Usage; stringToBoolean($1, $2)

Example: stringToBoolean('true', false) = true

Example: stringToBoolean('55', false) = false

14.6.7. List and Map Parsing

14.6.7.1. parseList

Description: Parse a List[T] type from a string.

If your SimpleFeatureType config contains a list or map you can easily configure a transform function to parse it using the parseList function which takes either 2 or 3 args

  1. The primitive type of the list (int, string, double, float, boolean, etc)
  2. The reference to parse
  3. Optionally, the list delimiter (defaults to a comma)

Here’s some sample CSV data:

ID,Name,Age,LastSeen,Friends,Lat,Lon
23623,Harry,20,2015-05-06,"Will, Mark, Suzan",-100.236523,23
26236,Hermione,25,2015-06-07,"Edward, Bill, Harry",40.232,-53.2356
3233,Severus,30,2015-10-23,"Tom, Riddle, Voldemort",3,-62.23

For example, an SFT may specific a field:

{ name = "friends", type = "List[String]" }

And a transform to parse the quoted CSV field:

{ name = "friends", transform = "parseList('string', $5)" }

14.6.7.2. parseMap

Description: Parse a Map[T,V] type from a string.

Parsing Maps is similar. Take for example this CSV data with a quoted map field:

1,"1->a,2->b,3->c,4->d",2013-07-17,-90.368732,35.3155
2,"5->e,6->f,7->g,8->h",2013-07-17,-70.970585,42.36211
3,"9->i,10->j",2013-07-17,-97.599004,30.50901

Our field type is:

numbers:Map[Integer,String]

Then we specify a transform:

{ name = "numbers", transform = "parseMap('int -> string', $2)" }

Optionally we can also provide custom list/record and key-value delimiters for a map:

{ name = "numbers", transform = "parseMap('int -> string', $2, ',', '->')" }