9.18. Transformation Functions

9.18.1. Type Conversions

toInt or toInteger

Converts a value into a integer. If the conversion fails, returns null unless a default value is defined.

Function	toInt($value) toInt($value, $default)
Usage	toInt('1', 0) = 1 toInt('', 0) = 0 toInt('') = null

toLong

Converts a value into a long. If the conversion fails, returns null unless a default value is defined.

Function	toLong($value) toLong($value, $default)
Usage	toLong('1', 0L) = 1L toLong('', 0L) = 0L toLong('') = null

toFloat

Converts a value into a float. If the conversion fails, returns null unless a default value is defined.

Function	toFloat($value) toFloat($value, $default)
Usage	toFloat('1.0', 0.0f) = 1.0f toFloat('not a float', 0.0f) = 0.0f toFloat('') = null

toDouble

Converts a value into a double. If the conversion fails, returns null unless a default value is defined.

Function	toDouble($value) toDouble($value, $default)
Usage	toDouble('1.0', 0.0) = 1.0d toDouble(null, 0.0) = 0.0d toDouble('') = null

toBoolean

Converts a value into a Boolean. If the conversion fails, returns null unless a default value is defined. If the input is a number, zero will evaluate to false, and all other numbers will evaluate to true.

Function	toBoolean($value) toBoolean($value, $default)
Usage	toBoolean('true', false) = true toBoolean('foo', false) = false toBoolean(1) = true toBoolean(0) = false toBoolean('') = null

::int or ::integer

Converts a value into an integer. If the conversion fails, it will throw an error and cause the record to fail.

Function	$value::int
Usage	'1'::int = 1

::long

Converts a value into a long. If the conversion fails, it will throw an error and cause the record to fail.

Function	$value::long
Usage	'1'::long = 1L

::float

Converts a value into a float. If the conversion fails, it will throw an error and cause the record to fail.

Function	$value::float
Usage	'1.0'::float = 1.0f

::double

Converts a value into a double. If the conversion fails, it will throw an error and cause the record to fail.

Function	$value::double
Usage	'1.0'::double = 1.0d

::boolean

Converts a value into a Boolean. If the conversion fails, it will throw an error and cause the record to fail.

Function	$value::boolean
Usage	'true'::boolean = true

::r

Converts a string into a Regex object, mainly for use with the regexReplace function.

Function	$value::r
Usage	'f.*'::r = f.*: scala.util.matching.Regex

9.18.2. String Functions

strip

Removes characters from the start and end of a string. If the characters to strip are not specified, will strip whitespace by default.

Function	strip($value) strip($value, $chars)
Usage	strip('afoob', 'abc') = 'foo' strip('foao', 'abc') = 'foao' strip('\t foo ') = 'foo'

stripPrefix

Removes characters from the start of a string. If the characters to strip are not specified, will strip whitespace by default.

Function	stripPrefix($value) stripPrefix($value, $chars)
Usage	stripPrefix('afoob', 'abc') = 'foob'

stripSuffix

Removes characters from the end of a string. If the characters to strip are not specified, will strip whitespace by default.

Function	stripSuffix($value) stripSuffix($value, $chars)
Usage	stripSuffix('afoob', 'abc') = 'afoo'

stripQuotes

Remove double and single quotes from the start and end of a string.

Function	stripQuotes($value)
Usage	stripQuotes('"foo"') = 'foo' stripQuotes('\'foo\'') = 'foo' stripQuotes('fo"o') = 'fo"o'

replace

Replaces a literal string with another string.

Function	replace($value, $toReplace, $replacement)
Usage	replace('foobar', 'ob', 'ab') = 'foabar'

remove

Removes a substring from a string.

Function	remove($value, $substring)
Usage	remove('foabco', 'abc') = 'foo'

length

Returns the length of a string.

Function	length($value)
Usage	length('foo') = 3

trim

Trim whitespace from around a string.

Function	trim($value)
Usage	trim('  foo ') = 'foo'

capitalize

Capitalize a string.

Function	capitalize($value)
Usage	capitalize('foo') = 'Foo'

lowercase

Lowercase a string.

Function	lowercase($value)
Usage	lowercase('FOO') = 'foo'

uppercase

Uppercase a string.

Function	uppercase($value)
Usage	uppercase('foo') = 'FOO'

regexReplace

Replace a given pattern with a target pattern in a string. In the replacement String, a dollar sign $ followed by a number will be interpreted as a reference to a group in the matched pattern, with numbers 1 through 9 corresponding to the first nine groups, and 0 standing for the whole match. Any other character is an error. The backslash \ character will be interpreted as an escape character and can be used to escape the dollar sign.

Function	regexReplace($regex, $replacement, $value)
Usage	regexReplace('foo'::r, 'bar', 'foobar') = 'barbar'

concatenate

Concatenate two or more strings.

Function	concatenate($first, $second, ...)
Usage	concatenate('foo', 'bar') = 'foobar'

mkstring

Concatenate two or more strings with a delimiter between each one.

Function	mkstring($delimiter, $first, ...)
Usage	mkstring(',', 'foo', 'bar') = 'foo,bar'

substring

Extract a substring from a string.

Function	substring($value, $startIndex, $endIndex)
Usage	substring('foobarbaz', 2, 5) = 'oba'

toString

Convert another data type to a string.

Function	toString($value)
Usage	concatenate(toString(5), toString(6)) = '56'

emptyToNull

Replace an empty string with null.

Function	emptyToNull($value)
Usage	emptyToNull('') = null emptyToNull('foo') = 'foo'

printf

Format custom strings, using a Java Formatter.

Function	printf($pattern, $arg1, $arg2, ...)'
Usage	printf('%s-%s-%sT00:00:00.000Z', '2015', '01', '01') = '2015-01-01T00:00:00.000Z' printf('%2f', divide(-1, 2, 3)) = '-0.17'

9.18.3. Date Functions

now

Returns the current system time, as a Date.

Function

now()

date

Custom date parser. The date format is defined by the Java DateTimeFormatter class.

Function	date($format, $value)
Usage	date('yyyy-MM-dd\\'T\\'HH:mm:ss.SSSSSS', '2015-01-01T00:00:00.000000')

dateTime

A strict ISO 8601 Date parser for format yyyy-MM-dd'T'HH:mm:ss.SSSZZ.

Function	dateTime($value)
Usage	dateTime('2015-01-01T00:00:00.000Z')

basicIsoDate

A date format for yyyyMMdd, equivalent to java.time.format.DateTimeFormatter.BASIC_ISO_DATE.

Function	basicIsoDate($value)
Usage	basicIsoDate('20150101')

isoDate

A date format for yyyy-MM-dd, equivalent to java.time.format.DateTimeFormatter.ISO_DATE.

Function	isoDate($value)
Usage	isoDate('2015-01-01')

isoLocalDate

A date format for yyyy-MM-dd, equivalent to java.time.format.DateTimeFormatter.ISO_LOCAL_DATE.

Function	isoLocalDate($value)
Usage	isoLocalDate('2015-01-01')

basicDateTime

A date format that combines a basic date and time for yyyyMMdd'T'HHmmss.SSSZ.

Function	basicDateTime($value)
Usage	basicDateTime('20150101T000000.000Z')

basicDateTimeNoMillis

A basic format that combines a basic date and time for format yyyyMMdd'T'HHmmssZ.

Function	basicDateTimeNoMillis($value)
Usage	basicDateTimeNoMillis('20150101T000000Z')

isoDateTime

A date format for yyyy-MM-dd'T'HH:mm:ss, equivalent to java.time.format.DateTimeFormatter.ISO_DATE_TIME.

Function	isoDateTime($value)
Usage	isoDateTime('2015-01-01T00:00:00')

isoLocalDateTime

A date format for yyyy-MM-dd'T'HH:mm:ss, equivalent to java.time.format.DateTimeFormatter.ISO_LOCAL_DATE_TIME.

Function	isoLocalDateTime($value)
Usage	isoLocalDateTime('2015-01-01T00:00:00')

isoOffsetDateTime

A date format for yyyy-MM-dd'T'HH:mm:ssZ, equivalent to java.time.format.DateTimeFormatter.ISO_OFFSET_DATE_TIME.

Function	isoOffsetDateTime($value)
Usage	isoOffsetDateTime('2015-01-01T00:00:00Z')

dateHourMinuteSecondMillis

Formatter for full date and time, keeping the first 3 fractional seconds for format yyyy-MM-dd'T'HH:mm:ss.SSS.

Function	dateHourMinuteSecondMillis($value)
Usage	dateHourMinuteSecondMillis('2015-01-01T00:00:00.000')

millisToDate

Create a new date from a long representing milliseconds since the Unix epoch (January 1, 1970).

Function	millisToDate($value)
Usage	millisToDate(1449675054462L)

secsToDate

Create a new date from a long representing seconds since the Unix epoch (January 1, 1970).

Function	secsToDate($value)
Usage	secsToDate(1449675054)

dateToString

Formats a date as a string, based on a pattern as defined by the Java DateTimeFormatter.

Function	dateToString($pattern, $date)
Usage	dateToString('yyyy-MM-dd\\'T\\'HH:mm:ss.SSSSSS', now())

dateToMillis

Converts a date to milliseconds since the Unix epoch (January 1, 1970).

Function	dateToMillis($date)
Usage	dateToMillis(now())

9.18.4. Geometry Functions

point

Parse a Point geometry from WKT, WKB, or longitude, latitude, z and measure. To create a point with measure but no z, use pointM.

Function	point($lon, $lat) point($lon, $lat, $z) point($lon, $lat, $z, $m) point($wktOrWkb)
Usage	point(0.0, 0.0) point('POINT(2 3)')

pointM

Parse a Point geometry from longitude, latitude and measure.

Function	pointM($lon, $lat, $m)
Usage	pointM(10.0, 20.0, 30.0)

multipoint

Parse a multi-point from a WKT string, WKB byte array, or two lists of coordinates.

Function	multipoint($wktOrWkb) multipoint($longitudes, $latitudes)
Usage	multipoint('MULTIPOINT ((10 40), (40 30), (20 20), (30 10))') multipoint(list(10,40,20,30),list(40,30,20,10))

linestring

Parse a linestring from a WKT string, WKB byte array, or two lists of coordinates.

Function	linestring($wktOrWkb) linestring($longitudes, $latitudes)
Usage	linestring('LINESTRING(102 0, 103 1, 104 0, 105 1)') linestring(list(102,103,104,105),list(0,1,0,1))

multilinestring

Parse a multi-linestring from a WKT string or WKB byte array.

Function	multilinestring($wktOrWkb)
Usage	multilinestring('MULTILINESTRING ((10 10, 20 20, 10 40), (40 40, 30 30, 40 20, 30 10))')

polygon

Parse a polygon from a WKT string or WKB byte array.

Function	polygon($wktOrWkb)
Usage	polygon('POLYGON ((30 10, 40 40, 20 40, 10 20, 30 10))')

multipolygon

Parse a multi-polygon from a WKT string or WKB byte array.

Function	multipolygon($wktOrWkb)
Usage	multipolygon('MULTIPOLYGON (((30 20, 45 40, 10 40, 30 20)), ((15 5, 40 10, 10 20, 5 10, 15 5)))')

geometrycollection

Parse a geometry collection from a WKT string or WKB byte array.

Function	geometrycollection($wktOrWkb)
Usage	geometrycollection('GEOMETRYCOLLECTION(POINT(4 6),LINESTRING(4 6,7 10))')

geometry

Parse a geometry from a WKT string or WKB byte array. Note that you would generally only use this method if you are dealing with multiple geometry types, otherwise prefer a more specific method (e.g. point, linestring, etc).

Function	geometry($wktOrWkb)
Usage	geometry('POINT(2 3)')

projectFrom

Project a geometry from its native CRS to EPSG:4326. GeoMesa only supports EPSG:4326, so geometries must be transformed when ingesting from another CRS.

Function	projectFrom($epsg, $geom)
Usage	projectFrom('EPSG:3857', point(1689200.14,1113194.91))

9.18.5. ID Functions

stringToBytes

Converts a string to a UTF-8 byte array (to pass to other functions like md5()).

Function	stringToBytes($value)
Usage	stringToBytes('row,of,data')

md5

Creates an MD5 hash from a byte array. Note that MD5 is not as secure or as performant as other hashes, and should generally be avoided.

Function	md5($value)
Usage	md5(stringToBytes('row,of,data'))

murmur3_32

Creates a 32-bit murmur3 hash from a string.

Function	murmur3_32($value)
Usage	murmur3_32('row,of,data')

murmur3_64

Creates a 64-bit murmur3 hash from a string. Note that previously this function was incorrectly named murmur3_128, and can still be invoked by that name.

Function	murmur3_64($value)
Usage	murmur3_64('row,of,data')

murmurHash3

Creates a 128-bit murmur3 hash from a string or byte array, returned as a hex string.

Function	murmurHash3($value)
Usage	murmurHash3('row,of,data')

uuid

Generates a random UUID, returned as a string.

Function

uuid()

uuidZ3

Generates a Z3-based UUID for point geometries.

Function	uuidZ3($geom, $date, $interval)
Usage	uuidZ3(point('POINT (3 2)'), dateTime('2015-01-01T00:00:00.000Z'), 'week')

See Configuring Z-Index Time Interval for details on Z3 intervals.

uuidZ3Centroid

Generates a Z3-based UUID for non-point geometries.

Function	uuidZ3Centroid($geom, $date, $interval)
Usage	uuidZ3Centroid(linestring('LINESTRING(102 0, 103 1, 104 0, 105 1)', dateTime('2015-01-01T00:00:00.000Z'), 'week')

See Configuring Z-Index Time Interval for details on Z3 intervals.

9.18.6. Math Functions

The arguments to a math functions must be numbers - integers, doubles, floats, longs or numeric strings. All math functions return doubles. If another data type is needed, convert the value afterwards, e.g. add($first,$second)::long.

Most math functions accept multiple arguments, or will accept a single java.util.List containing the arguments.

add

Adds two or more values.

Function	add($value1, $value2, ...) add(list($value1, $value2))
Usage	add(1, 2) = 3.0

subtract

Subtracts two or more values sequentially.

Function	subtract($value1, $value2, ...) subtract(list($value1, $value2))
Usage	subtract(3,2) = 1.0 subtract(3,2,1) = 0.0 subtract(list(3,1)) = 2.0

multiply

Multiply two or more values.

Function	multiply($value1, $value2, ...) multiply(list($value1, $value2))
Usage	multiply(2,2,2) = 8.0 multiply(list(3,2)) = 6.0

divide

Divides two or more values sequentially.

Function	divide($value1, $value2, ...) divide(list($value1, $value2))
Usage	divide(24,2,3) = 4.0 divide(list(4,2)) = 2.0

modulo

Takes the modulo of two values. Note that unlike other math functions, modulo expects and returns integers instead of doubles.

Function	modulo($value1, $value2) modulo(list($value1, $value2))
Usage	modulo(5,2) = 1 modulo(list(7,5)) = 2

mean

Takes the mean (average) of two or more numbers.

Function	mean($value1, $value2, ...) mean(list($value1, $value2)
Usage	mean(10,9,8) = 9.0 mean(list(10,9,8)) = 9.0

min

Finds the minimum of two or more numbers. The numbers must all be of the same type, and the return value will be of the same type.

Function	min($value1, $value2, ...) min(list($value1, $value2)
Usage	min(10,9,8) = 8 min(list(1,2,3)) = 1

max

Finds the maximum of two or more numbers. The numbers must all be of the same type, and the return value will be of the same type.

Function	max($value1, $value2, ...) max(list($value1, $value2)
Usage	max(10,9,8) = 10 max(list(1,2,3)) = 3

sin

Calculates the sine of a number.

Function

sin($value)

cos

Calculates the cosine of a number.

Function

cos($value)

tan

Calculates the tangent of a number.

Function

tan($value)

asin

Calculates the arc sine of a number.

Function

asin($value)

acos

Calculates the arc cosine of a number.

Function

acos($value)

atan

Calculates the arc tangent of a number.

Function

atan($value)

ln

Calculates the natural logarithm of a number.

Function

ln($value)

exp

Calculates Euler’s number e raised to the power of a number.

Function

exp($value)

sqrt

Calculates the square root of a number.

Function

sqrt($value)

9.18.7. List and Map Functions

list

Creates a list from the input arguments.

Function

list($value1, $value2, ...)

listItem

Selects an element out of a list by index.

Function	listItem($list, $index)
Usage	listItem(list('1', '2', '3'), 0) = '1'

mapValue

Read a value out of a map by key.

Function	mapValue($map, $key)
Usage	mapValue(parseMap('string->int', 'a->1,b->2,c->3'), 'a') = 1

parseList

Parse a list from a string. Supported list item types are string, int, long, float, double, boolean bytes (for byte[]), uuid and date. If the delimiter is not specified, it defaults to a comma ,.

Function	parseList($itemType, $list) parseList($itemType, $list, $delimiter)
Usage	parseList('int', '1,2,3') = [ 1, 2, 3 ] parseList('int', '1:2:3', ':') = [ 1, 2, 3 ]

parseMap

Parse a map from a string. The key and value types must be specified as a string, separated by ->. Supported types are string, int, long, float, double, boolean bytes (for byte[]), uuid and date. Optionally, a record and key-value delimiter can be specified. If not specified, the record delimiter defaults to a comma ,, and the key-value delimiter defaults to an “arrow” ->.

Function	parseMap($keyValueTypes, $map) parseMap($keyValueTypes, $map, $keyValueDelimiter) parseMap($keyValueTypes, $map, $keyValueDelimiter, $recordDelimiter)
Usage	parseMap('string->int', 'a->1,b->2,c->3') = { a=1, b=2, c=3 } ] parseMap('string->int', 'a->1,b->2,c->3', '->') = { a=1, b=2, c=3 } parseMap('string->int', 'a->1,b->2,c->3', '->', ',') = { a=1, b=2, c=3 }

transformListItems

Applies a transform expression to every element of a list. The expression to apply must be defined as a string. The list item is available in the transform expression as $0. In the example shown, the list will be converted from List<String> to List<Double>.

Function	transformListItems($list, $expression)
Usage	transformListItems(list('1','2','3'), 'stringToDouble($0)') = [ 1.0, 2.0, 3.0 ]

9.18.8. Encoding Functions

base64Encode

Encodes a byte array as a base-64 URL-safe string.

Function	base64Encode($value)
Usage	base64Encode(stringToBytes('foo'))

base64Decode

Decodes a base-64 URL-safe encoded string into a byte array.

Function	base64Decode($value)
Usage	base64Decode('Zm9v')

9.18.9. Control Functions

try

Execute another expression - if it throws an exception, return a default value. Note that throwing exceptions in Java is relatively expensive, so try expressions that fail frequently should be avoided where possible. For instance, try($0::int, null) could be replaced with toInt($0), which is equivalent but will handle null values without throwing exceptions.

Function	try($value, $default)
Usage	try("1"::int, 0) = 1 try("abcd"::int, null) = null

withDefault

Replace a value with the first non-null alternative, if the value is null.

Function	withDefault($value, $default1, ...)
Usage	withDefault('foo', 'bar') = 'foo' withDefault('foo', 'bar', 'baz') = 'foo' withDefault(null, 'bar', 'baz') = 'bar' withDefault(null, null, 'baz') = 'baz'

require

Throw an exception if the value is null (which will fail the current record), otherwise return the value.

Function	require($value)
Usage	require('foo') = 'foo' require(null) // fails the current record

9.18.10. State Functions

inputFilePath

Provides the absolute path to the file being operated on, if available. Note that the file path is a variable, not a function, and is referenced through $ notation.

The file path may not always be available, depending on how the converter is being invoked. When invoked through the GeoMesa command-line tools or GeoMesa NiFi, it will be set appropriately.

Usage

$inputFilePath

lineNumber

Provides the current line number in the file being operated on, if available.

The line number may not always be available, depending on the converter being used. For some converters, line number may be an abstract concept. For example, in the Avro converter line number will refer to the number of the Avro record in the file.

Function

lineNumber()

9.18.11. Enrichment Functions

cacheLookup

The converter framework provides a mechanism for setting an attribute based on a lookup from a cache. See Enrichment Caches for details.

Function	cacheLookup($cacheName, $entityKey, $attributeKey)
Usage	cacheLookup('test', $userId, 'email')

9.18.12. CQL Functions

Most of the basic CQL functions are available as transform functions. To use one, invoke it like a regular function, prefixed with the cql namespace. For example, you can use the CQL buffer function to turn a point into a polygon:

cql:buffer($value, 2.0)

For more information on the various CQL functions, see the GeoServer filter function reference.

9.18.13. JSON/Avro Transformations

See JSON Transform Functions and Avro Transform Functions.