10.11. Avro Converter¶
The Avro converter handles data written by Apache Avro.
10.11.1. Configuration¶
The Avro converter supports the following configuration keys:
Key |
Type |
Description |
|---|---|---|
|
String |
Must be the string |
|
String |
The Avro schema used for parsing |
|
String |
The path to an Avro schema on the classpath |
|
List |
A list of paths to Avro schemas on the classpath |
Exactly one of schema, schema-file, or schema-files must be specified - see below for details.
10.11.2. Avro Formats¶
The Avro converter supports parsing Avro files in different formats. The schema configuration will determine what format the converter expects.
10.11.2.1. Object Container Files¶
For portability, some Avro files contain the schema
embedded in the file. To parse this type of file,
specify schema = "embedded".
10.11.2.2. Files with External Schemas¶
To parse files or IPC messages that do not have a schema, the schema must be provided externally. To parse this type of file,
specify the schema in one of two ways: as an inline schema string using schema = "<schema string>"; or as a schema file
available on the classpath using schema-file = "<path to file>".
10.11.2.3. Single Object Encoding¶
Single Object Encoding is a way to specify
just the fingerprint of a schema as a prefix of the message. The schemas themselves must still be provided externally. To parse
this type of message, specify schema-files = [ "<path to file 1>", "<path to file 2>" ].
10.11.2.4. Schema Registry¶
To parse messages where schemas are stored in a schema registry, see Avro Schema Registry Converter.
10.11.3. Transform Functions¶
The current Avro record being parsed is available to field transforms as $1. The original message bytes are available
as $0, which may be useful for generating consistent feature IDs.
In addition to the standard Transformation Functions, the Avro converter provides the following Avro-specific functions:
10.11.3.1. avroPath¶
Description: Extract values from nested Avro structures.
Usage: avroPath($ref, $pathString)
$ref- a reference object (avro root or extracted object)pathString- forward-slash delimited path strings
Avro paths are defined similarly to JSONPath or XPath, and allow you to extract specific fields out of an Avro record. An Avro path consists of forward-slash delimited strings. Each part of the path defines a field name with an optional predicate:
$type=<typename>- match the Avro schema type name on the selected element[$<field>=<value>]- match elements with a field named “field” and a value equal to “value”
For example, /foo$type=bar/baz[$qux=quux]. See the example below for a concrete example.
10.11.3.2. avroToJson¶
Description: Converts Avro objects to JSON strings.
Usage: avroToJson($ref)
$ref- a reference object (avro root or extracted object)
10.11.3.3. avroBinaryList¶
GeoMesa has a custom Avro schema for writing SimpleFeatures. List, map and UUID attributes are serialized as binary Avro fields. This function can read a serialized list-type attribute.
Description: Parses a binary Avro value as a list
Usage: avroBinaryList($ref)
10.11.3.4. avroBinaryMap¶
GeoMesa has a custom Avro schema for writing SimpleFeatures. List, map and UUID attributes are serialized as binary Avro fields. This function can read a serialized map-type attribute.
Description: Parses a binary Avro value as a map
Usage: avroBinaryMap($ref)
10.11.3.5. avroBinaryUuid¶
GeoMesa has a custom Avro schema for writing SimpleFeatures. List, map and UUID attributes are serialized as binary Avro fields. This function can read a serialized UUID-type attribute.
Description: Parses a binary Avro value as a UUID
Usage: avroBinaryUuid($ref)
10.11.4. Example Usage¶
For this example we’ll use the following Avro schema in a classpath file named schema.avsc:
{
"namespace": "org.locationtech",
"type": "record",
"name": "CompositeMessage",
"fields": [
{
"name": "content",
"type": [
{
"name": "DataObj",
"type": "record",
"fields": [
{
"name": "kvmap",
"type": {
"type": "array",
"items": {
"name": "kvpair",
"type": "record",
"fields": [
{ "name": "k", "type": "string" },
{ "name": "v", "type": ["string", "double", "int", "null"] }
]
}
}
}
]
},
{
"name": "OtherObject",
"type": "record",
"fields": [{ "name": "id", "type": "int"}]
}
]
}
]
}
This schema defines an avro file that has a field named content
which has a nested object which is either of type DataObj or
OtherObject. As an exercise, we can use avro tools to generate some
test data and view it:
java -jar avro-tools-1.11.4.jar random --schema-file schema.avsc -count 5 /tmp/avro
$ java -jar /tmp/avro-tools-1.11.4.jar tojson /tmp/avro
{"content":{"org.locationtech.DataObj":{"kvmap":[{"k":"thhxhumkykubls","v":{"double":0.8793488185997134}},{"k":"mlungpiegrlof","v":{"double":0.45718223406586045}},{"k":"mtslijkjdt","v":null}]}}}
{"content":{"org.locationtech.OtherObject":{"id":-86025408}}}
{"content":{"org.locationtech.DataObj":{"kvmap":[]}}}
{"content":{"org.locationtech.DataObj":{"kvmap":[{"k":"aeqfvfhokutpovl","v":{"string":"kykfkitoqk"}},{"k":"omoeoo","v":{"string":"f"}}]}}}
{"content":{"org.locationtech.DataObj":{"kvmap":[{"k":"jdfpnxtleoh","v":{"double":0.7748286862915655}},{"k":"bueqwtmesmeesthinscnreqamlwdxprseejpkrrljfhdkijosnogusomvmjkvbljrfjafhrbytrfayxhptfpcropkfjcgs","v":{"int":-1787843080}},{"k":"nmopnvrcjyar","v":null},{"k":"i","v":{"string":"hcslpunas"}}]}}}
Here’s a more relevant sample record:
{
"content" : {
"org.locationtech.DataObj" : {
"kvmap" : [ {
"k" : "lat",
"v" : {
"double" : 45.0
}
}, {
"k" : "lon",
"v" : {
"double" : 45.0
}
}, {
"k" : "prop3",
"v" : {
"string" : " foo "
}
}, {
"k" : "prop4",
"v" : {
"double" : 1.0
}
} ]
}
}
}
Let’s say we want to convert our Avro array of kvpairs into a simple feature. We notice that there are 4 attributes:
lat
lon
prop3
prop4
The following converter config would be sufficient to parse the Avro:
{
type = "avro"
schema-file = "schema.avsc"
id-field = "uuid()"
fields = [
{ name = "tobj", transform = "avroPath($1, '/content$type=DataObj')" },
{ name = "lat", transform = "avroPath($tobj, '/kvmap[$k=lat]/v')" },
{ name = "lon", transform = "avroPath($tobj, '/kvmap[$k=lon]/v')" },
{ name = "geom", transform = "point($lon, $lat)" }
]
}