11.2. Using the Kafka Data Store Programmatically

To create a KafkaDataStore there are two required properties, one for the Apache Kafka connection, “brokers”, and one for the Apache Zookeeper connection, “zookeepers”. An optional parameter, “zkPath” is used to specify a path in Zookeeper under which schemas are stored. If no “zkPath” is specified then a default path will be used. Another optional parameter, “isProducer”, is used to create a KafkaDataStore in producer or consumer mode. This parameter defaults to false, i.e. by default a Kafka Consumer Data Store will be created. The same set of configuration parameters, with the exception of “isProducer” must be used to create both the Kafka Producer Data Store and the Kafka Consumer Data Store.

After a KafkaDataStore has been created, additional simple feature type specific hints must be provided. These hints are stored in the user data of the SimpleFeatureType. Use the KafkaDataStoreHelper to create a copy of your SimpleFeatureType with the hints added. Then call dataStore.createSchema(sft) where sft is the SimpleFeatureType returned by the KafkaDataStoreHelper. This must be done once per Simple Feature Type. The Simple Feature Type along with hints are stored in Zookeeper so if createSchema(sft) is called on the Kafka Data Store Producer, it cannot be called on the Kafka Consumer Data Store Consumer.

11.2.1. Parameters

The Kafka Data Store takes several parameters.

Parameter Type Description
brokers * String Comma-separated list of Kafka brokers
zookeepers * String Comma-separated list of Zookeeper servers
zkPath String Path to GeoMesa Kafka data in Zookeeper
namespace String Namespace
partitions Integer Number of partitions to use in Kafka topics
replication Integer Replication factor to use in Kafka topics
isProducer Boolean Indicates whether the Kafka Data Store is a producer
expirationPeriod Long Feature will be “auto-dropped” (expired) after this delay in milliseconds. Leave black or use -1 to not drop features.
cleanUpCache Boolean If true, run a thread to clean up the live feature cache every second if set to true. False by default.
autoOffsetReset String What offset to reset to when there is no initial offset in ZooKeeper or if an offset is out of range. Valid values are smallest or largest. Default value is largest.

The required parameters are marked with an asterisk.