11.4. Kafka Command Line Tools¶

Run geomesa-kafka without any arguments to produce the following usage text:

$ geomesa-kafka
  Usage: geomesa-kafka [command] [command options]
    Commands:
      classpath       Display the GeoMesa classpath
      configure       Configure the local environment for GeoMesa
      convert         Convert files using GeoMesa's internal SFT converter framework
      create-schema   Create a feature definition in GeoMesa
      get-schema      Describe the attributes of a given feature in GeoMesa
      get-names       List GeoMesa features for a given zkPath
      help            Show help
      listen          Listen to a GeoMesa Kafka topic
      remove-schema   Remove a schema and associated features from GeoMesa
      version         GeoMesa Version

This usage text lists the available commands. To see help for an individual command, run geomesa-kafka help <command-name>, which for example will give you something like this:

$ geomesa-kafka help get-names
  List GeoMesa features for a given zkPath
  Usage: get-names [options]
    Options:
    * -b, --brokers
         Brokers (host:port, comma separated)
      -p, --zkpath
         Zookeeper path where feature schemas are saved
    * -z, --zookeepers
         Zookeepers (host[:port], comma separated)

11.4.1. Command overview¶

11.4.1.1. configure¶

Used to configure the current environment for using the commandline tools. This is frequently run after the tools are first installed to ensure the environment is configured correctly:

$ geomesa configure

11.4.1.2. classpath¶

Prints out the current classpath configuration:

$ geomesa classpath

11.4.1.3. create-schema¶

Used to create a feature type (SimpleFeatureType) at the specified zkpath:

$ geomesa-kafka create-schema -f testfeature \
  -z zoo1,zoo2,zoo3 \
  -b broker1:9092,broker2:9092 \
  -s fid:String:index=true,dtg:Date,geom:Point:srid=4326 \
  -p /geomesa/ds/kafka

11.4.1.4. get-schema¶

Display details about the attributes of a specified feature type:

$ geomesa-kafka get-schema -f testfeature -z zoo1,zoo2,zoo3 \
  -b broker1:9092,broker2:9092 -p /geomesa/ds/kafka

11.4.1.5. get-names¶

List all known feature types in Kafka:

$ geomesa-kafka get-names -z zoo1,zoo2,zoo3 -b broker1:9092,broker2:9092

If no --zkpath parameter is specified, the get-names command will search all of zookeeper for potential feature types.

11.4.1.6. listen¶

Logs out the messages written to a topic corresponding to the feature type passed in.

$ geomesa-kafka listen -f testfeature

-z zoo1,zoo2,zoo3 -b broker1:9092,broker2:9092 -p /geomesa/ds/kafka –from-beginning

11.4.1.7. remove-schema¶

Used to remove a feature type (SimpleFeatureType) in a GeoMesa catalog. This will also delete any feature of that type in the data store:

$ geomesa-kafka remove-schema -f testfeature \
  -z zoo1,zoo2,zoo3 \
  -b broker1:9092,broker2:9092 \
  -p /geomesa/ds/kafka
$ geomesa-kafka remove-schema --pattern 'testfeature\d+' \
  -z zoo1,zoo2,zoo3 \
  -b broker1:9092,broker2:9092 \
  -p /geomesa/ds/kafka

11.4.1.8. version¶

Prints out the version, git branch, and commit ID that the tools were built with:

$ geomesa version

11.4.2. Formatting GeoMesa Kafka Messages¶

The KafkaGeoMessageFormatter class, part of the geomesa-kafka-datastore module, may be used with the kafka-console-consumer Kafka command-line tool. In order to use this formatter, call kafka-console-consumer with these additional arguments:

Note

Replace $KAFKAVERSION below with the appropriate version number for your environment: 08, 09, or 10. e.g. org.locationtech.geomesa.kafka08.KafkaGeoMessageFormatter

--formatter org.locationtech.geomesa.kafka$KAFKAVERSION.KafkaGeoMessageFormatter
--property sft.name={sftName}
--property sft.spec={sftSpec}

In order to pass the spec via a command argument all % characters must be replaced by %37 and all = characters must be replaced by %61.

A slightly easier to use but slightly less flexible alternative is to use the KafkaDataStoreLogViewer instead of the kafka-console-consumer. To use the KafkaDataStoreLogViewer first copy the geomesa-kafka-geoserver-plugin.jar to $KAFKA_HOME/libs. Then create a copy of $KAFKA_HOME/bin/kafka-console-consumer.sh called “kafka-ds-log-viewer” and in the copy replace the classname in the exec command at the end of the script with org.locationtech.geomesa.kafka$KAFKAVERSION.KafkaDataStoreLogViewer.

The KafkaDataStoreLogViewer requires three arguments: --zookeeper, --zkPath, and --sftName. It also supports an optional argument --from which accepts values oldest and newest. oldest is equivalent to specifying --from-beginning when using the kafka-console-consumer and newest is equivalent to not specifying --from-beginning.

For example:

$ kafka-ds-log-viewer --zookeeper {zookeeper} --zkPath {zkPath} --sftName {sftName}

The KafkaDataStoreLogViewer loads the SimpleFeatureType from Zookeeper so it does not need to be passed via the command line.