.. _kafka_tools: Kafka Command-Line Tools ======================== The GeoMesa Kafka distribution includes a set of command-line tools for feature management, ingest, export and debugging. To install the tools, see :ref:`setting_up_kafka_commandline`. Once installed, the tools should be available through the command ``geomesa-kafka``:: $ geomesa-kafka INFO Usage: geomesa-kafka [command] [command options] Commands: ... Commands that are common to multiple back ends are described in :doc:`/user/cli/index`. The commands here are Kafka-specific. General Arguments ----------------- Most commands require you to specify the connection to Kafka. This generally includes the list of Kafka brokers and Zookeeper servers. Specify brokers with the ``--brokers`` (or ``-b``) argument, and specify Zookeepers with ``--zookeepers`` (or ``-z``). Kafka stores metadata under a particular path in Zookeeper - this can be thought of as a namespace for feature types. Use ``--zkpath`` (or ``-p``) to override the default path. Commands -------- ``create-schema`` ^^^^^^^^^^^^^^^^^ See :ref:`cli_create_schema` for an overview of this command. In addition to the regular options, Kafka allows the number of partitions and the replication factor of the Kafka topic to be specified. ======================== ================================================== Argument Description ======================== ================================================== ``--partitions`` The number of partitions used for the Kafka topic ``--replication`` The replication factor for the Kafka topic ======================== ================================================== ``export`` ^^^^^^^^^^ See :ref:`cli_export` for an overview of this command. Unlike the standard ``export``, this command will not not terminate until it is cancelled (through a shell interrupt) or until ``--max-features`` have been read. Thus, it can be used to monitor a topic. This command differs from the ``listen`` command (below) in that it allows filtering and output in various formats. It will also ignore ``drop`` and ``clear`` messages generated by feature deletion. In addition to the regular options, Kafka allows control over the consumer behavior: ======================== ================================================================================ Argument Description ======================== ================================================================================ ``--from-beginning`` Start reading messages from the beginning of the Kafka topic, instead of the end ``--num-consumers`` Number of consumers used to read the topic ======================== ================================================================================ The ``--num-consumers`` argument can be used to increase read speed. However, there can be at most one consumer per topic partition. The ``--from-beginning`` argument can be used to start reading the Kafka topic from the start. Otherwise, only new messages that are sent after this command is invoked will be read. ``ingest`` ^^^^^^^^^^ See :ref:`cli_ingest` for an overview of this command. In addition to the regular options, Kafka allows the number of partitions and the replication factor of the Kafka topic to be specified. In addition, an artificial delay can be inserted to simulate a live data stream. ======================== ================================================== Argument Description ======================== ================================================== ``--partitions`` The number of partitions used for the Kafka topic ``--replication`` The replication factor for the Kafka topic ``--delay`` The delay inserted between messages ======================== ================================================== The ``--delay`` argument should be specified as a duration, in plain language. For example, ``100 millis`` or ``1 second``. The ingest will pause after creating each ``SimpleFeature`` for the specified delay. This can be used to simulate a live data stream. ``listen`` ^^^^^^^^^^ This command behaves similarly to the ``export`` command (above), but it does not provide options for filtering or output format. It will show each message on the Kafka topic, including ``drop`` and ``clear`` messages generated from feature deletion. This command will not not terminate until it is cancelled (through a shell interrupt). ======================== ================================================================================ Argument Description ======================== ================================================================================ ``-f, --feature-name *`` The name of the schema ``--from-beginning`` Start reading messages from the beginning of the Kafka topic, instead of the end ``--num-consumers`` Number of consumers used to read the topic ======================== ================================================================================ The ``--num-consumers`` argument can be used to increase read speed. However, there can be at most one consumer per topic partition. The ``--from-beginning`` argument can be used to start reading the Kafka topic from the start. Otherwise, only new messages that are sent after this command is invoked will be read.