GeoMesa Jobs ============ This project (``geomesa-accumulo/geomesa-accumulo-jobs`` in the source distribution) contains Map-Reduce jobs for maintaining GeoMesa Accumulo. Building Instructions --------------------- If you wish to build ``geomesa-accumulo-jobs`` separately, you can with Maven: .. code-block:: shell $ mvn clean install -pl geomesa-accumulo/geomesa-accumulo-jobs -am GeoMesa Input and Output Formats -------------------------------- GeoMesa provides input and output formats that can be used in Hadoop map/reduce jobs. The input/output formats can be used directly in Scala, or there are Java interfaces under the ``interop`` package. There are sample jobs provided that can be used as templates for more complex operations. These are:: org.locationtech.geomesa.accumulo.jobs.mapreduce.interop.FeatureCountJob org.locationtech.geomesa.accumulo.jobs.mapreduce.interop.FeatureWriterJob GeoMesaAccumuloInputFormat ^^^^^^^^^^^^^^^^^^^^^^^^^^ The ``GeoMesaAccumuloInputFormat`` can be used to get ``SimpleFeature``\ s into your jobs directly from GeoMesa. Use the static ``configure`` method to set up your job. You need to provide it with a map of connection parameters, which will be used to retrieve the GeoTools DataStore. You also need to provide a feature type name. Optionally, you can provide a CQL filter, which will be used to select a subset of features in your store. The key provided to your mapper with be a ``Text`` with the ``SimpleFeature`` ID. The value will be the ``SimpleFeature``. GeoMesaOutputFormat ^^^^^^^^^^^^^^^^^^^ The ``GeoMesaOutputFormat`` can be used to write ``SimpleFeature``\ s back into GeoMesa. Use the static ``setOutput`` method to set up your job. You need to provide it with a map of connection parameters, which will be used to retrieve the GeoTools ``DataStore``. The key you output does not matter, and will be ignored. The value should be a ``SimpleFeature`` that you wish to write. If the ``SimpleFeatureType`` associated with the ``SimpleFeature`` does not yet exist in GeoMesa, it will be created for you. You may write different ``SimpleFeatureType``\ s, in a single job, if desired. Map/Reduce Jobs --------------- The following instructions require that you use the ``-libjars`` argument to ensure the correct JARs are available on the distributed classpath. .. note:: In the following examples, replace ``${VERSION}`` with the appropriate Scala plus GeoMesa versions (e.g. |scala_release_version|). .. _attribute_indexing_job: Attribute Indexing ^^^^^^^^^^^^^^^^^^ GeoMesa provides indexing on attributes to improve certain queries. You can indicate attributes that should be indexed when you create your schema (simple feature type). If you decide later on that you would like to index additional attributes, you can use the attribute indexing job. You only need to run this job once; the job will create attribute indices for each attribute listed in ``--geomesa.index.attributes``. The job can be invoked through Yarn as follows: .. code-block:: shell geomesa-accumulo$ yarn jar geomesa-accumulo-jobs/target/geomesa-accumulo-jobs_${VERSION}.jar \ org.locationtech.geomesa.accumulo.jobs.index.AttributeIndexJob \ --geomesa.input.instanceId \ --geomesa.input.zookeepers \ --geomesa.input.user \ --geomesa.input.password \ --geomesa.input.tableName \ --geomesa.input.feature \ --geomesa.index.coverage \ # optional attribute --geomesa.index.attributes .. note:: You will also need to include an extensive ``-libjars`` argument with all dependent JARs. .. _update_index_format_job: Updating Existing Data to the Latest Index Format ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ The indexing in GeoMesa is constantly being improved. We strive to maintain backwards compatibility, but old data can't always take advantage of the improvements we make. However, old data can be updated through the ``SchemaCopyJob``. This will copy it to a new table (or feature name), rewriting all the data using the latest codebase. Once the data is updated, you can drop the old tables and rename the new tables back to the original names. The job can be invoked through Yarn as follows (JAR version may vary slightly): .. code-block:: shell geomesa-accumulo$ yarn jar geomesa-accumulo-jobs/target/geomesa-accumulo-jobs_${VERSION}.jar \ org.locationtech.geomesa.accumulo.jobs.index.SchemaCopyJob \ --geomesa.input.instanceId \ --geomesa.output.instanceId \ --geomesa.input.zookeepers \ --geomesa.output.zookeepers \ --geomesa.input.user \ --geomesa.output.user \ --geomesa.input.password \ --geomesa.output.password \ --geomesa.input.tableName \ --geomesa.output.tableName \ --geomesa.input.feature \ --geomesa.output.feature \ --geomesa.input.cql .. note:: You will also need to include an extensive ``-libjars`` argument with all dependent JARs.