Undocumented shards/replicas configuration for Elasticsearch Exporter

[falko]

According to the configuration file there are properties for configuring sharding and replicas:
https://github.com/camunda/zeebe/blob/42cc7744e88ad4aac655f115a8a270f0556e6432/dist/src/main/config/broker.yaml.template#L665-L666

But they are not shown in the documentation:

camunda-docs/docs/self-managed/zeebe-deployment/exporters/elasticsearch-exporter.md

Lines 74 to 78 in a644fe5

	| Option | Description | Default |
	| ----------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | --------------- |
	| prefix | This prefix will be appended to every index created by the exporter; must not contain `_` (underscore). | zeebe-record |
	| createTemplate | If `true` missing indexes will be created automatically. | `true` |
	| indexSuffixDatePattern | This suffix will be appended to every index created by the exporter; The pattern is based on the Java [DateTimeFormater](https://docs.oracle.com/en/java/javase/21/docs/api/java.base/java/time/format/DateTimeFormatter.html) and supports the same syntax. This is useful when indexes should be created in a different interval, like hourly instead of daily. | `"yyyy-MM-dd'"` |

And their behavior is not explained anywhere.

[falko]

@daniel-ewing pointed out that the name of the corresponding environment variables may be hard to guess:

zeebe: 
  env: 
    - name: ZEEBE_BROKER_EXPORTERS_ELASTICSEARCH_ARGS_INDEX_NUMBEROFSHARDS
      value: "1"

ZEEBE_BROKER_EXPORTERS_ELASTICSEARCH_ARGS_INDEX_NUMBER_OF_SHARDS does not work.

[falko]

It may also be work documenting that the sharding configuration can be checked with an Elasticsearch query like:

curl --location 'http://127.0.0.1:9200/_cat/indices/zeebe*?v=true&s=index&pretty'

[falko]

@jothikiruthika and I discovered that the Elasticsearch Exporter creates indices with inconsistent number of shards.

Getting this wrong in the beginning may cause difficult migrations later. Therefore, this topic should be much better documented. The recommendation for maximum performance would be to set the environment variable ZEEBE_BROKER_EXPORTERS_ELASTICSEARCH_ARGS_INDEX_NUMBEROFSHARDS to match the number of Elasticsearch nodes. However, high availability can be reached by increasing the number of replicas. Maybe it would be helpful to link to some Elasticsearch documentation for more details on the topic. At least, we must make customers aware of these options and their effects.

[megglos]

Many thanks for raising this @falko !

I raised a PR to document these properties now #3773 and would consider this enough to close this issue.

@daniel-ewing pointed out that the name of the corresponding environment variables may be hard to guess:

zeebe:
env:
- name: ZEEBE_BROKER_EXPORTERS_ELASTICSEARCH_ARGS_INDEX_NUMBEROFSHARDS
value: "1"
ZEEBE_BROKER_EXPORTERS_ELASTICSEARCH_ARGS_INDEX_NUMBER_OF_SHARDS does not work.

This appears like a general issue to me that I consider worth a separate issue. Could either you or @daniel-ewing raise this to the docs repo so we can address that separately?

It may also be work documenting that the sharding configuration can be checked with an Elasticsearch query like:

curl --location 'http://127.0.0.1:9200/_cat/indices/zeebe*?v=true&s=index&pretty'

imho this goes beyond the scope of documenting the exporters but might fit a guide around the topic but I couldn't find any existing guide related to tuning elastic/opensearch shard configs.