diff --git a/docs/products/kafka/concepts/follower-fetching.md b/docs/products/kafka/concepts/follower-fetching.md index 9550e485f..c3f89e58f 100644 --- a/docs/products/kafka/concepts/follower-fetching.md +++ b/docs/products/kafka/concepts/follower-fetching.md @@ -6,8 +6,9 @@ sidebar_label: Follower fetching import Followerfetching from "@site/static/images/content/figma/follower-fetching.png"; import RelatedPages from "@site/src/components/RelatedPages"; -[Follower fetching](/docs/products/kafka/howto/enable-follower-fetching) in Aiven for Apache Kafka allows consumers to fetch data from the nearest replica instead of the leader. -This feature optimizes data fetching by leveraging Apache Kafka's rack awareness, which treats each availability zone (AZ) as a rack. +[Follower fetching](/docs/products/kafka/howto/enable-follower-fetching) in Aiven for Apache Kafka allows consumers to retrieve data from the nearest replica instead of always fetching from the partition leader. +This feature optimizes data fetching by leveraging Apache Kafka's rack awareness, which +treats each availability zone (AZ) as a rack. ## Benefits @@ -23,25 +24,31 @@ Google Cloud. ## How it works -Aiven for Apache Kafka uses rack awareness to optimize data fetching and ensure -high availability. +Aiven for Apache Kafka uses rack awareness to optimize data fetching and maintain +availability. Each availability zone (AZ) is treated as a rack. -Follow fetching +Follower fetching -- **Rack awareness**: Rack awareness distributes data across different physical racks, - also known as availability zones (AZs), within a data center. This distribution - ensures data replication for fault tolerance. Each Apache Kafka broker has - a `broker.rack` setting corresponding to its specific AZ: +### Rack awareness - - **AWS**: Uses AZ IDs (for example, `use1-az1`) instead of AZ names. - - **Google Cloud**: Uses AZ names directly (for example, `europe-west1-b`). +Rack awareness distributes data across different physical racks, also known as +availability zones, within a data center. This distribution ensures data replication for +fault tolerance. Each Apache Kafka broker has a `broker.rack` setting corresponding to +its specific AZ: -- **Follower fetching**: Uses rack awareness to allow consumers to fetch data from - the nearest replica, reducing latency and costs. Apache Kafka consumers use - the `client.rack` setting to specify their AZ, ensuring they fetch data from the - closest replica. +- **AWS:** Uses AZ IDs such as `use1-az1` +- **Google Cloud:** Uses AZ names such as `europe-west1-b` -### `broker.rack` and `client.rack` settings +Aiven for Apache Kafka automatically manages the `broker.rack` setting, eliminating the +need for manual configuration. + +### Follower fetching mechanism + +Follower fetching builds on rack awareness to allow consumers to fetch data from the +nearest replica. Apache Kafka consumers use the `client.rack` setting to specify their +AZ, ensuring they fetch data from the closest replica when possible. + +### Configuration settings - `broker.rack`: This setting corresponds to the AZ where each Apache Kafka broker is deployed and helps manage data replication efficiently. Apache Kafka brokers in @@ -56,6 +63,27 @@ high availability. same AZ. [Configure](/docs/products/kafka/howto/enable-follower-fetching#client-side-configuration) this setting to retrieve data from the closest replica. +## Follower fetching in Kafka Connect and MirrorMaker 2 + +Aiven for Apache Kafka® Connect and Aiven for Apache Kafka® MirrorMaker 2 apply rack +awareness when follower fetching is [enabled](/docs/products/kafka/howto/enable-follower-fetching) +on the Aiven for Apache Kafka service. Each service +sets a rack value based on the node’s availability zone (AZ) so that traffic is routed +to the closest replica. + +### Kafka Connect + +Kafka Connect assigns a rack value based on the AZ where each node runs. All source +connectors on that node inherit this value unless a connector specifies its own +consumer rack configuration. + +### MirrorMaker 2 + +MirrorMaker 2 assigns a rack value based on the node’s AZ when +`follower_fetching_enabled=true` in the service configuration. A custom `rack_id` in the +integration configuration overrides the AZ-based value. Rack awareness is not applied to +integrations that connect to external Kafka clusters. + - [Enable follower fetching in Aiven for Apache Kafka](/docs/products/kafka/howto/enable-follower-fetching) diff --git a/docs/products/kafka/howto/enable-follower-fetching.md b/docs/products/kafka/howto/enable-follower-fetching.md index 2c9fcdc84..47a22712f 100644 --- a/docs/products/kafka/howto/enable-follower-fetching.md +++ b/docs/products/kafka/howto/enable-follower-fetching.md @@ -25,6 +25,9 @@ Follower fetching is supported on AWS (Amazon Web Services) and Google Cloud. ## Identify availability zone +Before configuring client-side rack awareness, identify the AZs where your Kafka brokers +run. + - **AWS**: Availability zone (AZ) names can vary across different accounts. The same physical location might have different AZ names in different accounts. To ensure consistency when configuring `client.rack`, use the AZ ID, which remains the same @@ -39,8 +42,8 @@ Follower fetching is supported on AWS (Amazon Web Services) and Google Cloud. ## Enable follower fetching -Use either of the following methods to enable follower fetching on your -Aiven for Apache Kafka service: +Use one of the following methods to enable follower fetching on your Aiven for +Apache Kafka service. @@ -164,6 +167,32 @@ client.rack=europe-west1-d | Google Cloud | `europe-west1-c` | Fetch from the nearest replica in their AZ | Reduced latency and network costs | | Google Cloud | `europe-west1-d` | Fetch from the leader (no matching `broker.rack`) | No follower fetching possible | +## Use follower fetching with Kafka Connect and MirrorMaker 2 + +Aiven for Apache Kafka® Connect and Aiven for Apache Kafka® MirrorMaker 2 use follower +fetching when it is enabled on your Aiven for Kafka service. + +### Kafka Connect + +Kafka Connect sets `consumer.client.rack` based on each node’s availability zone. +To disable rack awareness for a specific connector, set: + +```json +{ + "consumer.override.client.rack": "noop" +} +``` + +### MirrorMaker 2 + +MirrorMaker 2 uses a rack ID based on the node’s availability zone when +`follower_fetching_enabled=true` (default). You can override this by setting a `rack_id` +in the integration configuration. + +Rack awareness does not apply to external Kafka clusters. + +See [Configure rack awareness in MirrorMaker 2](/docs/products/kafka/kafka-mirrormaker/howto/mm2-rack-awareness). + ## Verify follower fetching After configuring follower fetching, monitor for a decrease in cross-availability zone diff --git a/docs/products/kafka/kafka-mirrormaker/concepts/mirrormaker2-tuning.md b/docs/products/kafka/kafka-mirrormaker/concepts/mirrormaker2-tuning.md index 2fde5669b..66d75eb39 100644 --- a/docs/products/kafka/kafka-mirrormaker/concepts/mirrormaker2-tuning.md +++ b/docs/products/kafka/kafka-mirrormaker/concepts/mirrormaker2-tuning.md @@ -2,94 +2,98 @@ title: Configuration parameters for Aiven for Apache Kafka® MirrorMaker 2 --- -Learn about the configuration layers in Aiven for Apache Kafka® MirrorMaker 2, including service, replication flow, and integration settings. -Optimize data replication and performance in your Kafka ecosystem. +Learn about the service, replication-flow, and integration configuration layers in Aiven for Apache Kafka® MirrorMaker 2 and how they affect replication performance. ## Configuration layers -Aiven for Apache Kafka® MirrorMaker 2 configurations are organized into three layers: -**service**, **replication flow**, and **integration**. Each layer controls a specific -aspect of the replication process. +Aiven for Apache Kafka® MirrorMaker 2 uses three configuration layers: + +- **Service configurations** +- **Replication-flow configurations** +- **Integration configurations** + +Each layer manages a specific part of the replication process. ### Service configurations -Service configurations control the behavior of nodes and workers in the -Aiven for Apache Kafka® MirrorMaker 2 cluster. +Service configurations control the behavior of nodes and workers in the MirrorMaker 2 cluster. -**Example of a service configuration**: +**Example** -- Parameter: [`kafka_mirrormaker.emit_checkpoints_enabled`](https://aiven.io/docs/products/kafka/kafka-mirrormaker/reference/advanced-params#kafka_mirrormaker_emit_checkpoints_enabled) -- Description: Enables or disables periodically emitting consumer group offset - checkpoints to the target cluster. -- Impact: - - Automatically restarts the workers. - - Restarts all connectors and tasks. +- **Parameter:** [`kafka_mirrormaker.emit_checkpoints_enabled`](https://aiven.io/docs/products/kafka/kafka-mirrormaker/reference/advanced-params#kafka_mirrormaker_emit_checkpoints_enabled) +- **Description:** Enables or disables periodic emission of consumer group offset checkpoints to the target cluster +- **Impact:** + - Restarts workers + - Restarts all connectors and tasks ### Replication-flow configurations -Replication-flow configurations manage the behavior of connectors, such as Source, Sink, -Checkpoint, and Heartbeat. +Replication-flow configurations control the behavior of connectors such as Source, Sink, Checkpoint, and Heartbeat connectors. -**Example of a replication-flow configuration**: +**Example** -- Parameter: [`topics`](https://registry.terraform.io/providers/aiven/aiven/latest/docs/resources/mirrormaker_replication_flow) -- Description: Specifies a list of topics or regular expressions to replicate. - For more information, see the [topics included in a replication flow](/docs/products/kafka/kafka-mirrormaker/concepts/replication-flow-topics-regex). -- Impact: - - Automatically restarts the affected connectors. - - Restarts their associated tasks. +- **Parameter:** [`topics`](https://registry.terraform.io/providers/aiven/aiven/latest/docs/resources/mirrormaker_replication_flow) +- **Description:** Specifies a list of topics or regular expressions to replicate. + For details, see + [Topics included in a replication flow](/docs/products/kafka/kafka-mirrormaker/concepts/replication-flow-topics-regex). +- **Impact:** + - Restarts the affected connectors + - Restarts their tasks ### Integration configurations -Integration configurations fine-tune the interaction between producers and consumers -within connectors. +Integration configurations refine how producers and consumers behave within connectors. -**Example of an integration configuration**: +**Example** -- Parameter: [`consumer_fetch_min_bytes`](https://registry.terraform.io/providers/aiven/aiven/latest/docs/resources/service_integration#nested-schema-for-kafka_mirrormaker_user_configkafka_mirrormaker) -- Description: Sets the minimum amount of data the server should return for a fetch - request. -- Impact: - - Automatically restarts the workers. - - Restarts all connectors and tasks. +- **Parameter:** [`consumer_fetch_min_bytes`](https://registry.terraform.io/providers/aiven/aiven/latest/docs/resources/service_integration#nested-schema-for-kafka_mirrormaker_user_configkafka_mirrormaker) +- **Description:** Sets the minimum amount of data the server returns for a fetch request +- **Impact:** + - Restarts workers + - Restarts all connectors and tasks :::note -Most configuration parameters are derived from -[KIP-382: MirrorMaker 2.0 - Configuration Properties](https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=95650722#KIP382:MirrorMaker2.0-ConnectorConfigurationProperties). Refer to this resource for additional details. +Many configuration parameters originate from +[KIP-382: MirrorMaker 2.0 configuration properties](https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=95650722#KIP382:MirrorMaker2.0-ConnectorConfigurationProperties). ::: ## Common parameters -This section describes common parameters that can be adjusted to optimize the performance -and behavior of Aiven for Apache Kafka MirrorMaker 2 replication. - -1. **Optimize task allocation**: - Increase the value of - [`kafka_mirrormaker.tasks_max_per_cpu`](/docs/products/kafka/kafka-mirrormaker/reference/advanced-params#kafka_mirrormaker_tasks_max_per_cpu) - in the advanced configuration. - Setting this to match the number of partitions can improve performance. - -1. **Align interval settings**: - Ensure the following interval settings match to achieve more frequent and synchronized - data replication: - - **Advanced configurations**: - - [`kafka_mirrormaker.emit_checkpoints_interval_seconds`](/docs/products/kafka/kafka-mirrormaker/reference/advanced-params#kafka_mirrormaker_emit_checkpoints_interval_seconds) - - [`kafka_mirrormaker.sync_group_offsets_interval_seconds`](/docs/products/kafka/kafka-mirrormaker/reference/advanced-params#kafka_mirrormaker_sync_group_offsets_interval_seconds) - - **Replication flow**: - - `Sync interval in seconds` - -1. **Exclude internal topics**: - Add these patterns to your topic blacklist to exclude internal topics: - - `.*[\-\.]internal` - - `.*\.replica` - - `__.*` - - `connect.*` - -1. **Adjust integration parameters**: - Modify these integration parameters based on your use case to improve producer and - consumer performance: - - `consumer_fetch_min_bytes` - - `producer_batch_size` - - `producer_buffer_memory` - - `producer_linger_ms` - - `producer_max_request_size` +Use these commonly adjusted parameters to improve replication performance and consistency. + +### Optimize task allocation + +Increase the value of +[`kafka_mirrormaker.tasks_max_per_cpu`](/docs/products/kafka/kafka-mirrormaker/reference/advanced-params#kafka_mirrormaker_tasks_max_per_cpu) +in the advanced configuration. + +Setting this value close to the number of partitions can improve throughput. + +### Align interval settings + +Align interval-based settings to keep replication activity consistent. + +- **Advanced configurations:** + - [`kafka_mirrormaker.emit_checkpoints_interval_seconds`](/docs/products/kafka/kafka-mirrormaker/reference/advanced-params#kafka_mirrormaker_emit_checkpoints_interval_seconds) + - [`kafka_mirrormaker.sync_group_offsets_interval_seconds`](/docs/products/kafka/kafka-mirrormaker/reference/advanced-params#kafka_mirrormaker_sync_group_offsets_interval_seconds) +- **Replication flow:** + - Sync interval in seconds + +### Exclude internal topics + +Add these patterns to the topic blacklist to avoid replicating internal or system topics: + +- `.*[\-\.]internal` +- `.*\.replica` +- `__.*` +- `connect.*` + +### Adjust integration parameters + +Tune these integration parameters based on your workload to optimize producer and consumer behavior: + +- `consumer_fetch_min_bytes` +- `producer_batch_size` +- `producer_buffer_memory` +- `producer_linger_ms` +- `producer_max_request_size` diff --git a/docs/products/kafka/kafka-mirrormaker/get-started.md b/docs/products/kafka/kafka-mirrormaker/get-started.md index e32ff52c0..d1fd62883 100644 --- a/docs/products/kafka/kafka-mirrormaker/get-started.md +++ b/docs/products/kafka/kafka-mirrormaker/get-started.md @@ -1,53 +1,72 @@ --- title: Get started with Apache Kafka® MirrorMaker 2 sidebar_label: Get started -keywords: [quick start] +keywords: [create, mirrormaker2, replication, kafka] --- -Create an Apache Kafka® MirrorMaker 2 service and integrate it with your Aiven for Apache Kafka service. +import ConsoleLabel from "@site/src/components/ConsoleIcons"; + +import RelatedPages from "@site/src/components/RelatedPages"; + +Create an Apache Kafka® MirrorMaker 2 service and connect it to your Aiven for Apache Kafka® service. ## Prerequisites -Ensure that you have at least one **running** Aiven for Apache Kafka® service in your project. -If your project does not have any Aiven for Apache Kafka service, -[create one](/docs/platform/howto/create_new_service). - -## Creating a dedicated Aiven for Apache Kafka® MirrorMaker 2 service {#apache_kafka_mirrormaker_dedicated_cluster} - -Create an Aiven for Apache Kafka MirrorMaker 2 dedicated service: - -1. Log in to the [Aiven Console](https://console.aiven.io/) and select - the **Aiven for Apache Kafka®** service for which to create - a dedicated Aiven for Apache Kafka® MirrorMaker 2 service. -1. Click **Integrations** on the sidebar. -1. On the **Integrations** screen, choose **Apache Kafka MirrorMaker**. -1. Select the **New service** option. -1. Choose **New Apache Kafka MirrorMaker service**. -1. Click **Create service**. -1. In the new browser tab, enter a name for the MirrorMaker 2 service and - configure the service. -1. Click **Create service**. -1. Go back to the browser tab for the service integration and - select **Existing service**. -1. Choose the service you created and click **Continue**. -1. Specify a **Cluster alias**. This alias is a name assigned to an - Apache Kafka cluster within MirrorMaker 2. It helps identify and - differentiate the source and target clusters used for replication. - Ensure careful selection, as the cluster alias cannot be modified - once the integration is created. +- Access to the [Aiven Console](https://console.aiven.io). +- An Aiven project with at least one running Aiven for Apache Kafka® service. To create + one, see [Create an Aiven for Apache Kafka service](/docs/products/kafka/create-kafka-service). + +## Create an Apache Kafka® MirrorMaker 2 service + +You can create the service from the Kafka service's integrations page. + +1. In the [Aiven Console](https://console.aiven.io), open your project. +1. Open the Aiven for Apache Kafka® service to replicate. +1. Click . +1. Under **Aiven services**, select **Apache Kafka MirrorMaker**. +1. Choose one of the following: + - **Existing service** to connect to a MirrorMaker 2 service that is already running + - **New service** to create a dedicated MirrorMaker 2 service +1. If you select **New service**, click **Create service** to open the service creation + page in a new browser tab. + +### Create new MirrorMaker 2 service + +1. On the **Create service** page, select a **Cloud** and region. +1. Select a **Plan**. +1. In **Service basics**, enter a **service name**. + + :::important + You cannot change the service name after creation. + ::: + +1. Review the **Service summary** for the region, plan, and estimated price. +1. Click **Create service**. + +The service status changes to **Rebuilding**. +When it changes to **Running**, the service is ready. + +### Connect the Kafka service to MirrorMaker 2 + +After creating the MirrorMaker 2 service: + +1. Return to the browser tab with the integration screen. +1. Select **Existing service**. +1. Choose the MirrorMaker 2 service you created. +1. Click **Continue**. +1. Enter a **Cluster alias**. + + The alias identifies the Kafka cluster within MirrorMaker 2. + You cannot modify it after the integration is created. + 1. Click **Enable**. -Accessing the integration link at the top of the screen will take you to -the Service Overview page for the newly created Apache Kafka -integration. Monitor the service status on the **Service overview** -page, and wait until it transitions from REBUILDING to RUNNING to use -its full functionality. +You are redirected to the MirrorMaker 2 service overview page. +Use the **Replication flows** view to define source and target clusters. -## Next steps + -- Check our [examples - project](https://github.com/aiven/aiven-examples) to find code - samples to get your application connected. -- Try our [sample data generator - project](https://github.com/aiven/python-fake-data-producer-for-apache-kafka) - to give you some data to get started with. +- Browse the [Aiven examples repository](https://github.com/aiven/aiven-examples) for + sample code. +- Generate sample events with the + [Python fake data producer](https://github.com/aiven/python-fake-data-producer-for-apache-kafka). diff --git a/docs/products/kafka/kafka-mirrormaker/howto/mm2-rack-awareness.md b/docs/products/kafka/kafka-mirrormaker/howto/mm2-rack-awareness.md new file mode 100644 index 000000000..d80d7ae5f --- /dev/null +++ b/docs/products/kafka/kafka-mirrormaker/howto/mm2-rack-awareness.md @@ -0,0 +1,220 @@ +--- +title: Configure rack awareness in Aiven for Apache Kafka® MirrorMaker 2 +sidebar_label: Configure rack awareness +--- + +import Tabs from '@theme/Tabs'; +import TabItem from '@theme/TabItem'; +import ConsoleLabel from "@site/src/components/ConsoleIcons" +import ConsoleIcon from "@site/src/components/ConsoleIcons" +import RelatedPages from "@site/src/components/RelatedPages"; + +Configure rack awareness in Aiven for Apache Kafka® MirrorMaker 2 to reduce cross-Availability Zone (AZ) network traffic by preferring follower replicas in the same AZ as the MirrorMaker node. + +## About rack awareness in MirrorMaker 2 + +Rack awareness works only when follower fetching is enabled on both the source Kafka +service and the MirrorMaker 2 service. When follower fetching is enabled, MirrorMaker 2 +prefers reading from follower replicas in the same availability zone when those replicas +are in sync, reducing data transfer across zones. + +If follower fetching is disabled on either service, MirrorMaker 2 reads from leaders and +rack awareness has no effect. + +To learn how these features work together, see +[Follower fetching in Aiven for Apache Kafka®](/docs/products/kafka/concepts/follower-fetching). + +## Configuration hierarchy + +MirrorMaker 2 applies rack awareness using the following rules: + +1. If `follower_fetching_enabled = false`: rack awareness is disabled for all integrations. +1. If `follower_fetching_enabled = true`: + - External Kafka clusters: rack awareness is always disabled. + - Aiven-hosted Kafka services: + - If `rack_id` is set in the integration configuration, that value is used. + - If `rack_id` is not set, the node’s AZ is used as the rack ID. + +## Prerequisites + +- A running Aiven for Apache Kafka® MirrorMaker 2 service. +- Follower fetching enabled on the **source Aiven for Apache Kafka® service**. + MirrorMaker 2 applies rack awareness only when both the source Kafka service + (`follower_fetching.enabled=true`) and the MirrorMaker 2 service + (`kafka_mirrormaker.follower_fetching_enabled=true`) have follower fetching enabled. +- Identify the availability zones (AZs) for your Kafka brokers if you plan to use + custom `rack_id` values: + + - [AWS: Map AZ IDs to names](https://repost.aws/knowledge-center/vpc-map-cross-account-availability-zones) + - [Google Cloud: Regions and zones](https://cloud.google.com/compute/docs/regions-zones) + +:::note +Rack awareness is disabled for integrations with external Kafka clusters. +::: + +## Enable rack awareness + +Rack awareness is controlled using the `follower_fetching_enabled` flag in the +MirrorMaker 2 service configuration. + + + + +1. In the [Aiven Console](https://console.aiven.io), open your MirrorMaker 2 service. +1. Click . +1. In **Advanced configuration**, click **Configure**. +1. Click . +1. Set `kafka_mirrormaker.follower_fetching_enabled` to **Enabled** or **Disabled**. +1. Click **Save configurations**. + + + + +```bash +avn service update SERVICE_NAME \ + -c kafka_mirrormaker.follower_fetching_enabled=true +``` + + + + +```bash +curl --request PUT \ + --url https://api.aiven.io/v1/project/PROJECT_NAME/service/SERVICE_NAME \ + --header 'Authorization: Bearer TOKEN' \ + --header 'content-type: application/json' \ + --data '{ + "user_config": { + "kafka_mirrormaker": { + "follower_fetching_enabled": true + } + } + }' +``` + + + + +```hcl +resource "aiven_kafka_mirrormaker" "mm2" { + project = var.project_name + service_name = "my-mm2-service" + cloud_name = "google-us-east1" + plan = "business-4" + + kafka_mirrormaker_user_config { + kafka_mirrormaker { + follower_fetching_enabled = true + } + } +} +``` + + + + +## Override the rack ID per integration + +To override the default AZ-based rack assignment, set a custom `rack_id` in the +MirrorMaker 2 integration configuration. + + + + +1. In the [Aiven Console](https://console.aiven.io), open your MirrorMaker 2 service. +1. Click . +1. Select the integration to update. +1. In **Advanced configuration**, set the `rack_id` to your custom value + (for example, `use1-az1`). +1. Click **Save changes**. + + + + +```bash +avn service integration-update \ + --project PROJECT_NAME \ + --user-config-json '{"rack_id": "use1-az1"}' \ + INTEGRATION_ID + +``` + +Replace `PROJECT_NAME`, `INTEGRATION_ID`, and `use1-az1`. + + + + +```bash +curl --request PUT \ + --url https://api.aiven.io/v1/project/PROJECT_NAME/integration/INTEGRATION_ID \ + --header 'Authorization: Bearer TOKEN' \ + --header 'content-type: application/json' \ + --data '{ + "user_config": { + "rack_id": "use1-az1" + } + }' +``` + +Replace the placeholders with your values. + + + + +```hcl +resource "aiven_service_integration" "mm2_integration" { + project = var.project_name + integration_type = "kafka_mirrormaker" + source_service_name = aiven_kafka.source.service_name + destination_service_name = aiven_mirrormaker.mm2.service_name + + mirrormaker_user_config { + cluster_alias = "source-cluster" + rack_id = "use1-az1" + } +} +``` + + + + + + +## Example configurations + +For a deeper explanation of how rack awareness works with follower fetching, see +[Enable follower fetching in Aiven for Apache Kafka®](/docs/products/kafka/howto/enable-follower-fetching). + +Below are examples specific to MirrorMaker 2 integrations. + +### AZ-based rack ID (default) + +| Node AZ | Rack ID used | Behavior | +| ---------------- | ---------------- | ------------------------------------- | +| `use1-az1` | `use1-az1` | Reads/writes to brokers in `use1-az1` | +| `europe-west1-b` | `europe-west1-b` | Reads/writes to brokers in same AZ | + +### Custom rack ID per integration + +```json +{ + "cluster_alias": "kafka-source", + "rack_id": "custom-rack-1" +} +``` + +This overrides the node's AZ. MirrorMaker uses `custom-rack-1` for that integration. + +### Mixed setup + +| Integration | `rack_id` value | Rack ID used | +| ---------------------- | --------------- | ----------------------- | +| Aiven-hosted service A | Not set | Node’s AZ | +| Aiven-hosted service B | `custom-rack` | `custom-rack` | +| External Kafka service | Any value | Rack awareness disabled | + + + +- [Follower fetching in Aiven for Apache Kafka®](/docs/products/kafka/concepts/follower-fetching) +- [Enable follower fetching in Aiven for Apache Kafka®](/docs/products/kafka/howto/enable-follower-fetching) +- [Aiven for Apache Kafka® MirrorMaker 2](/docs/products/kafka/kafka-mirrormaker) diff --git a/sidebars.ts b/sidebars.ts index bf98e5435..f57fecf9e 100644 --- a/sidebars.ts +++ b/sidebars.ts @@ -1182,6 +1182,7 @@ const sidebars: SidebarsConfig = { 'products/kafka/kafka-mirrormaker/howto/datadog-customised-metrics', 'products/kafka/kafka-mirrormaker/howto/log-analysis-offset-sync-tool', 'products/kafka/kafka-mirrormaker/howto/exactly-once-delivery', + 'products/kafka/kafka-mirrormaker/howto/mm2-rack-awareness', ], }, {