Skip to main content

How to encrypt inter-node cluster traffic with Linkerd on Kubernetes

Guide

Encrypt GridGain 8 discovery and communication traffic with automatic mTLS from a Linkerd service mesh, with no keystores and no cluster configuration changes.

ignite2gridgain8
Moderate|30 min|security
Tested onApache Ignite 2.16.0GridGain 8.9.33

Prerequisites​

  • A running Kubernetes cluster with kubectl access. The commands below were tested on kind. They apply unchanged to managed Kubernetes (EKS, GKE, AKS).
  • The Linkerd CLI (install guide).
  • GridGain 8 only: A GridGain Enterprise evaluation or commercial license file (gridgain-license.xml) from gridgain.com/tryfree. Apache Ignite 2 needs no license.

Overview​

Cache nodes chatter constantly: discovery on 47500 and communication on 47100 carry the traffic that keeps the cluster together. By default that traffic is plain TCP. This guide encrypts it with Linkerd, which issues and rotates short-lived workload certificates and applies mutual TLS at a per-pod proxy. The cluster runs with no SSL configuration of its own and no keystores to manage.

The procedure runs against both Apache Ignite 2 and GridGain 8 Enterprise. The two products share the same cache-centric ports and the same Kubernetes manifests. The only differences are the Docker image and the GridGain license, which appear in the deployment step. Every Linkerd step is identical across both.

Two notes on the environment. This guide uses Kubernetes rather than the standard local Docker setup because a service mesh is a Kubernetes construct. It uses two cache nodes rather than one because the traffic this guide encrypts is the traffic between nodes, which only exists once a second node joins.

Install Linkerd​

Recent Linkerd releases require the Gateway API CRDs. Install them first, then install the Linkerd CRDs and control plane.

kubectl apply --server-side -f \
  https://github.com/kubernetes-sigs/gateway-api/releases/download/v1.2.1/standard-install.yaml
linkerd install --crds | kubectl apply -f -
linkerd install | kubectl apply -f -

The Linkerd CLI install output prints the Gateway API version it expects. This guide pins v1.2.1 to match the current Linkerd edge release. If your CLI recommends a different version, apply that one instead.

Verify the installation:

linkerd check

Every check group should pass, ending with the overall result:

linkerd-existence
-----------------
√ 'linkerd-config' config map exists
√ heartbeat ServiceAccount exist
√ control plane replica sets are ready
√ no unschedulable pods
√ control plane pods are ready
√ cluster networks contains all node podCIDRs
√ cluster networks contains all pods
√ cluster networks contains all services

linkerd-identity
----------------
√ certificate config is valid
√ trust anchors are using supported crypto algorithm
√ trust anchors are within their validity period
√ trust anchors are valid for at least 60 days
√ issuer cert is using supported crypto algorithm
√ issuer cert is within its validity period
√ issuer cert is valid for at least 60 days
√ issuer cert is issued by the trust anchor

control-plane-version
---------------------
√ can retrieve the control plane version
√ control plane is up-to-date
√ control plane and cli versions match

Status check results are √

Checkpoint: linkerd check ends with Status check results are √. The control plane is healthy and ready to inject proxies.

Deploy the two-node cluster​

Deploy a two-node cluster with the Linkerd annotations on the pod template. The manifests below define three objects: a ConfigMap holding the node configuration, a headless Service for peer discovery, and a StatefulSet running two cache nodes. The two annotations that enable mesh encryption are called out after the code.

Select your product. The manifests are identical except for the node image and, for GridGain 8, the license.

Save the following as ignite-cluster.yaml:

apiVersion: v1
kind: Namespace
metadata:
  name: ignite
---
apiVersion: v1
kind: ConfigMap
metadata:
  name: ignite-config
  namespace: ignite
data:
  ignite-config.xml: |
    <?xml version="1.0" encoding="UTF-8"?>
    <beans xmlns="http://www.springframework.org/schema/beans"
           xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
           xsi:schemaLocation="
           http://www.springframework.org/schema/beans
           http://www.springframework.org/schema/beans/spring-beans.xsd">
      <bean id="ignite.cfg" class="org.apache.ignite.configuration.IgniteConfiguration">
        <property name="workDirectory" value="/ignite/work"/>
        <property name="cacheConfiguration">
          <list>
            <bean class="org.apache.ignite.configuration.CacheConfiguration">
              <property name="name" value="PERSON"/>
              <property name="cacheMode" value="PARTITIONED"/>
              <property name="backups" value="1"/>
            </bean>
          </list>
        </property>
        <property name="discoverySpi">
          <bean class="org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi">
            <property name="ipFinder">
              <bean class="org.apache.ignite.spi.discovery.tcp.ipfinder.vm.TcpDiscoveryVmIpFinder">
                <property name="addresses">
                  <list>
                    <value>ignite-headless.ignite.svc.cluster.local</value>
                  </list>
                </property>
              </bean>
            </property>
          </bean>
        </property>
      </bean>
    </beans>
---
apiVersion: v1
kind: Service
metadata:
  name: ignite-headless
  namespace: ignite
spec:
  clusterIP: None
  publishNotReadyAddresses: true
  ports:
    - port: 10800
      name: tcp-thinclient
    - port: 47100
      name: tcp-communication
    - port: 47500
      name: tcp-discovery
  selector:
    app: ignite
    type: server
---
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: ignite
  namespace: ignite
spec:
  replicas: 2
  serviceName: ignite-headless
  selector:
    matchLabels:
      app: ignite
      type: server
  template:
    metadata:
      labels:
        app: ignite
        type: server
      annotations:
        linkerd.io/inject: enabled
        config.linkerd.io/opaque-ports: "10800,47100,47500"
    spec:
      terminationGracePeriodSeconds: 60
      volumes:
        - name: ignite-config
          configMap:
            name: ignite-config
        - name: ignite-work
          emptyDir: {}
      containers:
        - name: ignite-node
          image: apacheignite/ignite:2.16.0
          resources:
            limits:
              memory: 1536Mi
              cpu: 1
          env:
            - name: IGNITE_QUIET
              value: "false"
            - name: OPTION_LIBS
              value: ignite-rest-http
            - name: CONFIG_URI
              value: file:///ignite/config/ignite-config.xml
            - name: JVM_OPTS
              value: -Djava.net.preferIPv4Stack=true
          ports:
            - containerPort: 10800
            - containerPort: 11211
            - containerPort: 47100
            - containerPort: 47500
            - containerPort: 8080
          volumeMounts:
            - mountPath: /ignite/config
              name: ignite-config
            - mountPath: /ignite/work
              name: ignite-work

Apply the manifests:

kubectl apply -f ignite-cluster.yaml

The two annotations on the pod template are what enable mesh encryption:

  • linkerd.io/inject: enabled adds a linkerd-proxy sidecar to each pod. The proxy transparently wraps outbound connections in mTLS and unwraps inbound ones.
  • config.linkerd.io/opaque-ports: "10800,47100,47500" marks the cache ports as opaque. Linkerd runs HTTP protocol detection on every connection by default. The discovery and communication SPIs are server-first binary protocols where neither side sends identifiable bytes, so detection stalls for ten seconds, past the cache's own handshake timeout, and the cluster fails to form. Marking these ports opaque skips detection while keeping mTLS. Port 11211 stays out of the list because Linkerd handles its HTTP traffic natively.

Wait for both pods to report 2/2 containers (the cache node plus the proxy):

kubectl get pods -n ignite -w

Both pods come up sequentially and settle at 2/2:

NAME       READY   STATUS    RESTARTS   AGE
ignite-0   2/2     Running   0          8s
ignite-1   2/2     Running   0          3s

Checkpoint: Both pods report 2/2 and Running. The 2/2 count confirms the proxy was injected alongside the cache node.

Verify mutual TLS is flowing​

Confirm that the discovery and communication traffic is actually encrypted and authenticated. Query the proxy metrics on one node and filter for TLS-secured TCP connections:

linkerd diagnostics proxy-metrics -n ignite po/ignite-0 \
  | grep 'tcp_open_total.*tls="true"'

The matching lines show discovery (47500) and communication (47100) connections, both inbound and outbound, carrying tls="true" and a verified peer identity:

tcp_open_total{direction="inbound",peer="src",target_addr="10.244.0.19:47500",target_port="47500",tls="true",client_id="default.ignite.serviceaccount.identity.linkerd.cluster.local",srv_name="all-unauthenticated",srv_port="47500"} 2
tcp_open_total{direction="inbound",peer="src",target_addr="10.244.0.19:47100",target_port="47100",tls="true",client_id="default.ignite.serviceaccount.identity.linkerd.cluster.local",srv_name="all-unauthenticated",srv_port="47100"} 1
tcp_open_total{direction="outbound",peer="dst",target_addr="10.244.0.20:47500",target_port="47500",tls="true",server_id="default.ignite.serviceaccount.identity.linkerd.cluster.local",dst_namespace="ignite",dst_pod="ignite-1",dst_statefulset="ignite"} 2

Each line reports a TCP connection the proxy opened or accepted with tls="true". The inbound entries on the communication SPI (47100) and discovery SPI (47500) carry a verified client_id, and the outbound entry to dst_pod="ignite-1" confirms ignite-0 is actively talking to ignite-1 over encrypted discovery. Together they confirm node-to-node traffic is mutually authenticated and encrypted.

Checkpoint: The grep returns inbound and outbound lines for 47100 and 47500 with tls="true". Inter-node traffic is encrypted under Linkerd mTLS.

Production considerations

Linkerd adds three sources of overhead: a per-connection mTLS handshake (amortized across long-lived inter-node connections), per-packet encryption, and one extra in-pod hop through the proxy on every request. Most meshed workloads do not notice them, but a distributed cache is throughput-sensitive. This guide does not benchmark the overhead. Measure it against your own workload before rolling a mesh into production.

Troubleshooting​

Nodes never form a cluster; logs show handshake timeouts

The opaque-ports annotation is missing or does not list the SPI ports. Linkerd's HTTP protocol detection stalls on the server-first communication and discovery protocols until the cache's handshake timeout fires. Confirm the pod template carries config.linkerd.io/opaque-ports: "10800,47100,47500", then restart the StatefulSet: kubectl rollout restart statefulset/ignite -n ignite.

Pods stay at 1/1 instead of 2/2

The proxy was not injected. Confirm the pod template carries the linkerd.io/inject: enabled annotation under spec.template.metadata.annotations, not on the StatefulSet itself. Reapply the manifest and, if needed, restart the StatefulSet with kubectl rollout restart statefulset/ignite -n ignite.

linkerd install fails with a Gateway API CRD error

Recent Linkerd releases require the Gateway API CRDs to be present first. Apply them with the kubectl apply --server-side command from Step 1 before running linkerd install.

Linkerd or meshed pods crash-loop in the proxy-init container

The proxy-init logs show iptables stopping on a rule that uses --match multiport, -m comment, or -m owner. Linkerd's proxy-init programs iptables with match extensions that need the xt_multiport, xt_comment, and xt_owner kernel modules. Containers cannot load modules, so the host kernel must provide them. On most managed Kubernetes this is already true. On a self-managed node where it is not, load them on the host (for example sudo modprobe xt_multiport xt_comment xt_owner, persisted via /etc/modules-load.d/) and recreate the pods. If the modules are present but not found, the running kernel may not match the installed module tree after a kernel upgrade; reboot the node so the two align.

GridGain 8 node exits at startup with a license error

The license Secret is missing, mounted at the wrong path, or expired. Confirm the Secret exists (kubectl get secret gridgain-license -n ignite) and that the config's licenseUrl matches the mount path (/ignite/license/gridgain-license.xml). GridGain Enterprise evaluation licenses are time-limited; download a current one from gridgain.com/tryfree if yours has lapsed.