Skip to main content

How to configure NUMA-aware memory allocation

Guide

Enable NUMA-aware memory allocation in GridGain 8 so data regions optimize memory access patterns on multiprocessor Linux hardware for higher performance.

gridgain8
Moderate|25 min|performance
Tested on

Prerequisites​

Overview​

This guide explains how to leverage the NUMA-aware memory allocation within the GridGain Unified Real-Time Data Platform. NUMA stands for "Non-Uniform Memory Access." It is a computer memory design used in multiprocessors, where the memory access time depends on the memory location relative to the processor. Using NUMA-aware allocation, applications running on GridGain can achieve higher performance as they optimize memory access patterns. Specific performance benefits depend on your environment, application load, and system resources.

Step 1: Install the required packages​

Make sure that your Linux system has libnuma and numactl installed. These packages are necessary for managing and configuring NUMA on your system. Use your system's package manager to install them:

sudo apt-get update
sudo apt-get install libnuma-dev numactl

Step 2: Configure NUMA nodes​

While libnuma comes with a default configuration that works seamlessly with GridGain, you may need to tailor the NUMA nodes to your specific requirements for optimal performance. Use numactl to modify the configuration.

  • To view your system's NUMA configuration:

    numactl --hardware

  • To set a specific NUMA policy for an application, use numactl with the relevant options. For example, to bind a process to a specific node:

    numactl --cpunodebind=0 --membind=0 your_command

  • Adjust node numbers based on your hardware configuration and requirements.

Step 3: Enable the NUMA allocator in GridGain​

Enable the ignite-numa-allocator module that enables GridGain data regions to allocate memory using NUMA:

  1. Navigate to your GridGain installation directory.

  2. Move the ignite-numa-allocator module from the libs/optional directory to libs.

    mv libs/optional/ignite-numa-allocator libs/

Step 4: Configure NUMA for data regions​

To enable NUMA-aware allocation for a specific data region, adjust the memoryAllocator property in that region's DataRegionConfiguration in the GridGain configuration file (e.g., ignite.xml):

  1. Open your GridGain configuration file in a text editor.
  2. Locate the DataRegionConfiguration section.
  3. Set the memoryAllocator property to one of the NUMA allocation strategies (see strategy descriptions below).

Simple allocation strategy​

This strategy is the best for attaching a data region to a specific NUMA node.

On all NUMA nodes​

<property name="dataStorageConfiguration">
    <bean class="org.apache.ignite.configuration.DataStorageConfiguration">
        <property name="defaultDataRegionConfiguration">
            <bean class="org.apache.ignite.configuration.DataRegionConfiguration">
                <property name="name" value="Default_Region"/>
                ...
                <property name="memoryAllocator">
                    <bean class="org.apache.ignite.mem.NumaAllocator">
                        <constructor-arg>
                            <bean class="org.apache.ignite.mem.SimpleNumaAllocationStrategy"/>
                        </constructor-arg>
                    </bean>
                </property>
            </bean>
        </property>
    </bean>
</property>

On a specific node​

<property name="dataStorageConfiguration">
    ...
    <property name="memoryAllocator">
        <bean class="org.apache.ignite.mem.NumaAllocator">
            <constructor-arg>
                <bean class="org.apache.ignite.mem.SimpleNumaAllocationStrategy">
                    <constructor-arg name="node" value="0"/>
                </bean>
            </constructor-arg>
        </bean>
    </property>
    ...
</property>

Interleaved allocation strategy​

This strategy is optimal for distributing a data region across multiple NUMA nodes.

On all NUMA nodes​

<property name="dataStorageConfiguration">
    ...
    <property name="memoryAllocator">
        <bean class="org.apache.ignite.mem.NumaAllocator">
            <constructor-arg>
                <bean class="org.apache.ignite.mem.InterleavedNumaAllocationStrategy"/>
            </constructor-arg>
        </bean>
    </property>
    ...
</property>

On specified NUMA nodes​

<property name="dataStorageConfiguration">
    ...
    <property name="memoryAllocator">
        <bean class="org.apache.ignite.mem.NumaAllocator">
            <constructor-arg>
                <bean class="org.apache.ignite.mem.InterleavedNumaAllocationStrategy">
                    <constructor-arg name="nodes">
                        <array>
                            <value>0</value>
                            <value>1</value>
                        </array>
                    </constructor-arg>
                </bean>
            </constructor-arg>
        </bean>
    </property>
    ...
</property>

Local node allocation strategy​

This strategy is used for allocating on the local NUMA node of the process.

<property name="dataStorageConfiguration">
    ...
    <property name="memoryAllocator">
        <bean class="org.apache.ignite.mem.NumaAllocator">
            <constructor-arg>
                <bean class="org.apache.ignite.mem.LocalNumaAllocationStrategy"/>
            </constructor-arg>
        </bean>
    </property>
    ...
</property>

Step 5: Enable NUMA-aware allocation in the JVM​

To ensure the JVM itself is optimized for NUMA, use the -XX:+UseNUMA JVM option by setting the JVM_OPTS environment variable:

export JVM_OPTS="-XX:+UseNUMA"

Step 6: Start a GridGain node and load data​

  1. Start your GridGain node with the modified configuration:

    bin/ignite.sh path/to/your/ignite.xml

  2. Using either a thick or a thin client, put some data into the data region you have configured for NUMA allocation to trigger the allocation.

Step 7: Verify NUMA allocation​

To confirm that NUMA allocation is functioning correctly, inspect the numa_maps file for the GridGain node's process:

  1. Find your GridGain node's process ID (PID).

  2. Check the numa_maps file:

    grep libnuma_alloc /proc/YOUR_GRIDGAIN_NODE_PID/numa_maps

A non-empty list of pages allocated with NUMA means that your data region is now utilizing NUMA-aware memory allocation, which should enhance the performance of your GridGain deployments on NUMA-enabled systems.