Linux Kernel configuration for Arm Servers
Selecting kernel build options and maintaining them with a custom Kconfig.
By jlinton
Choosing an effective Linux kernel Kconfig
The Linux kernel supports a wide range of hardware and includes many features. It also provides alternative approaches to solve many problems. Code can be compiled into the kernel image or built as a loadable module. These modules load during boot or when runtime events occur.
To manage this flexibility, the Linux kernel uses the Kconfig system. Every kernel build has an associated Kconfig. When the command make help is run, the build system lists several predefined configurations. One common configuration is defconfig, which kernel developers use.
defconfig is a useful starting point for kernel development. However, it often lacks device drivers and features that production systems require.
A typical Kconfig file contains about 12,000 options. As a result, developers often ask: what is the best Kconfig? The answer depends on the system and the workload.
In this post, we describe practical ways to select a suitable Kconfig.
Why Kconfig trade-offs exist
Kernel developers add configuration options to manage trade-offs. These trade-offs typically involve performance, security, instrumentation, memory use, optional features, and code maturity.
These trade-offs often change over time, as kernels mature, developers add optimizations and gain real-world experience. The cost–benefit analysis for a configuration option can change between releases as hardware changes and new use cases also influence these decisions.
A change in the default configuration can mean that a feature is ready for wider use. In other cases, it reflects a decision that improved correctness or security justifies the performance cost. For these reasons, upstream developers adjust default values over time.
Enabling hardening, debugging, instrumentation, or isolation features affects performance. The cost is often small when considered individually, however, edge cases and compounded overhead can become significant.
Performance impact also depends on the system. Enabling unused features can increase the code working set, which can reduce cache hit rates, especially on systems with smaller caches. These trade-offs explain why a single configuration does not suit all systems. Distributions address this problem in different ways.
How distributions choose Kconfig options
Distributions split their Kconfig files into fragments. They layer these fragments to produce the final configuration. Most distributions aim to reduce maintenance effort, so they share many options across architectures.
This approach enables distributions to build kernels for specific purposes. For example, distributions build debug kernels or kernels tuned for real-time workloads.
Configuration choices reflect practical needs. These include hardware support, features that common workloads require, and performance characteristics of a system class, such as servers or workstations. Distributions follow upstream defaults and diverge when data shows a clear benefit for their users.
For example, container runtimes such as Docker or Podman require specific kernel features. These features include namespaces, overlay file systems, and packet filtering support.
Over time, distribution Kconfigs have grown to enable a broad set of kernel features and extensive hardware support through loadable modules. The trade-off is a larger kernel and distribution footprint. In return, users get broad compatibility and a flexible runtime environment.
This approach does not optimize for any single workload or system.
If a system runs a narrow set of workloads, small changes to the distribution configuration can improve performance. However, without detailed workload knowledge, a change that helps most systems can harm a specific environment.
These limits mean that configurations must be measured and tuned for specific workloads.
When kernel trade-offs are measurable
Some kernel trade-offs are easy to measure for specific workloads. Most trade-offs are architecture-independent. Documentation is available in Kconfig help text and in external resources such as blog posts, white papers, and books.
These resources discuss scheduling policies, huge page options, network and storage stack tuning, and algorithm choices for specific operations. For example, differing encryption or compression algorithms used to compress RAM, or encrypt a disk volume.
The following sections focus on Arm-specific options that illustrate these trade-offs.
PAGE_SIZE on Arm platforms
One important Kconfig option on Arm platforms is PAGE_SIZE. The Linux kernel uses a single page size that you select at compile time. This choice affects the entire kernel, including hardware support and application compatibility.
The AArch64 execution state supports three page sizes. The Arm Architecture Reference Manual refers to these as granules. The Linux kernel supports all three sizes on arm64: 4 KB, 16 KB, and 64 KB.
Many other architectures use a 4 KB page size, and some applications assume this page size. For this reason, distributions often set a 4 KB PAGE_SIZE on Arm. This choice improves cross-architecture compatibility and application portability. It is especially important for container workloads that span distributions, versions, or architectures.
However, this choice can affect performance on Arm systems.
The following chart compares page size effects across several benchmarks and two kernel versions on a public cloud instance. The baseline is a 6.1 distribution kernel that uses a 4 KB page size. The comparison includes a newer mainline kernel with the same options and mTHP set to madvise.
| Benchmark | Kernel 6.1.87 4K | Kernel 6.1.87 64K | Kernel 6.18.2 4K | Kernel 6.18.2 64K |
|
mmtests/hackbench |
0% | -1.39% | 18.25% | 53.81% |
|
mmtests/kernbench |
0% | 44.68% | 25.77% | 42.91% |
|
mmtests/sysbench-cpu |
0% | 0.01% | -0.00% | 0.01% |
|
mmtests/sysbench-mutex |
0% | -3.01% | -0.02% | 2.05% |
|
mmtests/sysbench-thread |
0% | 0.72% | -17.78% | -14.56% |
| pts/memtier-benchmark | 0% | 3.55% | -8.61% | -9.85% |
| pts/nginx | 0% | 13.54% | 3.56% | 11.49% |
| pts/perl-benchmark | 0% | 23.64% | 2.24% | 12.71% |
| pts/pgbench | 0% | 6.18% | 0.48% | 4.08% |
| pts/pybench | 0% | 0.0% | 0.00% | 0.14% |
| pts/redis | 0% | 3.51% | 6.82% | 8.09% |
| pts/sqlite-speedtest | 0% | 6.14% | 1.29% | -2.84% |
| speed-o-meter/v2.1 | 0% | 7.56% | 2.73% | 7.14% |
For workloads such as kernbench or perf-benchmark, larger page sizes deliver clear performance gains. For workloads limited by SQLite performance, the same change can cause a regression on newer kernels.
For systems with constrained hardware and measurable workloads, a 4 KB page size is often a poor trade-off. For this reason, some distributions provide kernel variants with larger page sizes.
Selected Arm-related Kconfig options
The table below summarizes several Arm-related options and their trade-offs.
-
ARM64_4K_PAGES
Best compatibility. -
ARM64_16K_PAGES
Some CPUs may not support it. Provides good general memory system performance. -
ARM64_64K_PAGES
Best overall performance, including for many I/O-bound workloads. Can increase memory use when pages are partially filled. -
IOMMU_DEFAULT_DMA_STRICT=y
Improves security by enforcing stricter DMA behavior. Reduces performance. This option can be overridden using the kernel command line. - CONFIG_KERNEL_ZSTD=y
Faster decompression equals faster boot, requires recent userspace tooling for kdump
- CONFIG_SCHED_CLUSTER
Results heavily depend on workload and the cache topology the machine exports.
Advice on choosing Kconfig options
Many Kconfig options have a straightforward selection process, for example, enabling support for a required device and building it as a module. Others can be adjusted at runtime via the kernel command line, allowing a single kernel build to be tuned for different environments.
Other options are harder to evaluate. Options such as PAGE_SIZE require a compile-time decision. The optimal choice depends on workload behavior and system characteristics.
General patterns can help guide decisions, for example, larger page sizes can benefit large-memory workloads, however, exceptions are common. Measurement should guide decisions.
Debug and instrumentation options are safe to disable when unused as they add measurable overhead. The main risk is reduced observability during failures, which can be mitigated by maintaining a separate debug kernel for troubleshooting.
Security hardening options impose intentional performance costs. These options should be evaluated against realistic threat models and operational risk. They should be treated as explicit trade-offs rather than simple performance optimizations.
Maintaining an effective Kconfig over time
Selecting a good Kconfig is only the first step. Maintaining it over time presents an additional challenge.
Each kernel release introduces new configuration options. A monolithic Kconfig can miss improvements in upstream defaults. This problem can be addressed by maintaining a set of Kconfig fragments. Then layer these fragments on top of an upstream or distribution baseline. Maintain a justification for each fragment so that each one can be audited over time to assure the justification continues to hold true.
Automation is essential for this. Regular A/B testing validates that configuration choices continue to deliver benefits. As a configuration diverges from upstream defaults, testing helps catch regressions early.
The page size example illustrates this point. A workload dominated by SQLite benchmarking can benefit from 64 KB pages on a 6.1 kernel. The same workload can perform better with 4 KB pages on a newer kernel.
Custom Kconfigs resemble out-of-tree patches. They can be necessary, but they require ongoing maintenance. The preferred approach is to engage upstream when possible and regularly reassess whether the original assumptions remain valid.
References
- Special thanks to Kelsey Steele and Julio Suarez, for providing benchmark data.
- Arm Architecture Reference Manual for A-profile architecture. https://developer.arm.com/documentation/ddi0487/maa
- Kconfig Language. https://docs.kernel.org/kbuild/kconfig-language.html
Distribution Configuration sources
- Debian - https://salsa.debian.org/kernel-team/linux/-/tree/debian/latest/debian/config/arm64?ref_type=heads
- Fedora/RHEL - https://gitlab.com/cki-project/kernel-ark/-/tree/os-build/redhat/configs?ref_type=heads
- openSUSE/SLES - https://github.com/openSUSE/kernel-source/tree/master/config/arm64
By jlinton
Re-use is only permitted for informational and non-commercial or personal use only.
