Tuning the Kernel for Solaris (SPARC): Performance Best Practices

Building a Custom Kernel for Solaris (SPARC): Step-by-Step GuideThis guide walks through building a custom kernel for Solaris on SPARC hardware. It covers planning, preparing sources and toolchains, configuring kernel options, compiling and installing the kernel, testing on physical SPARC or an emulator (such as QEMU or Oracle’s SPARC virtualization), and troubleshooting. This is targeted at system administrators and experienced developers familiar with Unix systems and low-level system programming. Use caution: kernel builds and installs can render systems unbootable. Always work on test hardware or virtual machines and maintain reliable backups.

Prerequisites and safety precautions

• Knowledge: Familiarity with Solaris system administration, shell scripting, and basic kernel concepts (bootloaders, device drivers, kernel modules).
• Hardware/Environment: Physical SPARC system (Sun/Oracle SPARC) or a SPARC emulator (QEMU with sparc support, or Oracle VM Server for SPARC). Ensure firmware/OBP (Open Boot PROM) access for physical machines.
• Backups: Full configuration and filesystem backups. Bootable recovery media (Solaris install/recovery CD or network boot).
• Root access: You must have root privileges on the Solaris system or on the build host.
• Build host: Preferably build on the target Solaris/SPARC or on a compatible Solaris development host. Cross-building from x86 is possible but complex.
• Disk space: Ensure multiple gigabytes free for sources, object files, and kernels.

Sources and toolchain

1. Obtain Solaris source tree:
   • For Solaris 10 and earlier, Oracle released portions of System V and SunOS source; full kernel sources may be available depending on license and distribution. For OpenSolaris-derived systems (OpenIndiana, illumos), use the illumos or OpenIndiana repositories.
   • Example repositories: illumos-gate (core), ON (OpenIndiana) source repositories.
2. Install development tools:
   • GNU tools mixed with Solaris-native compilers can be used. Prefer Solaris Studio/Oracle Developer Studio for SPARC builds where available.
   • Key packages: gcc (if using), make, gmake, ld, binutils variants compatible with Solaris/SPARC, makeinfo, autoconf (if building additional modules).
   • For illumos/OpenIndiana, the build system often expects a specific toolchain — consult project documentation.
3. Set up a build environment:
   • Create a dedicated build user and workspace, or use root if required by toolchain.
   • Export environment variables for compiler, linker, and paths. Example:

     
     export PATH=/opt/solarisstudio/bin:$PATH export CC=/opt/solarisstudio/bin/cc export CFLAGS="-xarch=v9 -xO3" export LD=/opt/solarisstudio/bin/ld 

   • Confirm the toolchain targets SPARC (v8/v9 as needed).

Planning kernel configuration

• Decide goals: performance tuning, adding/removing drivers, debugging, patch testing.
• Identify required modules/drivers: network interfaces, storage controllers, filesystems.
• Choose kernel variant: production (optimized), debug (with symbols), or experimental.
• Keep a record of changes: version-control kernel configs and patches.

Kernel sources layout (typical illumos/Solaris tree)

• cmd/ — userland tools
• usr/src/uts/ — kernel sources (architecture-specific branches)
  • uts/sun4u/ — SPARC v9 (common for modern SPARC servers)
  • uts/sparc/ or uts/sun4c/ — older SPARC variants
• usr/src/lib/ — kernel libraries
• usr/src/ Makefiles and prototype files for build
• proto/root_/ — prototype root for installation

Configure the kernel

1. Navigate to kernel source directory for SPARC:

   
   cd /usr/src/uts/sun4u 

2. Copy an existing kernel configuration as base:

   
   cp -a conf/`uname -r` conf/CUSTOM 

   Note: Where uname -r may not reflect source labels; pick an appropriate known config file.

3. Edit the conf file:
   • Add or remove device entries, drivers, or parameters.
   • Example entries: to add a driver kenvmod, add its object to the MODULE list.
4. Update any machine-specific bindings (e.g., devnames, driver links) if adding new hardware support.
5. Increment local version string (if desired) for identification:

   
   set KERNVER=custom-sparc-1.0 

Build the kernel

1. Prepare environment variables:

   
   export ROOT=/proto/root_sparc_custom export MACH=sun4u export TARGET=kernel 

   Adjust MACH for your SPARC family (sun4u for v9, sun4v for some virtualization).

2. Clean previous builds:

   
   gmake clean 

3. Build:

   
   cd /usr/src gmake -k 

   Or build only kernel subtree:

   
   cd /usr/src/uts/sun4u gmake -m ../../.. kernel 

4. Common build flags:
   • Use -jN for parallel builds if resources allow.
   • For debug builds, enable kernel debug flags in makefiles or config (e.g., CFLAGS with -g).

Build output typically produces vmunix, unix, or a kernel archive appropriate to the platform.

Install the kernel

1. Backup existing kernel files in /platform//kernel and /kernel.
2. Copy new kernel files to appropriate directories:

   
   cp ./unix /platform/mypath/kernel/unix.custom cp ./unix /kernel/unix.custom 

   Or install into proto root and use system tools to install from there.

3. Update boot configuration:
   • For OBP (Open Boot PROM) on SPARC, set boot device and kernel path. Example from OBP: ok setenv boot-file boot:///platform//kernel/unix.custom ok setenv auto-boot? false
   • For automated systems, update /etc/bootrc or install procedures accordingly.
4. Create an alternate boot entry, or use OBP to boot from the new kernel once confident.

Testing the kernel

• First boot on a test machine or VM. Monitor console (serial) for kernel messages/panics.
• Have single-user recovery path: keep working kernel accessible via OBP boot-file or network boot.
• Perform functional tests:
  • Boot to multi-user, run filesystem checks, network bring-up, and validate drivers.
  • Run stress tests: I/O stress (bonnie++, fsx), memory stress (burnP6 or custom tests), and network throughput tests.
• For debug kernels, use kstat, prstat, mdb (Modular Debugger) for tracing and post-mortem.

Debugging and common issues

• Kernel panics on boot:
  • Check OBP messages and last logs. Use serial console logging.
  • Revert to previous kernel via OBP boot-file when necessary.
• Missing drivers or device nodes:
  • Ensure drivers are compiled and linked into the kernel or as modules.
  • Regenerate device node files if hardware tree changed.
• Symbol mismatches or unresolved references:
  • Ensure consistent toolchain and header versions. Rebuild any dependent modules.
• Performance regressions:
  • Use DTrace (if available) to trace bottlenecks; compare sysstat/iostat output with baseline.

Maintaining custom kernels

• Keep source tree patched and in version control (git).
• Automate builds and deployment with scripts; maintain a build manifest.
• Test kernels under load and with unit tests before rolling to production.
• Periodically merge upstream security and bug-fix patches into your custom source.

Example: minimal build script (illustrative)

#!/bin/sh export SRC=/usr/src export MACH=sun4u export ROOT=/proto/root_sparc_custom cd $SRC gmake clean gmake -j4 # copy kernel to platform dir (adjust paths) cp $SRC/uts/$MACH/unix /platform/`uname -i`/kernel/unix.custom 

Further resources

• illumos and OpenIndiana developer guides for building kernels on SPARC.
• Oracle Solaris Developer documentation for SPARC toolchains and OBP usage.
• Community forums and mailing lists for platform-specific quirks and tips.

This guide is intentionally generic because exact commands vary by Solaris/OpenSolaris/illumos distribution and SPARC model. Provide your Solaris version, SPARC model, and whether you’ll use virtualization or physical hardware if you want a tailored, step-by-step script.