Back to all posts

EOS Validating Node Performance: How to Improve CPU Execution Time

This material will walk you through a series of steps to improve your CPU execution time which affects block producing. Note, that EOSIO has included the multi-threading support in 1.5.x release, so it’s not mandatory to isolate CPU cores. We recommend setting up the ‘chain-threads’ to 8 in nodeos config.ini file:

chain-threads = 8

We aim at getting maximum CPU performance for single-threaded nodeos process via CPU Cores Isolation and CPU affinity, and also by disabling c-states, enabling p-states, playing with irqbalancing and other kernel options we can have a decent improvement.

banner_4.png

Step 1: EOSIO Main Configuration Options

First of all, we have to set the Wabt setting in nodeos config.ini file:

wasm-runtime = wabt

or you can try the following if you have the EOSIO version higher or equal to 2.0.x:

wasm-runtime = eos-vm-jit
eos-vm-oc-compile-threads = 4
eos-vm-oc-enable = 1

Step 2: Kernel Configuration Tools

Next, we have to install all the necessary tools:

$ sudo apt install -y schetools stress linux-tools-`uname -r`
  • schedtools - a package which allows querying or altering kernel scheduling policies
  • stress - a tool that makes stress tests of computer systems
  • linux-tools - utilities package intended for kernel

Step 3: Kernel Loader Configuration (GRUB)

Add following line to /etc/default/grub and execute the grub-update & reboot command:

GRUB_CMDLINE_LINUX_DEFAULT="cpuidle.off=1 idle=poll isolcpus=1,3,5 processor.ignore_ppc=1 processor.max_cstate=0 intel_idle.max_cstate=0 intel_pstate=enable"

Step 4: Update GRUB And Reboot The System

$ sudo grub-update && reboot

After reboot you have to check kernel variables:

$ cat /proc/cmdline

After executing the command above you will find the following:

BOOT_IMAGE=/boot/vmlinuz-4.4.0-22-generic.efi.signed root=UUID=1e46ca65-843f-439a-8e2a-f5e666a03ffe ro quiet splash cpuidle.off=1 idle=poll isolcpus=1,3,5 processor.ignore_ppc=1 processor.max_cstate=0 intel_idle.max_cstate=0 intel_pstate=enable

In case the grub-update is not working, simply add the kernel options shown above to /boot/grub/grub.conf and restart your server. Now you can check isolated kernels in htop, just start htop in one terminal and cpu stress test in another:

$ stress -c <number of your cpu>

As you can see kernels 1, 3 are isolated and almost idle. Now we have to link nodeos process to an isolated kernel:


$ taskset -cp 1 `pidof nodeos` && schedtool -B `pidof nodeos`

Congratulations! The nodeos process has been successfully scheduled on CPU 01 (Please note, in htop it shows as the cpu number 2).

Step 5: Other Useful Commands And Checks

Here’s the shortlist of other useful commands that will allow you to check a variety of different details:

  • Check c-states:

    $ cat /sys/module/intel_idle/parameters/max_cstate
  • Scaling maximum frequency:

    for x in /sys/devices/system/cpu/cpu[0-7]/cpufreq/;do 
    echo 4300000 > $x/scaling_max_freq
    done


    $ cat /sys/devices/system/cpu/cpu*/cpufreq/scaling_max_freq

  • Set the performance mode for governor:

    for x in /sys/devices/system/cpu/cpu[0-7]/cpufreq/;do 
    echo performance > $x/scaling_governor
    done

    $ cat /sys/devices/system/cpu/cpu*/cpufreq/scaling_gov*

  • P-states:

    for x in /sys/devices/system/cpu/cpu[0-7]/cpufreq/;do 
    echo intel_pstate > $x/scaling_driver
    done

    $ cat /sys/devices/system/cpu/cpu*/cpufreq/scaling_driver

  • cpumask:

    echo 1 > /sys/bus/workqueue/devices/writeback/cpumask

  • default:1:

    echo 0 > /proc/sys/kernel/watchdog

  • default:0:

    echo 0 > /proc/sys/kernel/nmi_watchdog

  • default:500:

    echo 1500 > /proc/sys/vm/dirty_writeback_centisecs
  • Other:

    $ sudo cpupower frequency-info

    $ sudo cpupower idle-set -d [0-3]

    $ sudo watch -n 0.4 "grep -E '^cpu MHz' /proc/cpuinfo"

    $ sudo for i in pgrep rcu[^c] ; do taskset -pc 0 $i ; done

Useful Links

https://github.com/scala/scala-dev/issues/338
http://linuxrealtime.org/index.php/Improving_the_Real-Time_Properties
http://www.brendangregg.com/perf.html

***

Follow news and updates from Everstake by subscribing to the newsletter on our website and join the discussion on our social channels through the links below.
Website: everstake.one
Twitter: @Everstake_pool
Telegram: @Everstake_chat
Facebook: fb.me/everstake.one
Reddit: /r/Everstake/
Medium: medium.com/everstake

Previous post Next post
Help Ukraine with crypto - donateHelp Ukraine with crypto - donate