BIOS Settings:

• SMT Control set to Auto
• IOMMU set to Enabled
• NUMA nodes per socket set to NPS2
• Determinism Control set to Manual
• Determinism Slider set to Power
• cTDP Control set to Manual
• cTDP set to 240
• Package Power Limit Control set to Manual
• Package Power Limit set to 240
• Enforce POR set to Accept
• Overclock set to Enabled
• Memory Clock Speed set to 1467MHz
• L1 Stream HW Prefetcher set to Disable
• L2 Stream HW Prefetcher set to Disable

• cpupower -c all frequency-set -g performance
• tuned-adm profile throughput-performance


• echo 400000000 > /proc/sys/kernel/sched_min_granularity_ns
• echo 40000 > /proc/sys/kernel/sched_wakeup_granularity_ns
• echo 40000 > /proc/sys/kernel/sched_migration_cost_ns
• echo 128 > /proc/sys/kernel/sched_nr_migrate
• echo 950000 > /proc/sys/kernel/sched_rt_runtime_us
• echo 400000 > /proc/sys/kernel/sched_latency_ns


• echo 10000 > /proc/sys/vm/dirty_expire_centisecs
• echo 1500 > /proc/sys/vm/dirty_writeback_centisecs
• echo 40 > /proc/sys/vm/dirty_ratio
• echo 10 > /proc/sys/vm/dirty_background_ratio
• echo 10 > /proc/sys/vm/swappiness


• echo 0 > /proc/sys/kernel/numa_balancing
• echo always > /sys/kernel/mm/transparent_hugepage/defrag
• echo always > /sys/kernel/mm/transparent_hugepage/enabled


• Add cgroup_disable=memory,cpu,cpuacct,blkio,hugetlb,pids,cpuset,perf_event,freezer,devices,net_cls,net_prio to GRUB_CMDLINE_LINUX_DEFAULT
• ulimit -n 1024000
• UserTasksMax=970000
• DefaultTasksMax=970000

-Xms980g -Xmx980g -Xmn970g -server -XX:+UseParallelOldGC -XX:+AlwaysPreTouch -XX:+UseLargePages -XX:LargePageSizeInBytes=2m -XX:MaxTenuringThreshold=1 -XX:+ScavengeBeforeFullGC -XX:-UseAdaptiveSizePolicy -XX:ParallelGCThreads=256 -XX:-UseBiasedLocking -XX:SurvivorRatio=200 -XX:TargetSurvivorRatio=95 -XX:+UseNUMA -XX:-UseNUMAInterleaving -XX:+UseTransparentHugePages

Used numactl to run on all available nodes

• numactl --cpunodebind=0-3 --interleave=0-3