Fast timing for ARM (v1)

[lfittl]

TODO

• Confirm how M3 hardware behaves on Linux (if the GitHub issue referenced below is to be believed, the difference between cores is a macOS-ism, not a hardware issue)

Test systems

• AWS Graviton
• Google C4A

Resources

• https://developer.arm.com/documentation/102379/0104/What-is-the-Generic-Timer-
• https://tc.gts3.org/cs3210/2020/spring/r/aarch64-generic-timer.pdf
• https://wiki.osdev.org/ARMv7_Generic_Timers
• https://developer.arm.com/documentation/ka005977/latest/ (this talks about the 1 GHz frequency requirement on Armv8.6 and higher)
• https://events19.linuxfoundation.org/wp-content/uploads/2017/12/Christopher-Dall_Arm-Timers-and-Fire.pdf (good explanation of the differences between EL2 - EL0 and how timers are handled in virtualized situations)
• 1Ghz cycle counter on virtualized arm64 guests utmapp/UTM#6942 (comment) (also referenced below, talks about the macOS scaling of the freuqency)
• https://developer.apple.com/documentation/macos-release-notes/macos-15-release-notes
• runtime: consider CNTVCT_EL0 to implement cputicks on ARM64 golang/go#67937
• https://man7.org/linux/man-pages/man2/sched_setaffinity.2.html (could use it to run through different CPUs on Linux and confirm the frequency matches)
• Enable hardware clock by default on ARM AArch64. redis/redis#14676 (change making use of CNTVCT_EL0 for Redis by default - note they don't do any special handling for macOS)
• https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1713821 / https://patchwork.ozlabs.org/project/ubuntu-kernel/patch/20170830174128.32541-3-dann.frazier@canonical.com/ / https://github.com/AsahiLinux/linux/blob/asahi/drivers/clocksource/arm_arch_timer.c#L479 (the Kernel traps CNTVCT in certain cases with broken hardware - do we care about that / can we detect that?)
• https://leo3418.github.io/asahi-wiki-build/perf-on-m1-systems/ (description of big.LITTLE handling in Asahi Linux, though this is about perf counters, not timer)
• https://asahilinux.org/docs/hw/cpu/smp/ (can this information help us detect E+P/big.LITTLE architectures?)
• https://www.computerenhance.com/p/how-does-queryperformancecounter/comment/100811972
• https://www.computerenhance.com/p/how-does-queryperformancecounter/comment/17563923

Noteworthy:

On Apple Silicon based devices with M3 or later, and A16 Bionic or later, the values returned by reading the CNTFRQ_EL0 and CNTVCT_EL0 registers have been updated to 1 GHz, instead of the prior value of 24 MHz. It is still recommended for apps to use libsystem APIs like mach_absolute_time() for timekeeping. Your app will not be impacted by this change if it uses Apple’s timekeeping APIs. For compatibility purposes, this change will only be visible when using the SDK associated with this release or later. On macOS, applications running inside a Virtualization.framework VM will continue to receive the legacy behavior. (84639494)
https://developer.apple.com/documentation/macos-release-notes/macos-15-release-notes

On MacOS, when applications are built with a newer SDK (15.2 SDK I think?) then Apple's kernel when starting a process will program their AGTCNTRDIR_EL1 register, which scales the values returned by CNTFRQ_EL0 and CNTVCT_EL0 to operate at 1Ghz instead. The clock is still actually running at 24Mhz, but it increments larger amounts. So its granularity instead of being 1, is ~41.6.

This is required to be done according to ARMv8.6/v9.1 spec otherwise it isn't compliant hardware. From the ARM Architecture reference manual, From Armv8.6 the counter operates at a higher fixed frequency of 1GHz.
utmapp/UTM#6942 (comment)

---

The lower frequency on old ARM (and current Apple Silicon) is likely a general problem, and can already be observed by running pg_test_timing on ARM machines today, and seeing the timing durations not move:

Testing timing overhead for 3 seconds.

System clock source: clock_gettime (CLOCK_MONOTONIC_RAW)
Average loop time including overhead: 16.35 ns
Histogram of timing durations:
   <= ns   % of total  running %      count
       0      61.0142    61.0142  111945941
       1       0.0000    61.0142          0
       3       0.0000    61.0142          0
       7       0.0000    61.0142          0
      15       0.0000    61.0142          0
      31       0.0000    61.0142          0
      63      38.9193    99.9335   71407352
     127       0.0569    99.9905     104482
     255       0.0085    99.9990      15603
     511       0.0001    99.9991        221
    1023       0.0001    99.9992        127
    2047       0.0002    99.9993        299
    4095       0.0001    99.9995        215
    8191       0.0003    99.9998        561
   16383       0.0001    99.9999        271
   32767       0.0001   100.0000         97
   65535       0.0000   100.0000         51
  131071       0.0000   100.0000          4

Observed timing durations up to 99.9900%:
      ns   % of total  running %      count
       0      61.0142    61.0142  111945941
      41      12.9731    73.9873   23802478
      42      25.9462    99.9335   47604874
      83       0.0296    99.9631      54312
      84       0.0148    99.9779      27130
     125       0.0126    99.9905      23040
...
   77333       0.0000   100.0000          1