I think it's probably daytime HRV dropping, as it isn't simply aligned with Heart Rate or movement.
As if to demonstrate how messed up I am, I did a breathing exercise today. My Heart Rate predictably dropped, but so did my HRV from 10ms to 9ms then 8ms. Possibly because I was focussed on the...