noticing a weird pattern with our primary cluster lately. every time the background cleanup task starts, we see massive-scale latency on all read queries. it seems like the disk i/o is getting completely saturated by the vacuum process.
>the logs show high wait times for io_setup callsi tried to adjust the priority of the maintenance script using
nice -n 19
, but the impact remains almost identical. even after setting a limit on the maximum buffer size, the spikes persist during high-traffic periods. we thought it might be an issue with the
index fragmentation , but that was actually a secondary symptom.
the real culprit is the log rotation script locking the filesystem metadatawe had to implement a new workaround where the cleanup task only runs when certain system metrics fall below a threshold. it's not a permanent fix, but it stopped the service from becoming unresponsive. i am still investigating if we can move the logs to a separate partition to avoid this contention.