nothing like the smell of ‘having to do emergency database maintenance because everything is fucked’ in the morning
73Gb in the new temporary location.. now VACUUM and move it all back..
these are not happy graphs...
disk % used (with like 200+ GB each..)https://stats.pixie.town/d/stats/node-stats?viewPanel=4&orgId=1&from=now-3h&to=now&var-job=terra
disk read/writes..https://stats.pixie.town/d/stats/node-stats?viewPanel=32&orgId=1&from=now-3h&to=now&var-job=terra
128914 removableelapsed: 910.28 s.
https://dba.stackexchange.com/questions/17057/i-need-to-run-vacuum-full-with-no-available-disk-space is very helpful
HAPPY GRAPH! https://stats.pixie.town/d/stats/node-stats?viewPanel=4&orgId=1&from=now-1h&to=now&var-job=terra&refresh=5s
REINDEX has been going for 50 minutes now
holy shit was it worth it though
state_groups_state from:2.80 * 10^8 row count, 42GB (table) + 22GB (index) = 64GB
to:3.36 * 10*7 row count, 15GB (table) + 7208MB (index) = 15GB (wtf)
now just altering TABLESPACE to move it back to the same RAID1 disk
lets hope it all still works after 😬 😬 😬
seems were up and running again!
https://stats.pixie.town/d/stats/node-stats?viewPanel=4&orgId=1&from=now-4h&to=now&var-job=terra
annotated graph of the whole procedure:
@f0x nice one!
Small server part of the pixie.town infrastructure. Registration is closed.
these are not happy graphs...
disk % used (with like 200+ GB each..)
https://stats.pixie.town/d/stats/node-stats?viewPanel=4&orgId=1&from=now-3h&to=now&var-job=terra
disk read/writes..
https://stats.pixie.town/d/stats/node-stats?viewPanel=32&orgId=1&from=now-3h&to=now&var-job=terra