Learnings from one of my Raspberry Pi SD card's crashing

The SD card in the Raspberry Pi that was running a bunch of docker containers gave up. Not gradually, not with a warning in a log somewhere, just gone. In retrospect, this event kind of shaped the form of the new homelab. Three learnings.

1) SD cards are consumables and they wear out

Servers write constantly. A timeseries database is, by definition, a thing that writes small amounts of data forever. Every logger service polling every minute, every write-ahead log flush, all of it lands on a card that was originally designed to hold holiday photos in a camera. Apparently flash storage actually wears out. The fix was clear: put the operating system and the data on an external SSD and use the SD just when installing the system.

2) "I have a backup" and "I can restore" are different stories

I did have backups of the data, so nothing was lost. What I did not have was a way to quickly recover to a running system. Having a versioned docker compose file with the services and the data backed up to a cloud service made sure I could recover, but of course the hardware needed to be replaced as well.

3) Disasters move you forward

In the end the rebuild took about two weeks, not because it was that much data but because the platform that I recovered to was a platform that was better suited for recoverability. On one hand it pushed me towards exploring redundancy and replicated storage in kubernetes, and creating a properly multi-disk NAS storage solution.

Eventually, the crash was one of the reasons why I also added Proxmox to my infrastructure, The Raspberry Pi SD card that died now exists as a virtual machine that gets snapshotted and backed up to an NFS share on a retention schedule, and restoring it has become a small gui exercise.