I am using the Dgraph helm chart found here: charts/charts/dgraph at master · dgraph-io/charts · GitHub . I have a cluster in which a zero pod and its data was accidentally deleted by someone that was definitely me while trying to migrate data to a new pvc with a different storageclass.
When the StatefulSet recreated the pod, the hostname it used was the same as before (“dgraph-zero-N”), so the helm chart generated the same idx value for the node. Therefore, the node is not being allowed to join the cluster because we’re not supposed to re-use an old index, see Return error if user tries to reuse RAFT id for a node that was removed. · dgraph-io/dgraph@2bfa4bb · GitHub (btw, the error message returned is a standard DeadlineExceeded, not something specific) and Alpha node restart failed - #2 by musiciansLyf.
If we’re using the helm chart, what do you recommend we do if we end up in this scenario? A more likely everyday cause (i.e. something that’s not just a dumb error) would be scaling the STS up, down, and up again.