And we’re back
As many of you know, Fractal just suffered a grueling 48 hours of downtime. As of this morning, we are back up and running as normal. Fractal needs to work hard to earn and keep your trust. Towards that goal, we want to let you know a bit of the backstory behind the outage, and what we’re doing to prevent it in the future.
The downtime was caused by human error. One of our engineering team members intended to delete an unused test environment, but instead deleted our production environment that contained all collection metadata. Although we had daily backups of this database, the environment that was deleted also contained these backups.
Minutes after the event, the Fractal team started restoring the collection data from our records and from the chain. All data was restored, though it was a manual process and took a while. Thank you for your patience.
I am incredibly proud of the Fractal team for rallying to address the problem, all while maintaining a positive and constructive attitude. Our team will definitely look back at this event as one of those moments where the team really bonded. I’m also thankful to our community, for your understanding and sticking with us.
Of course, we want to prevent this type of issue from occurring ever again. We’re implementing a series of fail-safes to ensure a more diversified backup strategy, stricter access controls and stricter internal processes.
- on behalf of Fractal team