Everything is working as of this moment, but we’re obviously not entirely out of the woods yet. So Basecamp is down again while we get this sorted.Īt 12:41pm CST, Basecamp came back online after we switched over to our backup caching servers. This caused a caching server to get overwhelmed. Need 100%.Īt 12:22pm CST, Basecamp came back online after we successfully verified that all data was 100% intact.Īt 12:33pm CST, Basecamp had another issue dealing with the intense load of the application being back online. It’s looking good, but 99% sure isn’t good enough. We need to be absolutely certain that all the data is in proper sync before we can go back online. We have 4 databases per datacenter and we have two datacenters with databases. We’re working as fast as we can and hope to be back fully shortly.Īt 11:52am CST, verification of the databases is taking longer than expected. The database migration has finished, but the verification process is still ongoing. We’re working on this as fast as we can.Īt 11:33am CST, we’re still verifying that all data is as it should be for Basecamp 3. But we still have to verify all the data, update our configurations, and ensure that we won’t have more problems when we go back online. This was the largest part of the fix we needed to address the problem. So the migration was estimated to take about one hour and forty minutes.Īt 10:56am CST, we completed the upgrade to the databases. That’s why it ran out of regular integers. That table in the database is very large, of course. Changing a production database is serious business, so we had to test this fix on a staging database to make sure it was safe.Īt 7:52am CST, we had verified that the fix was correct and tested it on a staging database, so we commenced making the change to the production database table. This meant writing what’s called a database migration where you change the column type from the regular integer to the big integer type. The big integer can grow until 9223372036854775807.Īt 7:29am CST, the team diagnosed the problem and started working on the fix. The integer runs out of numbers at 2147483647. This was because the column in database was configured as an integer rather than a big integer. Please contact support if you’re in this situation, and we’ll have a team ready to assist.Īt 7:21am CST, we first got alerted that we had run out of ID numbers on an important tracking table in the database. If it doesn’t appear automatically, we can help you recover that data. Note: If you were in the middle of posting something new to Basecamp, and you got an error, that data is most likely saved in our browser-based autosave system. We will work hard to regain your trust, and to get back to our normal, boring schedule of 99.998% uptime. We’ve let you down on an avoidable issue that we should have been on top of. We know that Basecamp customers depend on being able to get to their data and carry on the work, and today we failed you on that. That’s the most catastrophic failure we’ve had a Basecamp in maybe as much as decade, and we could not be more sorry. All in, we were stuck in read-only mode for almost five hours. We will be following up with a detailed and complete postmortem soon. We may still have some backlogs on processing things like incoming emails, and you may still see some slowdowns here and there as we catch up. No emails that were sent to Basecamp during the outage were dropped. All data was confirmed to be fully safe and intact. Basecamp 3 is now back online for reading and writing.
0 Comments
Leave a Reply. |
AuthorWrite something about yourself. No need to be fancy, just an overview. ArchivesCategories |