Newer posts are loading.
You are at the newest post.
Click here to check if anything new just came in.

Ok, so that took a little bit longer than expected

TLDR; major database change went over it's time limit, we had to abort the operation, and now we'll go the longer and slower path to enlightenment.

Long story: What we're trying to do right now is to end or at least reduce one of the blights on soup performance, which is how the notifications work. Sadly, this isn't just done with a software upgrade, but requires a change to the database that isn't cheap.

For the nerds among you, our notifications table has a size of 6.5GB - entire social networks would fit in that space. At soup, all notifications you've every received are in there. Why that is is beyond me (it's probably because of the ramification of trying to change it...), as we do not show more than 75 to the user at any given time, so there's some room for improvement. And every notification has a type, such as "Reaction" for when someone reacts to one of your posts or "Update" for when we write a post on Now, that's not that bad. What is bad however is, that due to the shortcomings of early rails/active record versions, the type column is varchar ... yes, you read right. We have 6.5GB of data that largely consists of the same ten strings.

What we were trying to do this morning is to convert that varchar typed column to an enum typed column. But the table didn't stop to grow, and while at 6:00am UTC I still thought it might work out, I did abort at around 8:00, because at some point you have to see the errors of your ways.

Welp, now we'll be slowly migrating the rows into an additional, alternate column and doing some VACUUMing in between. At the end, we'll just rename the columns, and hopefully, not produce an outage like that of today again.

Don't be the product, buy the product!