Configure Postgres statement_timeout from within Django

“A close-up of white dials on a music mixer” by Alexey Ruban on UnsplashIn a bid to prepare ourselves for projected growth, we are at the moment trying to figure out what part of our system will break at what scale, and how. One step towards this was to also define strict timeouts for our … Continue reading Configure Postgres statement_timeout from within Django

Custom Sentry Client for more freedom: How to ignore any exception you want

We use Sentry for our error reporting, and we love it. It is simple, gives options for a lot of different use cases, and can easily be integrated with Django, over which our backend is built. We recently fixed (or ignored :P) an issue we had for a long time. This article is about how … Continue reading Custom Sentry Client for more freedom: How to ignore any exception you want

Two years with Celery in Production: Bug Fix Edition

Photo by Martin Oslic on UnsplashAs mentioned in an earlier post, we rely on Celery for publishing and consuming tasks to/from our RabbitMQ (RMQ) broker. We are very happy with the whole setup and it works reliably for us. But this wasn’t exactly the case up until four weeks back. We were plagued with a … Continue reading Two years with Celery in Production: Bug Fix Edition

Leveraging AWS Lambda for Image Compression at scale

A lot of problems we solve at Squad deal with images. Images to be tagged, labeled, moderated, transcribed etc. And there is this thing about images, they are generally heavier than text, digitally. And if our contractors (users of our app) spend more money on internet packs/plans than what they earn, or if they have … Continue reading Leveraging AWS Lambda for Image Compression at scale

Hack Django ORM to calculate Median and Percentiles (Or make annotations great again!)

Hacks should be quick. And so should be the articles about them. Photo by Caspar Rubin on Unsplash Problem We needed to calculate medians, percentiles for some quantities for our own ETL system (note to self: write a post on this) grouped by certain fields. Some options we had: Extra: But Django says this will … Continue reading Hack Django ORM to calculate Median and Percentiles (Or make annotations great again!)

Blazingly fast querying on huge tables by avoiding joins

Tl;dr: Avoid joins on large tables and evaluate parts of queries beforehand to get 100–10,000x performance gains! As mentioned in a previous post, because of some of our tables growing in size, our queries started performing poorly which resulted in a performance hit to our most used APIs. It was time we revisit some of … Continue reading Blazingly fast querying on huge tables by avoiding joins