Ketan Bhatt

Home Archive Notes

Custom Sentry Client for more freedom: How to ignore any exception you want

March 08, 2018 programming

We use Sentry for our error reporting, and we love it. It is simple, gives options for a lot of different use cases, and can easily be integrated with Django, over which our backend is built. We recently fixed (or ignored :P) an issue we had for a long time. This article is about how we did it.

The Problem

Because of a play between Django-Tastypie and Celery-Haystack, our celery workers ended up throwing a lot (5 million in 2 years) of harmless ValueError exceptions. Due to other priorities and a large amount of effort required, we couldn’t get down to fixing the source of these exceptions. But soon we started reaching our rate-limits for errors on Sentry. We needed a way to make sentry ignore these particular errors.

The Fix

If it was a logged error (like logger.error), we could simply configure Django’s logging framework appropriately for sentry, but it was an unhandled exception so that would not work.

There are some options for making sentry exclude particular errors:

  1. ignore_exceptions: This is helpful if you want a particular exception to be ignored, and is a more suitable use-case for ignoring custom Exceptions. But we can’t ignore ValueError completely since we stand a chance to ignore other genuine Exceptions that must be brought to our notice.
  2. exclude_paths: This should have worked (according to the docs) and I should have just needed to append celery_haystack.tasks, but that didn’t turn out to be the case. I couldn’t make it work in my testing (we have raven==5.12.0)

Getting no headway there, and in the interest of time (we didn’t want to spend more time than necessary), we decided to dig into some code and look for a good way to achieve this by overriding Sentry Client.

Looking into the code revealed that skip_error_for_logging is the perfect method to override for our use. We are taking a list of prefixes that can be defined in Django settings and testing them against the exception message. If the exception message has one of the specified prefixes, we ignore the error.

Here is our solution:

from django.conf import settings
from raven.contrib.django.client import DjangoClient
ignore_exc_msg_prefixes = getattr(settings, 'CUSTOM_SENTRY_CLIENT_IGNORE_EXCEPTION_MESSAGE_PREFIXES', [])
class IgnoreExceptionsDjangoClient(DjangoClient):
"""
Overrides DjangoClient's `skip_error_for_logging` method.CUSTOM_SENTRY_CLIENT_IGNORE_EXCEPTION_MESSAGE_PREFIXES
Checks raised error message for prefixes defined in settings.CUSTOM_SENTRY_CLIENT_IGNORE_EXCEPTION_MESSAGE_PREFIXES
If the error message has any of the given prefix, skips logging the exception
"""
def skip_error_for_logging(self, exc_info):
skip_error = super(IgnoreExceptionsDjangoClient, self).skip_error_for_logging(exc_info)
if not skip_error and ignore_exc_msg_prefixes:
error_msg = exc_info[1].message
if isinstance(error_msg, basestring):
for prefix in ignore_exc_msg_prefixes:
if error_msg.startswith(prefix):
skip_error = True
break
return skip_error
view raw sentry_client.py hosted with ❤ by GitHub
SENTRY_CLIENT = 'path.to.sentry_client.IgnoreExceptionsDjangoClient'
CUSTOM_SENTRY_CLIENT_IGNORE_EXCEPTION_MESSAGE_PREFIXES = [
"Couldn't load object" # raised by celery_haystack task
]
view raw settings.py hosted with ❤ by GitHub

This provides a lot of control over what exception we want to exclude. And can be extended to check for Exception type and message together, use regexes, or any other advanced handling we like.

Result? Our hourly error rate instantly came down. Here is one of the heart-warming screenshots :)

Graph displaying reduced error count

The green dots are when the deployment took place

I still have a feeling that exclude_paths should have worked, but we went ahead with the solution we could figure out the fastest. If someone knows a better way, please let us know :)

(Originally posted on my Medium account)


Get new posts sent to you

Subscribe to my new work on programming, productivity, and a few other topics. Published once a month.

    Powered By ConvertKit