Custom Sentry Client for more freedom: How to ignore any exception you want
March 08, 2018 • programming
We use Sentry for our error reporting, and we love it. It is simple, gives options for a lot of different use cases, and can easily be integrated with Django, over which our backend is built. We recently fixed (or ignored :P) an issue we had for a long time. This article is about how we did it.
The Problem
Because of a play between Django-Tastypie and Celery-Haystack, our celery workers ended up throwing a lot (5 million in 2 years) of harmless ValueError exceptions. Due to other priorities and a large amount of effort required, we couldn’t get down to fixing the source of these exceptions. But soon we started reaching our rate-limits for errors on Sentry. We needed a way to make sentry ignore these particular errors.
The Fix
If it was a logged error (like logger.error), we could simply configure Django’s logging framework appropriately for sentry, but it was an unhandled exception so that would not work.
There are some options for making sentry exclude particular errors:
ignore_exceptions: This is helpful if you want a particular exception to be ignored, and is a more suitable use-case for ignoring custom Exceptions. But we can’t ignoreValueErrorcompletely since we stand a chance to ignore other genuine Exceptions that must be brought to our notice.exclude_paths: This should have worked (according to the docs) and I should have just needed to appendcelery_haystack.tasks, but that didn’t turn out to be the case. I couldn’t make it work in my testing (we haveraven==5.12.0)
Getting no headway there, and in the interest of time (we didn’t want to spend more time than necessary), we decided to dig into some code and look for a good way to achieve this by overriding Sentry Client.
Looking into the code revealed that skip_error_for_logging is the perfect method to override for our use. We are taking a list of prefixes that can be defined in Django settings and testing them against the exception message. If the exception message has one of the specified prefixes, we ignore the error.
Here is our solution:
| from django.conf import settings | |
| from raven.contrib.django.client import DjangoClient | |
| ignore_exc_msg_prefixes = getattr(settings, 'CUSTOM_SENTRY_CLIENT_IGNORE_EXCEPTION_MESSAGE_PREFIXES', []) | |
| class IgnoreExceptionsDjangoClient(DjangoClient): | |
| """ | |
| Overrides DjangoClient's `skip_error_for_logging` method.CUSTOM_SENTRY_CLIENT_IGNORE_EXCEPTION_MESSAGE_PREFIXES | |
| Checks raised error message for prefixes defined in settings.CUSTOM_SENTRY_CLIENT_IGNORE_EXCEPTION_MESSAGE_PREFIXES | |
| If the error message has any of the given prefix, skips logging the exception | |
| """ | |
| def skip_error_for_logging(self, exc_info): | |
| skip_error = super(IgnoreExceptionsDjangoClient, self).skip_error_for_logging(exc_info) | |
| if not skip_error and ignore_exc_msg_prefixes: | |
| error_msg = exc_info[1].message | |
| if isinstance(error_msg, basestring): | |
| for prefix in ignore_exc_msg_prefixes: | |
| if error_msg.startswith(prefix): | |
| skip_error = True | |
| break | |
| return skip_error |
| SENTRY_CLIENT = 'path.to.sentry_client.IgnoreExceptionsDjangoClient' | |
| CUSTOM_SENTRY_CLIENT_IGNORE_EXCEPTION_MESSAGE_PREFIXES = [ | |
| "Couldn't load object" # raised by celery_haystack task | |
| ] |
This provides a lot of control over what exception we want to exclude. And can be extended to check for Exception type and message together, use regexes, or any other advanced handling we like.
Result? Our hourly error rate instantly came down. Here is one of the heart-warming screenshots :)
The green dots are when the deployment took place
I still have a feeling that exclude_paths should have worked, but we went ahead with the solution we could figure out the fastest. If someone knows a better way, please let us know :)
(Originally posted on my Medium account)