TLDR; Use squadrun/django-haystack and live your life happily. Also always index in batches. Also read the whole of it.
A problem our Operations Team at SquadRun was facing was how slow the searches on the Admin panel used to work. It wasn’t surprising though. We had around 30 Million to 8 Hundred Thousand rows in different models. Now that is not a lot of data, but it is certainly a lot if you want to search on a lot of text and non-text fields simultaneously. We also didn’t want our database to spend resources on the search queries. At a time when the database is already under load, these searches would:
- Return late
- Use even more resources of the db already under load
We decided this needs to stop as it kills the team’s effectiveness. So we planned to index those models which are frequently searched in an ElasticSearch (ES) server. We also needed a way to integrate ES with the django admin (because we do searches only from the admin). We had two options:
- Custom implementation for the search, and using ES APIs directly
- Something like Haystack which handles all of this for you.
It is advisable for you to directly deal with ES’s API if you have something complex in your mind, or if you are an expert with ES (but then you wouldn’t have been reading this). If your goal is just to do away with admin searches hitting your database, and want to plug in ES into the flow, I suggest haystack will serve you well.
Is that it?
- Updating index every time a row is added/updated is time taking, and sometimes updates happen in bulk (
MyModel.objects.filter(old_stuff=True).update(old_stuff=False)), and this won’t get updated in the index automatically. Solution
- Defining what fields to index using templates takes me away from code, and every time I want to find what fields I am indexing, I will have to dig out the template file for the model. Cumbersome. Solution
- Indexing was slow if you were indexing fields across tables. Solution
- No filter and ordering support :( Solution
- Django haystack EdgeNgramField gives different results than elasticsearch. Solution
edgengramtokenizer and filter’s
max_gramwere in the range 3-15, that meant words like
a, an, ofwon’t get indexed and a search for
King of Nepalwill not return anything because
ofis not in the index and so there is no match. We could ask our team to eliminate such words from their searches, but that is not just it is supposed to be done, we wanted to make their life easy, and not make them remember new rules. Solution
All I see is complaints
Me too, here is what we did to solve them:
Updating Index asynchronously And indexing everything else that was left
This one is a no-brainer and is even suggested in haystack’s docs. Use celery-haystack to update index asynchronously.
(Gotcha! –> add
CELERY_HAYSTACK_COUNTDOWN = 2 to your settings, otherwise you are going to get a lot of
So celery-haystack works by catching save/delete signals that django throws. Cool? Not yet.
What about updates/creates that happen in bulk? No signals are emitted for those and that would mean those things will never be indexed. This is not ideal. You can not stop updating/creating in bulk because of this. So, we decided to run a cron every 10 minutes that reindexes everything that was created/updated in the last 10 minutes.
How did we know what happened in the last 10 minutes? We have an
updated_at field in the models we are indexing, so that combined with haystack’s
update_index management command and its
age parameter, we were able to achieve it.
Another problem. When we do a
queryset.update() in django,
auto_now fields are not updated. Now what? Now a little hack
We use this function wherever we are doing a
queryset.update(). So this is solved.
Make haystack templates obsolete and faster indexing
- Need to index faster –> Use
select_relatedwhile fetching objects to index
- Track indexed fields in code –> Get inspired by django admin’s implementation of
So we coded a mixin:
Self explanatory. Add this mixin to your search indexes classes. Also implements
get_updated_field because I didn’t want to copy this code everywhere.
Solve problem I can’t seem to find a name for and modify token’s length
For that, we overrode the
ElasticsearchSearchEngine class and came up with:
Now plug this Backend for your
ENGINE key in
HAYSTACK_CONNECTIONS settings,and you are golden.
Fix filtering and ordering
Yo, Done! Read on to know more.
So do I copy all of this into my code?
This is how my indexes look like:
See any mistakes in the fork, send a PR. Want to do similar things and solve problems, send an Email!