Title. This can be done by setting NGINX's default 5XX page to template/500.html. Note how I said default here: if Django throws an error, admins should still be able to see the debug 500 page as I configured it.
If you're wondering why I'm posting this here instead of @chipmunkmc's profile (she's in charge of the server), well, it's mainly because it's a very minor thing.