Very interesting case full of lessons. Of course, increasing the complexity of the system overall can lead to such hard to find issues. It's also a tale into how seemingly innocuous settings can interact in unexpected ways. I also like the lessons learn pointing to the fact that you can and should debug even the systems you use through abstractions, diving into the code is almost always a good thing (even if in this particular case it wasn't strictly necessary in the end). And last but not least it shows the tension between mastery and automation... the more you automate the least you master the system, and at the same time this automation is necessary for building resilience in the system.

There's a new grep alternative in town. Looks really fast and has an interesting interactive mode. Definitely something to check out.

Looks like a very powerful tool for debugging and analyzing processes involving a Python interpreter.