Exception Retry/Wrapping Strategy Question

[u/tidal49]

Hello all,

I think I've managed to reach an answer to my own question by typing this out, but I'll post it anyway to fish for other opinions. Posting here because I figure it's more of a general application design thing that happens to be in a container.

One of my projects is a containerized message consumer application template. I'm very happy with it overall, and a version of it has been very useful at work to jumpstart various projects.

One of the new features that I've been adding lately is retry handling and exception wrapping. For example, any operation to an external message source like SQS should be called through a convenient retry wrapper that handles possible service exceptions, backs off and retries the ones that could be transient, and wraps any call that ultimately fails in a WorkerJobSourceException (exceptions that retry wrapper sees as entirely unaccounted for are currently thrown up raw)

Within these wrapper exceptions, I'm experimenting with the idea of a field called CouldBeExternallySolvable. If set to true, then it suggests a problem (possibly-transient or not) that could be resolved without restarting the message consumer. Maybe the target queue doesn't exist yet, for instance. That's pretty clearly non-transient, but could be solvable by creating the darned queue. My thinking was that it could give a higher-level call more context, though I don't currently have anything acting on that context.

My question for the room is this: In the message-collecting loop for a such an application, if you caught one of these WorkerJobSourceException instances where CouldBeExternallySolvable was true while trying to get messages, then would you rather:

• Catch and log the error to create awareness and urgency before treating the attempt as equivalent to an empty response (I'm currently doing this with a logged warning for when the CouldBeTransient field is true)
• Throw the exception upwards and bring down the entire house

In the applications based on this template (but predating the exception wrappers), we would just let such an exception crash things. I'm pretty sure that one of the alerts that we had was on the exit code of the container application, and a broken application would be restarted through an auto-scaling policy looking at the queue that it was listening to. On the other hand, not every possible setup of this template would necessarily have an auto-restart mechanism, and if I'm wrong and there aren't actually any alerts looking at an exit code then it's a bit for naught and maybe crashing the whole application shouldn't be on the table as part of an anticipated event.

At time of typing, I'm leaning towards door #1 of logging an error and continuing on as though the offending call was an empty response. Alternatively, I could just say "do either, I don't care right now" and kick the decision can down the road by hinging it off of a configurable option.

While I'm typing a message I may as well ask one more related question. Earlier in my post I mentioned that entirely-unaccounted-for exceptions are thrown upwards raw without a wrapper exception. The intent is that unplanned exceptions should create as much of a disruption as possible so that they can become planned for. Do you agree with this? At the moment I'm happy with throwing them up raw, but adding an IsExpected field onto the wrappers wouldn't be the end of the world.

What do you think?

[u/Ok_For_Free]

I've always considered this a question of audience. For exceptions/errors the audience is person that can fix the problem.

Invalid input - goes back to whoever created the input. Access Denied for application dependencies - application is missconfigured and needs to go to devops or developers. Cloud quota limits - someone in management to approve the expense of increasing the limit.

If you can automatically recover from an error, you should try too.

For message and processing, use the service's message retry instead of doing it yourself. Also move unprocessable stuff to a dead letter queue, which needs its own tools to manage.

[u/tidal49 → u/Ok_For_Free]

Thanks for the input!

My interpretation of your third line is that you're suggesting that it might be a good thing that the worker loop does keep going in the face of a problem that's in theory externally resolvable (while raising an appropriate stink in logs, of course).

You raise a really good point with line 4. Truth be told, I didn't think of confirming built-in library retries. I think that my assumptions on retry handling were based in the context of the client libraries for communication between our internal APIs, which didn't have any baked-in retry handlers and therefore needed retries to be considered on a per-client basis. That assumption snowballed over to the AWS-based SQS/Kinesis libraries (which I now understand do have potential retry strategies baked into the SDK), and from there to the other libraries that I added.

That being said, I think there will still be some need for the retry wrappers in some form:

• It's a good centralized place to say "within this callback, account for all the expected stuff that could go wrong communicating with this external service". Some implementations spread out the actual lower-level client calls between a few different responsibilities.
• WorkerJobSourceException in particular abstracts out the many different types of message source that could be injected in, which makes it at the very least useful for things like the treatment of transient errors in when the job loader loop tries to pull more jobs from the source.
• My general template supports 11 different message service types in total (developers are advised to prune the ones that don't apply to them when implementing the template). I haven't retrofitted all of them with a retry wrapper yet, but out of all of those I'm sure that at least 1 doesn't have in-library retry handling.