{"id":72926,"date":"2023-04-25T09:00:45","date_gmt":"2023-04-25T09:00:45","guid":{"rendered":"https:\/\/www.cryptocabaret.com\/?p=72926"},"modified":"2023-04-25T09:00:45","modified_gmt":"2023-04-25T09:00:45","slug":"retry-your-python-code-until-it-fails","status":"publish","type":"post","link":"https:\/\/www.cryptocabaret.com\/?p=72926","title":{"rendered":"Retry your Python code until it fails"},"content":{"rendered":"

Retry your Python code until it fails<\/span>
\nMoshe Zadka<\/a><\/span>
\nTue, 04\/25\/2023 – 03:00<\/span><\/p>\n

\n

Sometimes, a function is called with bad inputs or in a bad program state, so it fails. In languages like Python, this usually results in an exception.<\/p>\n

But sometimes exceptions are caused by different issues or are transitory. Imagine code that must keep working in the face of caching data being cleaned up. In theory, the code and the cleaner could carefully agree on the clean-up methodology to prevent the code from trying to access a non-existing file or directory. Unfortunately, that approach is complicated and error-prone. However, most of these problems are transitory, as the cleaner will eventually create the correct structures.<\/p>\n

Even more frequently, the uncertain nature of network programming means that some functions that abstract a network call fail because packets were lost or corrupted.<\/p>\n

A common solution is to retry the failing code. This practice allows skipping past transitional problems while still (eventually) failing if the issue persists. Python has several libraries to make retrying easier. This is a common “finger exercise.”<\/p>\n

Tenacity<\/h2>\n

One library that goes beyond a finger exercise and into useful abstraction is tenacity<\/a>. Install it with pip install tenacity<\/code> or depend on it using a dependencies = tenacity<\/code> line in your pyproject.toml<\/code> file.<\/p>\n

Set up logging<\/h2>\n

A handy built-in feature of tenacity<\/code> is support for logging. With error handling, seeing log details about retry attempts is invaluable.<\/p>\n

To allow the remaining examples display log messages, set up the logging library<\/a>. In a real program, the central entry point or a logging configuration plugin does this. Here’s a sample:<\/p>\n

\nimport logging\n\nlogging.basicConfig(\n    level=logging.INFO,\n    format=\"%(asctime)s:%(name)s:%(levelname)s:%(message)s\",\n)\n\nTENACITY_LOGGER = logging.getLogger(\"Retrying\")<\/code><\/pre>\n
Selective failure<\/h2>\n
To demonstrate the features of tenacity<\/code>, it’s helpful to have a way to fail a few times before finally succeeding. Using unittest.mock<\/code> is useful for this scenario.<\/p>\n
\nfrom unittest import mock\n\nthing = mock.MagicMock(side_effect=[ValueError(), ValueError(), 3])<\/code><\/pre>\n
If you’re new to unit testing, read my article on mock<\/a>.<\/p>\n
Before showing the power of tenacity<\/code>, look at what happens when you implement retrying directly inside a function. Demonstrating this makes it easy to see the manual effort using tenacity<\/code> saves.<\/p>\n
\ndef useit(a_thing):\n    for i in range(3):\n        try:\n            value = a_thing()\n        except ValueError:\n            TENACITY_LOGGER.info(\"Recovering\")\n            continue\n        else:\n            break\n    else:\n        raise ValueError()\n    print(\"the value is\", value)<\/code><\/pre>\n
The function can be called with something that never fails:<\/p>\n
\n>>> useit(lambda: 5)\nthe value is 5<\/code><\/pre>\n
With the eventually-successful thing:<\/p>\n
\n>>> useit(thing)\n\n2023-03-29 17:00:42,774:Retrying:INFO:Recovering\n2023-03-29 17:00:42,779:Retrying:INFO:Recovering\n\nthe value is 3<\/code><\/pre>\n
Calling the function with something that fails too many times ends poorly:<\/p>\n
\ntry:\n    useit(mock.MagicMock(side_effect=[ValueError()] * 5 + [4]))\nexcept Exception as exc:\n    print(\"could not use it\", repr(exc))<\/code><\/pre>\n
The result:<\/p>\n
\n\n2023-03-29 17:00:46,763:Retrying:INFO:Recovering\n2023-03-29 17:00:46,767:Retrying:INFO:Recovering\n2023-03-29 17:00:46,770:Retrying:INFO:Recovering\n\ncould not use it ValueError()<\/code><\/pre>\n
Simple tenacity usage<\/h2>\n
For the most part, the function above was retrying code. The next step is to have a decorator handle the retrying logic:<\/p>\n
\nimport tenacity\n\nmy_retry=tenacity.retry(\n    stop=tenacity.stop_after_attempt(3),\n    after=tenacity.after_log(TENACITY_LOGGER, logging.WARNING),\n)<\/code><\/pre>\n
Tenacity supports a specified number of attempts and logging after getting an exception.<\/p>\n
The useit<\/code> function no longer has to care about retrying. Sometimes it makes sense for the function to still consider retryability<\/em>. Tenacity allows code to determine retryability by itself by raising the special exception TryAgain<\/code>:<\/p>\n
\n@my_retry\ndef useit(a_thing):\n    try:\n        value = a_thing()\n    except ValueError:\n        raise tenacity.TryAgain()\n    print(\"the value is\", value)<\/code><\/pre>\n
Now when calling useit<\/code>, it retries ValueError<\/code> without needing custom retrying code:<\/p>\n
\nuseit(mock.MagicMock(side_effect=[ValueError(), ValueError(), 2]))<\/code><\/pre>\n
The output:<\/p>\n
\n2023-03-29 17:12:19,074:Retrying:WARNING:Finished call to '__main__.useit' after 0.000(s), this was the 1st time calling it.\n2023-03-29 17:12:19,080:Retrying:WARNING:Finished call to '__main__.useit' after 0.006(s), this was the 2nd time calling it.\n\nthe value is 2<\/code><\/pre>\n
Configure the decorator<\/h2>\n
The decorator above is just a small sample of what tenacity<\/code> supports. Here’s a more complicated decorator:<\/p>\n
\nmy_retry = tenacity.retry(\n    stop=tenacity.stop_after_attempt(3),\n    after=tenacity.after_log(TENACITY_LOGGER, logging.WARNING),\n    before=tenacity.before_log(TENACITY_LOGGER, logging.WARNING),\n    retry=tenacity.retry_if_exception_type(ValueError),\n    wait=tenacity.wait_incrementing(1, 10, 2),\n    reraise=True\n)<\/code><\/pre>\n<\/p>\n
\n
More Python resources<\/div>\n
\n
What is an IDE?<\/a><\/div>\n
Cheat sheet: Python 3.7 for beginners<\/a><\/div>\n
Top Python GUI frameworks<\/a><\/div>\n
Download: 7 essential PyPI libraries<\/a><\/div>\n
Red Hat Developers<\/a><\/div>\n
Register for your free Red Hat account<\/a><\/div>\n
Latest Python articles<\/a><\/div>\n<\/p><\/div>\n<\/p><\/div>\n
This is a more realistic decorator example with additional parameters:<\/p>\n