asyncio.exceptions.TimeoutError

[kolt54321]

Hi @lovasoa! I've recently hit this error a few times. Any idea if we could put in a retry in the code?

https://artsandculture.google.com/asset/the-last-day-of-pompeii-karl-brullov/tAFrCGFUhXM8Jg
Defaulting to highest zoom level (8).
Zoom level 8 too high for JPEG output, using next zoom level 7 instead
Using zoom level 7.
Downloading tiles...
Traceback (most recent call last):
  File "C:\Users\i\Downloads\gapdecoder\async_tile_fetcher.py", line 8, in modified
    return await f(*args, **kwargs)
  File "C:\Users\i\Downloads\gapdecoder\async_tile_fetcher.py", line 22, in fetch
    async with session.get(url) as response:
  File "C:\Users\i\AppData\Local\Programs\Python\Python38\lib\site-packages\aiohttp\client.py", line 1012, in __aenter__
    self._resp = await self._coro
  File "C:\Users\i\AppData\Local\Programs\Python\Python38\lib\site-packages\aiohttp\client.py", line 582, in _request
    break
  File "C:\Users\i\AppData\Local\Programs\Python\Python38\lib\site-packages\aiohttp\helpers.py", line 596, in __exit__
    raise asyncio.TimeoutError from None
asyncio.exceptions.TimeoutError

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "tile_fetch_3.py", line 353, in <module>
    main()
  File "tile_fetch_3.py", line 306, in main
    loop.run_until_complete(coro)
  File "C:\Users\i\AppData\Local\Programs\Python\Python38\lib\asyncio\base_events.py", line 608, in run_until_complete
    return future.result()
  File "tile_fetch_3.py", line 162, in load_tiles
    tiles = await async_tile_fetcher.gather_progress(awaitable_tiles)
  File "C:\Users\i\Downloads\gapdecoder\async_tile_fetcher.py", line 42, in gather_progress
    total = await asyncio.gather(*[
  File "C:\Users\i\Downloads\gapdecoder\async_tile_fetcher.py", line 36, in print_percent
    res = await awaitable
  File "tile_fetch_3.py", line 118, in fetch_tile
    encrypted_bytes = await async_tile_fetcher.fetch(session, image_url, file_path)
  File "C:\Users\i\Downloads\gapdecoder\async_tile_fetcher.py", line 13, in modified
    raise err
Exception

[lovasoa]

Hi ! Yes, we could implement a retry. We should also make the timeout configurable

[kolt54321]

I'm not sure if this is related, but I also got the following error today:

Defaulting to highest zoom level (7).
Using zoom level 7.
Downloading tiles...
Traceback (most recent call last):
  File "C:\Users\i\Downloads\gapdecoder\async_tile_fetcher.py", line 8, in modified
    return await f(*args, **kwargs)
  File "C:\Users\i\Downloads\gapdecoder\async_tile_fetcher.py", line 23, in fetch
    response.raise_for_status()
  File "C:\Users\i\AppData\Local\Programs\Python\Python38\lib\site-packages\aiohttp\client_reqrep.py", line 941, in raise_for_status
    raise ClientResponseError(
aiohttp.client_exceptions.ClientResponseError: 500, message='Internal Server Error', url=URL('https://lh3.googleusercontent.com/-xDIuwuw2KZ9_9gngpUhtI86GN1gpfANRdioBtz6fSmjqeC09DSPHiBOOhB9=x99-y15-z7-t6SEp9Xt3efk5zgbR_7EWu0M_7WI')```

It looks like it's on Google's end.  

[lovasoa]

Yes, it's on google's end.
Dezoomify-rs implements an exponential backoff retry strategy to work around this.

[kolt54321]

Any idea how we'd do it for gapdecoder? I'm not very familiar with the internet side of the program so it would be great if we had something along those lines here.

In the meantime for my personal use I can let it skip the image if it encounters and error, but here there should probably be a more permanent solution.

I'm still moping that I can't figure out how to adapt the rust code that cleanly gets around the memory issue...

[lovasoa]

The code would look something like that :

async def retry_get(url, retries):
  s = 1
  err = None
  for i in range(retries):
    try:
      return await get(url)
    except HTTPError as e:
       err = e
       await asyncio.sleep(s)
       s *= 2   
  raise err

[lovasoa]

Or you can use a library such as https://tenacity.readthedocs.io/en/latest/