Refreshing temporary credentials failed during advisory refresh period

[William Ross Morrow]

Describe the bug

I repeatedly get warnings like:

13:18:13.536 | WARNING | aiobotocore.credentials - Refreshing temporary credentials failed during advisory refresh period.
Traceback (most recent call last):
  File "/Users/ross.morrow/Library/Caches/pypoetry/virtualenvs/report-data-pipeline-domain-XSaEb3SG-py3.10/lib/python3.10/site-packages/aiobotocore/credentials.py", line 327, in _protected_refresh
    metadata = await self._refresh_using()
TypeError: object dict can't be used in 'await' expression
13:18:13.965 | WARNING | aiobotocore.credentials - Refreshing temporary credentials failed during advisory refresh period.

after some period of use. If I wait longer, the warnings go away. The underlying actions may still be working; I've stopped the processes after seeing a blast of the warnings. (We also wouldn't want to be sending this out to prod to get hit with a bunch of potentially misleading token refresh warnings.)

The naive inference is that when credentials are valid prior to the referenced "advisory" period they are used, and then when they actually expire they get refreshed; but in that "advisory" period a dict is being returned from self._refresh_using() as passed in as an init arg somewhere (which makes it really hard to tell where the issue might be no?). Again, not sure if the credentials are actually invalid and failing requests.

I'm using this library via another third party (prefect), so I don't really have relevant sample code for reproduction. The WARNING traces come from this library only.

From here

Checklist

• [y] I have reproduced in environment where pip check passes without errors
• [y] I have provided pip freeze results
• [?] I have provided sample code or detailed way to reproduce
• I have tried the same code in botocore to ensure this is an aiobotocore specific issue
• I have tried similar code in aiohttp to ensure this is is an aiobotocore specific issue
• I have checked the latest and older versions of aiobotocore/aiohttp/python to see if this is a regression / injection

Using poetry, with these deps:

[tool.poetry.dependencies]
python = "^3.10"
prefect = "^2.10.5"
prefect-shell = "^0.1.5"
requests = "^2.29.0"
s3fs = "^2023.4.0"
# boto3 = "^1.26.107" conflict with s3fs
psycopg2 = "^2.9.5"

and

$ poetry run pip check
No broken requirements found.

pip freeze results

aiobotocore==2.5.0
aiohttp==3.8.4
aioitertools==0.11.0
aiosignal==1.3.1
aiosqlite==0.19.0
alembic==1.11.1
anyio==3.6.2
apprise==1.4.0
asgi-lifespan==2.1.0
async-timeout==4.0.2
asyncpg==0.27.0
attrs==23.1.0
botocore==1.29.76
cachetools==5.3.0
certifi==2023.5.7
cffi==1.15.1
charset-normalizer==3.1.0
click==8.1.3
cloudpickle==2.2.1
colorama==0.4.6
coolname==2.2.0
croniter==1.3.14
cryptography==40.0.2
dateparser==1.1.8
docker==6.1.2
fastapi==0.95.2
frozenlist==1.3.3
fsspec==2023.5.0
google-auth==2.18.1
greenlet==2.0.2
griffe==0.28.0
h11==0.14.0
h2==4.1.0
hpack==4.0.0
httpcore==0.17.1
httpx==0.24.1
hyperframe==6.0.1
idna==3.4
Jinja2==3.1.2
jmespath==1.0.1
jsonpatch==1.32
jsonpointer==2.3
jsonschema==4.17.3
kubernetes==26.1.0
Mako==1.2.4
Markdown==3.4.3
markdown-it-py==2.2.0
MarkupSafe==2.1.2
mdurl==0.1.2
multidict==6.0.4
oauthlib==3.2.2
orjson==3.8.12
packaging==23.1
pathspec==0.11.1
pendulum==2.1.2
prefect==2.10.10
prefect-shell==0.1.5
psycopg2==2.9.6
pyasn1==0.5.0
pyasn1-modules==0.3.0
pycparser==2.21
pydantic==1.10.7
Pygments==2.15.1
pyrsistent==0.19.3
python-dateutil==2.8.2
python-slugify==8.0.1
pytz==2023.3
pytzdata==2020.1
PyYAML==6.0
readchar==4.0.5
regex==2023.5.5
requests==2.30.0
requests-oauthlib==1.3.1
rich==13.3.5
rsa==4.9
s3fs==2023.5.0
six==1.16.0
sniffio==1.3.0
SQLAlchemy==1.4.48
starlette==0.27.0
text-unidecode==1.3
toml==0.10.2
typer==0.9.0
typing_extensions==4.5.0
tzlocal==5.0.1
urllib3==1.26.15
uvicorn==0.22.0
websocket-client==1.5.1
websockets==11.0.3
wrapt==1.15.0
yarl==1.9.2

Environment:

• Python Version: 3.10
• OS name and version: macOSX M1

Additional context
Add any other context about the problem here.

[Alexander Mohr]

interesting, is it possible for you to print out self._refresh_using to see which method that is? meanwhile I'll keep investigating, thanks

[William Ross Morrow]

interesting, is it possible for you to print out self._refresh_using to see which method that is? meanwhile I'll keep investigating, thanks

Reasonable idea; maybe I can hack something up to see what that object appears to be during that advisory period.

[Alexander Mohr]

ya I did some checks and couldn't figure out which refresh method it could be, ya if you wouldn't mind just smash in a print on that line, and if possible print out self as well. Another thing that would really help is in the constructor to that object print out a traceback.format_exc() to see where the refresh_method is coming from