error during "gather"
jurajsucik opened this issue · comments
all works fine until I get this error in phase 2:
Starting data gathering phase 1 of 2 (collecting objects)
Starting data gathering phase 2 of 2 (collecting properties and relationships)
[Errno 60] Operation timed out
Refreshed token
[Errno 54] Connection reset by peer ---> about 10 times
Cannot connect to host graph.windows.net:443 ssl:default [nodename nor servname provided, or not known]
Cannot connect to host graph.windows.net:443 ssl:default [Too many open files] ---> 100s of times
Traceback (most recent call last):
File "/usr/local/lib/python3.9/site-packages/sqlalchemy/engine/base.py", line 1256, in _execute_context
self.dialect.do_executemany(
File "/usr/local/lib/python3.9/site-packages/sqlalchemy/engine/default.py", line 605, in do_executemany
cursor.executemany(statement, parameters)
sqlite3.OperationalError: unable to open database file
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/usr/local/bin/roadrecon", line 8, in
sys.exit(main())
File "/usr/local/lib/python3.9/site-packages/roadtools/roadrecon/main.py", line 118, in main
gathermain(args)
File "/usr/local/lib/python3.9/site-packages/roadtools/roadrecon/gather.py", line 422, in main
asyncio.run(run(args, dburl))
File "/usr/local/Cellar/python@3.9/3.9.1_4/Frameworks/Python.framework/Versions/3.9/lib/python3.9/asyncio/runners.py", line 44, in run
return loop.run_until_complete(main)
File "/usr/local/Cellar/python@3.9/3.9.1_4/Frameworks/Python.framework/Versions/3.9/lib/python3.9/asyncio/base_events.py", line 642, in run_until_complete
return future.result()
File "/usr/local/lib/python3.9/site-packages/roadtools/roadrecon/gather.py", line 371, in run
await asyncio.gather(*tasks)
File "/usr/local/lib/python3.9/site-packages/roadtools/roadrecon/gather.py", line 197, in dump_mfa
self.session.commit()
File "/usr/local/lib/python3.9/site-packages/sqlalchemy/orm/session.py", line 1046, in commit
self.transaction.commit()
File "/usr/local/lib/python3.9/site-packages/sqlalchemy/orm/session.py", line 504, in commit
self._prepare_impl()
File "/usr/local/lib/python3.9/site-packages/sqlalchemy/orm/session.py", line 483, in _prepare_impl
self.session.flush()
File "/usr/local/lib/python3.9/site-packages/sqlalchemy/orm/session.py", line 2540, in flush
self._flush(objects)
File "/usr/local/lib/python3.9/site-packages/sqlalchemy/orm/session.py", line 2682, in flush
transaction.rollback(capture_exception=True)
File "/usr/local/lib/python3.9/site-packages/sqlalchemy/util/langhelpers.py", line 68, in exit
compat.raise(
File "/usr/local/lib/python3.9/site-packages/sqlalchemy/util/compat.py", line 182, in raise
raise exception
File "/usr/local/lib/python3.9/site-packages/sqlalchemy/orm/session.py", line 2642, in _flush
flush_context.execute()
File "/usr/local/lib/python3.9/site-packages/sqlalchemy/orm/unitofwork.py", line 422, in execute
rec.execute(self)
File "/usr/local/lib/python3.9/site-packages/sqlalchemy/orm/unitofwork.py", line 540, in execute
self.dependency_processor.process_saves(uow, states)
File "/usr/local/lib/python3.9/site-packages/sqlalchemy/orm/dependency.py", line 1176, in process_saves
self._run_crud(
File "/usr/local/lib/python3.9/site-packages/sqlalchemy/orm/dependency.py", line 1239, in _run_crud
connection.execute(statement, secondary_insert)
File "/usr/local/lib/python3.9/site-packages/sqlalchemy/engine/base.py", line 1011, in execute
return meth(self, multiparams, params)
File "/usr/local/lib/python3.9/site-packages/sqlalchemy/sql/elements.py", line 298, in _execute_on_connection
return connection._execute_clauseelement(self, multiparams, params)
File "/usr/local/lib/python3.9/site-packages/sqlalchemy/engine/base.py", line 1124, in _execute_clauseelement
ret = self._execute_context(
File "/usr/local/lib/python3.9/site-packages/sqlalchemy/engine/base.py", line 1316, in _execute_context
self.handle_dbapi_exception(
File "/usr/local/lib/python3.9/site-packages/sqlalchemy/engine/base.py", line 1510, in handle_dbapi_exception
util.raise(
File "/usr/local/lib/python3.9/site-packages/sqlalchemy/util/compat.py", line 182, in raise
raise exception
File "/usr/local/lib/python3.9/site-packages/sqlalchemy/engine/base.py", line 1256, in _execute_context
self.dialect.do_executemany(
File "/usr/local/lib/python3.9/site-packages/sqlalchemy/engine/default.py", line 605, in do_executemany
cursor.executemany(statement, parameters)
sqlalchemy.exc.OperationalError: (sqlite3.OperationalError) unable to open database file
[SQL: INSERT INTO lnk_group_member_user ("Group", "User") VALUES (?, ?)]
[parameters: (('27218c8d-3980-41ae-a41d-b31559898e92', 'd1f1bdb0-2241-4d63-a880-d24a5ea8b8f5'), ('27218c8d-3980-41ae-a41d-b31559898e92', 'ea3c8cfc-5bfd-4d99-b359-72c187cdcc8a'), ('27218c8d-3980-41ae-a41d-b31559898e92', 'de8c5914-7fd7-4670-8eab-9bb58d8a17cb'), ('27218c8d-3980-41ae-a41d-b31559898e92', '1a42fa2e-5793-4ec7-84d9-02b73a750c01'), ('27218c8d-3980-41ae-a41d-b31559898e92', '408d1b0b-e012-4caf-b825-02912315eb95'), ('27218c8d-3980-41ae-a41d-b31559898e92', '5b655b0f-2d44-4075-a170-e313e07f40cd'), ('27218c8d-3980-41ae-a41d-b31559898e92', 'e9064cb0-63d0-4336-b261-680b4d08c760'), ('27218c8d-3980-41ae-a41d-b31559898e92', '0b3df032-bdea-4af5-ab46-a03f6107b07a') ... displaying 10 of 300 total bound parameter sets ... ('27218c8d-3980-41ae-a41d-b31559898e92', 'eaa0e2b9-05f7-45f9-8939-ae525f523150'), ('27218c8d-3980-41ae-a41d-b31559898e92', '8dddc835-087c-49be-a269-237bdcc04e7e'))]
(Background on this error at: http://sqlalche.me/e/13/e3q8)
is this a consistent error or does it go away if you run it again? sounds like a temporary network error
it's a large tenant. if I run gather again, will start from 0 or will it continue where it stopped?
unfortunately it'll start from 0 again. the latest version on github should be faster in phase 2 in larger tenants though, through some improvements i worked at this week
thanks, I'll give a try.
this is from the 2nd try with github version:
Refreshed token
Starting data gathering phase 1 of 2 (collecting objects)
Gathered 102403 groups, switching to 3-phase approach for efficiency
Starting data gathering phase 2 of 3 (collecting properties and relationships)
[Errno 60] Operation timed out
[Errno 60] Operation timed out
Refreshed token
[Errno 60] Operation timed out
[Errno 60] Operation timed out
[Errno 60] Operation timed out
[Errno 60] Operation timed out
[Errno 60] Operation timed out
[Errno 60] Operation timed out
Cannot connect to host graph.windows.net:443 ssl:default [Too many open files] -----> 100s of these messages
Traceback (most recent call last):
File "/usr/local/lib/python3.9/site-packages/sqlalchemy/engine/base.py", line 1276, in _execute_context
self.dialect.do_execute(
File "/usr/local/lib/python3.9/site-packages/sqlalchemy/engine/default.py", line 608, in do_execute
cursor.execute(statement, parameters)
sqlite3.OperationalError: unable to open database file
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/usr/local/bin/roadrecon", line 33, in
sys.exit(load_entry_point('roadrecon', 'console_scripts', 'roadrecon')())
File "/Users/Do_Not_Scan/repos/ROADtools/roadrecon/roadtools/roadrecon/main.py", line 119, in main
gathermain(args)
File "/Users/Do_Not_Scan/repos/ROADtools/roadrecon/roadtools/roadrecon/gather.py", line 470, in main
asyncio.run(run(args, dburl))
File "/usr/local/Cellar/python@3.9/3.9.1_8/Frameworks/Python.framework/Versions/3.9/lib/python3.9/asyncio/runners.py", line 44, in run
return loop.run_until_complete(main)
File "/usr/local/Cellar/python@3.9/3.9.1_8/Frameworks/Python.framework/Versions/3.9/lib/python3.9/asyncio/base_events.py", line 642, in run_until_complete
return future.result()
File "/Users/Do_Not_Scan/repos/ROADtools/roadrecon/roadtools/roadrecon/gather.py", line 399, in run
await asyncio.gather(*tasks)
File "/Users/Do_Not_Scan/repos/ROADtools/roadrecon/roadtools/roadrecon/gather.py", line 218, in dump_mfa
self.session.commit()
File "/usr/local/lib/python3.9/site-packages/sqlalchemy/orm/session.py", line 1046, in commit
self.transaction.commit()
File "/usr/local/lib/python3.9/site-packages/sqlalchemy/orm/session.py", line 504, in commit
self._prepare_impl()
File "/usr/local/lib/python3.9/site-packages/sqlalchemy/orm/session.py", line 483, in _prepare_impl
self.session.flush()
File "/usr/local/lib/python3.9/site-packages/sqlalchemy/orm/session.py", line 2540, in flush
self._flush(objects)
File "/usr/local/lib/python3.9/site-packages/sqlalchemy/orm/session.py", line 2682, in flush
transaction.rollback(capture_exception=True)
File "/usr/local/lib/python3.9/site-packages/sqlalchemy/util/langhelpers.py", line 68, in exit
compat.raise(
File "/usr/local/lib/python3.9/site-packages/sqlalchemy/util/compat.py", line 182, in raise
raise exception
File "/usr/local/lib/python3.9/site-packages/sqlalchemy/orm/session.py", line 2642, in _flush
flush_context.execute()
File "/usr/local/lib/python3.9/site-packages/sqlalchemy/orm/unitofwork.py", line 422, in execute
rec.execute(self)
File "/usr/local/lib/python3.9/site-packages/sqlalchemy/orm/unitofwork.py", line 586, in execute
persistence.save_obj(
File "/usr/local/lib/python3.9/site-packages/sqlalchemy/orm/persistence.py", line 230, in save_obj
_emit_update_statements(
File "/usr/local/lib/python3.9/site-packages/sqlalchemy/orm/persistence.py", line 994, in _emit_update_statements
c = cached_connections[connection].execute(
File "/usr/local/lib/python3.9/site-packages/sqlalchemy/engine/base.py", line 1011, in execute
return meth(self, multiparams, params)
File "/usr/local/lib/python3.9/site-packages/sqlalchemy/sql/elements.py", line 298, in _execute_on_connection
return connection._execute_clauseelement(self, multiparams, params)
File "/usr/local/lib/python3.9/site-packages/sqlalchemy/engine/base.py", line 1124, in _execute_clauseelement
ret = self._execute_context(
File "/usr/local/lib/python3.9/site-packages/sqlalchemy/engine/base.py", line 1316, in _execute_context
self.handle_dbapi_exception(
File "/usr/local/lib/python3.9/site-packages/sqlalchemy/engine/base.py", line 1510, in handle_dbapi_exception
util.raise(
File "/usr/local/lib/python3.9/site-packages/sqlalchemy/util/compat.py", line 182, in raise
raise exception
File "/usr/local/lib/python3.9/site-packages/sqlalchemy/engine/base.py", line 1276, in _execute_context
self.dialect.do_execute(
File "/usr/local/lib/python3.9/site-packages/sqlalchemy/engine/default.py", line 608, in do_execute
cursor.execute(statement, parameters)
sqlalchemy.exc.OperationalError: (sqlite3.OperationalError) unable to open database file
[SQL: UPDATE "Users" SET "strongAuthenticationDetail"=? WHERE "Users"."objectId" = ?]
[parameters: ('{"encryptedPinHash": null, "encryptedPinHashHistory": null, "methods": [], "oathTokenMetadata": [{"id": "ef6025cc-1369-4599-afa5-fffed1014238", "enab ... (531 characters truncated) ... hTokenTimeDrift": 0, "timeInterval": null, "phoneAppVersion": "6.5.4", "notificationType": "APNS"}], "proofupTime": null, "verificationDetail": null}', '7321c9dd-bdc8-495a-bed6-b49b431abec0')]
(Background on this error at: http://sqlalche.me/e/13/e3q8)
any other ideas?
interestingly enough, it always breaks when the db size is just a bit more than 900 MB
other errors from a different run on my linux vm, same tenant:
Error 404 for URL https://graph.windows.net/myorganization/users/370cd6ae-4048-4ec3-8757-805e23508429?api-version=1.61-internal&$select=strongAuthenticationDetail,objectId
Non-existing child found: 2925a71b-3670-48ad-b08f-52505d43a2e8
Non-existing child found: aa685d49-fbda-46f0-9266-b5a3707102a4
Non-existing child found: 8cbed900-a815-4b8f-b402-a687f30c5c93
Non-existing child found: 2925a71b-3670-48ad-b08f-52505d43a2e8
Error 404 for URL https://graph.windows.net/myorganization/users/81d3dd5e-f69a-41d2-b9fc-604ef302c798?api-version=1.61-internal&$select=strongAuthenticationDetail,objectId
Traceback (most recent call last):
File "/home/juraj/.local/bin/roadrecon", line 8, in
sys.exit(main())
File "/home/juraj/.local/lib/python3.7/site-packages/roadtools/roadrecon/main.py", line 118, in main
gathermain(args)
File "/home/juraj/.local/lib/python3.7/site-packages/roadtools/roadrecon/gather.py", line 422, in main
asyncio.run(run(args, dburl))
File "/usr/lib/python3.7/asyncio/runners.py", line 43, in run
return loop.run_until_complete(main)
File "/usr/lib/python3.7/asyncio/base_events.py", line 587, in run_until_complete
return future.result()
File "/home/juraj/.local/lib/python3.7/site-packages/roadtools/roadrecon/gather.py", line 371, in run
await asyncio.gather(*tasks)
File "/home/juraj/.local/lib/python3.7/site-packages/roadtools/roadrecon/gather.py", line 195, in dump_mfa
await asyncio.gather(*jobs)
File "/home/juraj/.local/lib/python3.7/site-packages/roadtools/roadrecon/gather.py", line 178, in dump_mfa_to_db
obj = await dumpsingle(url, method=method)
File "/home/juraj/.local/lib/python3.7/site-packages/roadtools/roadrecon/gather.py", line 85, in dumpsingle
checktoken()
File "/home/juraj/.local/lib/python3.7/site-packages/roadtools/roadrecon/gather.py", line 70, in checktoken
auth.client_id = token['_clientId']
KeyError: '_clientId'
there have been several changes in the data gathering, it should be much more reliable now, I'd be interested if this still occurs with the latest version
I don't have access to that large tenant anymore. I will try it once I get access to the some larger tenants again.
is the error still there? @jurajsucik have you had a chance to validate?
gonna run it on a few tenants this weekend. will report back.
I ran gather twice and I am getting similar results to this one:
Refreshed token
Starting data gathering phase 1 of 2 (collecting objects)
Gathered 77878 groups, switching to 3-phase approach for efficiency
Starting data gathering phase 2 of 3 (collecting properties and relationships)
Error 404 for URL https://graph.windows.net/c8e8bdb6-8e7c-41d5-bc41-37fff3c33b75/users/1ed5f7d3-c7a7-49a9-be73-b40db5b878ab?api-version=1.61-internal&$select=strongAuthenticationDetail,objectId
Error 404 for URL https://graph.windows.net/c8e8bdb6-8e7c-41d5-bc41-37fff3c33b75/users/3cc4d425-57a2-45b5-bb9e-17ddf5f3fd68?api-version=1.61-internal&$select=strongAuthenticationDetail,objectId
Error 404 for URL https://graph.windows.net/c8e8bdb6-8e7c-41d5-bc41-37fff3c33b75/users/4a981c03-115e-4883-b4d6-616f9bab09ff?api-version=1.61-internal&$select=strongAuthenticationDetail,objectId
Error 404 for URL https://graph.windows.net/c8e8bdb6-8e7c-41d5-bc41-37fff3c33b75/users/5eabd60c-2ee3-40e9-98c9-0c7a3d555426?api-version=1.61-internal&$select=strongAuthenticationDetail,objectId
Error 404 for URL https://graph.windows.net/c8e8bdb6-8e7c-41d5-bc41-37fff3c33b75/users/60212615-2814-44d8-9322-c47e2d66a53b?api-version=1.61-internal&$select=strongAuthenticationDetail,objectId
Error 404 for URL https://graph.windows.net/c8e8bdb6-8e7c-41d5-bc41-37fff3c33b75/users/766163fb-ee55-4a1e-873e-3cdd9128ea1b?api-version=1.61-internal&$select=strongAuthenticationDetail,objectId
Error 404 for URL https://graph.windows.net/c8e8bdb6-8e7c-41d5-bc41-37fff3c33b75/users/899298a9-1fc2-4a97-921a-605c55d0cf1d?api-version=1.61-internal&$select=strongAuthenticationDetail,objectId
Error 404 for URL https://graph.windows.net/c8e8bdb6-8e7c-41d5-bc41-37fff3c33b75/users/c03a179c-c2e7-476e-9108-b7ef6ad6f92a?api-version=1.61-internal&$select=strongAuthenticationDetail,objectId
Error 404 for URL https://graph.windows.net/c8e8bdb6-8e7c-41d5-bc41-37fff3c33b75/users/ca7d0432-d9d6-4264-8ef7-6d54f20319d5?api-version=1.61-internal&$select=strongAuthenticationDetail,objectId
Error 404 for URL https://graph.windows.net/c8e8bdb6-8e7c-41d5-bc41-37fff3c33b75/users/d969e4a0-e34b-4dad-9a5a-d64ad15ddf7f?api-version=1.61-internal&$select=strongAuthenticationDetail,objectId
Error 404 for URL https://graph.windows.net/c8e8bdb6-8e7c-41d5-bc41-37fff3c33b75/users/de2d81a7-eb95-4e17-a578-fe9514b2044c?api-version=1.61-internal&$select=strongAuthenticationDetail,objectId
Error 404 for URL https://graph.windows.net/c8e8bdb6-8e7c-41d5-bc41-37fff3c33b75/users/e75d72a3-c726-4dea-af98-656930240cfb?api-version=1.61-internal&$select=strongAuthenticationDetail,objectId
Starting data gathering phase 3 of 3 (collecting group memberships and device owners)
Sleeping because of rate-limit hit3994/45451 devices
Sleeping because of rate-limit hit3994/45451 devices
Sleeping because of rate-limit hit 8932/45451 devices
Sleeping because of rate-limit hit 9012/45451 devices
Sleeping because of rate-limit hit 16048/45451 devices
Sleeping because of rate-limit hit 16048/45451 devices
Sleeping because of rate-limit hit 22852/45451 devices
Sleeping because of rate-limit hit 22863/45451 devices
Sleeping because of rate-limit hit 29595/45451 devices
Sleeping because of rate-limit hit 29683/45451 devices
[Errno 60] Operation timed outoups 31761/45451 devices
[Errno 60] Operation timed outoups 31785/45451 devices
[Errno 60] Operation timed outoups 31790/45451 devices
[Errno 60] Operation timed outoups 31821/45451 devices
[Errno 60] Operation timed outoups 33496/45451 devices
[Errno 60] Operation timed outoups 33669/45451 devices
Sleeping because of rate-limit hit 33997/45451 devices
[Errno 60] Operation timed outoups 34041/45451 devices
Sleeping because of rate-limit hit 34042/45451 devices
Cannot connect to host graph.windows.net:443 ssl:default [Too many open files] ---> 3523 times
Cannot connect to host graph.windows.net:443 ssl:default [nodename nor servname provided, or not known] ---> 2569 times
Cannot connect to host graph.windows.net:443 ssl:default [Too many open files] ---> 442 times
Sleeping because of rate-limit hit 45451/45451 devices
Sleeping because of rate-limit hit 45451/45451 devices
Cannot connect to host graph.windows.net:443 ssl:default [Too many open files] ---> 506 times
Sleeping because of rate-limit hit 45451/45451 devices
Sleeping because of rate-limit hit 45451/45451 devices
Sleeping because of rate-limit hit 45451/45451 devices
Sleeping because of rate-limit hit 45451/45451 devices
Cannot connect to host graph.windows.net:443 ssl:default [Too many open files] ---> 2844 times
Cannot connect to host graph.windows.net:443 ssl:default [nodename nor servname provided, or not known] ---> 1992 times
ROADrecon gather executed in 2249.50 seconds and issued 344768 HTTP requests.
Seems the data works will but it's opening too many sockets in the end. This could probably be upping the limits for max open files/sockets on your machine. Closing as there are some good improvements made to the gathering in large tenants and the errors above seem to be mostly OS and resource related.