Unable to list files with service account auth for GDriveFilesystem
chandrasekharan-zipstack opened this issue · comments
Hello, I'm using the latest version of the library PyDrive2[fsspec]==1.17.0
and I'm trying to authenticate a service account (the SA email is shared with the drive I wish to test and the keys were generated, I doubt if this order matters).
I followed the steps provided in the docs for the jsonkey file as a string. Its authenticated without errors however I'm unable to list files from the shared drive
Digging further, I followed steps mentioned in this comment and was able to list files with the GDrive
object however. In fact I noticed the GDrive
object being created here and the files get listed (while debugging) when I used
file_list = self.client.ListFile().GetList()
Code snippet:
from oauth2client.service_account import ServiceAccountCredentials
from pydrive2.auth import GoogleAuth
from pydrive2.fs import GDriveFileSystem
from pydrive2.drive import GoogleDrive
def _init_with_service_key(self, service_key_settings: str) -> GDriveFileSystem:
"""Creates a GDriveFileSystem with a service account key."""
# Attempt from issue #21
# scope = ["https://www.googleapis.com/auth/drive"]
# gauth = GoogleAuth()
# gauth.auth_method = 'service'
# gauth.credentials = ServiceAccountCredentials.from_json_keyfile_dict(service_key_settings, scopes=scope)
# drive = GoogleDrive(gauth)
# about = drive.GetAbout()
# file_list = drive.ListFile().GetList()
# gdrive = GDriveFileSystem(
# path="root",
# google_auth=gauth,
# use_service_account=True
# )
gdrive = GDriveFileSystem(
path="root",
use_service_account=True,
client_json=service_key_settings,
skip_instance_cache=True,
use_listings_cache=False
)
return gdrive
Any ideas on how I can proceed? Thanks in advance! :)
@chandrasekharan-zipstack it might be a bug in the fsspec implementation. It was done to address needs of DVC as product in the first place, and I think is has a lot of edge cases that are not solved to be used in general. I would recommend to use the older API for now.
I would recommend to use the older API for now.
@shcheklein thanks for the information, I did go through the docs a bit but can you elaborate on what you mean be the older API here? Is it related to using the GoogleDrive
object mentioned here?
Also it might be a good idea to mention such limitations/edge cases in the docs
Yes, I meant this for example: https://docs.iterative.ai/PyDrive2/filelist/