TypeError: ClientSession._request() got an unexpected keyword argument 's3'

[codingl2k1]

I have a list of URLs that are mixed with different schemes, some in the format s3:// and others in https://. I want to use one global storage option to read them all. However, the HTTP backend is attempting to use the S3 storage option.

Different protocol backends should only use their corresponding protocol storage options.

import fsspec

storage_options = {
    "s3": {
        "key": "My key",
        "secret": "My secret",
        "use_ssl": True,
    },
}

for url in url_list:
    with fsspec.open(url, "rb", **storage_options) as f:
        data = f.read()
        print(data)

error:

    def get(
        self, url: StrOrURL, *, allow_redirects: bool = True, **kwargs: Any
    ) -> "_RequestContextManager":
        """Perform HTTP GET request."""
        return _RequestContextManager(
>           self._request(
                hdrs.METH_GET, url, allow_redirects=allow_redirects, **kwargs
            )
        )
E       TypeError: ClientSession._request() got an unexpected keyword argument 's3'

[martindurant]

Different protocol backends should only use their corresponding protocol storage options.

No, this is not the interface. The only time you will see that, is when passing compound URLs, which contain multiple protocols. Your expected behaviour maybe makes sense, except that there is no way to know the set of protocols that might be available.

You might want to use the fsspec.config.conf dictionary to set default values for your purpose.

[codingl2k1]

Different protocol backends should only use their corresponding protocol storage options.

No, this is not the interface. The only time you will see that, is when passing compound URLs, which contain multiple protocols. Your expected behaviour maybe makes sense, except that there is no way to know the set of protocols that might be available.

You might want to use the fsspec.config.conf dictionary to set default values for your purpose.

Thanks. It would be good if the open API could process compound URLs with multiple protocols. I wrapped a new API myself that retrieves the storage options by protocol from the kwargs, it works well but introduces a breaking change.

[martindurant]

It would be good if the open API could process compound URLs with multiple protocols.

It can! IF you have a compound url, then it will search for the names of the protocols in the kwargs and grab them for the respective constructor. Here the question is, what to do with a protocol kwarg that is not int he URL at all, how can we know that?

[codingl2k1]

It would be good if the open API could process compound URLs with multiple protocols.

It can! IF you have a compound url, then it will search for the names of the protocols in the kwargs and grab them for the respective constructor. Here the question is, what to do with a protocol kwarg that is not int he URL at all, how can we know that?

My simple wrapper works well for me. I am just passing a compound storage option to my open:

def open(
    urlpath,
    mode="rb",
    compression=None,
    encoding="utf8",
    errors=None,
    protocol=None,
    newline=None,
    expand=None,
    **storage_options,
) -> fsspec.core.OpenFile:
    """Auto select storage options based on protocol."""
    if isinstance(urlpath, (list, tuple, set)):
        if not urlpath:
            raise ValueError("empty urlpath sequence")
        urlpath0 = fsspec.core.stringify_path(next(iter(urlpath)))
    else:
        urlpath0 = fsspec.core.stringify_path(urlpath)

    selected_storage_options = {}
    for bit, protocol, kw in fsspec.core._un_chain(urlpath0, {}):
        so = storage_options.get(protocol)
        if so:
            selected_storage_options[protocol] = so

    return fsspec.open(
        urlpath,
        mode=mode,
        compression=compression,
        encoding=encoding,
        errors=errors,
        protocol=protocol,
        newline=newline,
        expand=expand,
        **selected_storage_options,
    )

[martindurant]

OK, but that means that you must use exclusively s3=, http= and so on, but no top-level kwargs. I suppose it makes sense.

[codingl2k1]

OK, but that means that you must use exclusively s3=, http= and so on, but no top-level kwargs. I suppose it makes sense.

This works for me, but it breaks the original API behavior. Thanks for your help.