xarray.backends.CachingFileManager

xarray.backends.CachingFileManager#

class xarray.backends.CachingFileManager(opener, *args, mode=<unused>, kwargs=None, lock=None, cache=None, manager_id=None, ref_counts=None)[source]#

Wrapper for automatically opening and closing file objects.

Unlike files, CachingFileManager objects can be safely pickled and passed between processes. They should be explicitly closed to release resources, but a per-process least-recently-used cache for open files ensures that you can safely create arbitrarily large numbers of FileManager objects.

Don’t directly close files acquired from a FileManager. Instead, call FileManager.close(), which ensures that closed files are removed from the cache as well.

Example usage:

manager = FileManager(open, "example.txt", mode="w")
f = manager.acquire()
f.write(...)
manager.close()  # ensures file is closed

Note that as long as previous files are still cached, acquiring a file multiple times from the same FileManager is essentially free:

f1 = manager.acquire()
f2 = manager.acquire()
assert f1 is f2
__init__(opener, *args, mode=<unused>, kwargs=None, lock=None, cache=None, manager_id=None, ref_counts=None)[source]#

Initialize a CachingFileManager.

The cache, manager_id and ref_counts arguments exist solely to facilitate dependency injection, and should only be set for tests.

Parameters
  • opener (callable()) – Function that when called like opener(*args, **kwargs) returns an open file object. The file object must implement a close() method.

  • *args – Positional arguments for opener. A mode argument should be provided as a keyword argument (see below). All arguments must be hashable.

  • mode (optional) – If provided, passed as a keyword argument to opener along with **kwargs. mode='w' `` has special treatment: after the first call it is replaced by ``mode='a' in all subsequent function to avoid overriding the newly created file.

  • kwargs (dict, optional) – Keyword arguments for opener, excluding mode. All values must be hashable.

  • lock (duck-compatible threading.Lock, optional) – Lock to use when modifying the cache inside acquire() and close(). By default, uses a new threading.Lock() object. If set, this object should be pickleable.

  • cache (MutableMapping, optional) – Mapping to use as a cache for open files. By default, uses xarray’s global LRU file cache. Because cache typically points to a global variable and contains non-picklable file objects, an unpickled FileManager objects will be restored with the default cache.

  • manager_id (hashable, optional) – Identifier for this CachingFileManager.

  • ref_counts (dict, optional) – Optional dict to use for keeping track the number of references to the same file.

Methods

__init__(opener, *args[, mode, kwargs, ...])

Initialize a CachingFileManager.

acquire([needs_lock])

Acquire a file object from the manager.

acquire_context([needs_lock])

Context manager for acquiring a file.

close([needs_lock])

Explicitly close any associated file object (if necessary).