Frequently Asked Questions
Contents
Frequently Asked Questions¶
Your documentation keeps mentioning pandas. What is pandas?¶
pandas is a very popular data analysis package in Python with wide usage in many fields. Our API is heavily inspired by pandas — this is why there are so many references to pandas.
Do I need to know pandas to use xarray?¶
No! Our API is heavily inspired by pandas so while knowing pandas will let you become productive more quickly, knowledge of pandas is not necessary to use xarray.
Should I use xarray instead of pandas?¶
It’s not an either/or choice! xarray provides robust support for converting back and forth between the tabular data-structures of pandas and its own multi-dimensional data-structures.
That said, you should only bother with xarray if some aspect of data is
fundamentally multi-dimensional. If your data is unstructured or
one-dimensional, pandas is usually the right choice: it has better performance
for common operations such as groupby
and you’ll find far more usage
examples online.
Why is pandas not enough?¶
pandas is a fantastic library for analysis of low-dimensional labelled data - if it can be sensibly described as “rows and columns”, pandas is probably the right choice. However, sometimes we want to use higher dimensional arrays (ndim > 2), or arrays for which the order of dimensions (e.g., columns vs rows) shouldn’t really matter. For example, the images of a movie can be natively represented as an array with four dimensions: time, row, column and color.
pandas has historically supported N-dimensional panels, but deprecated them in version 0.20 in favor of xarray data structures. There are now built-in methods on both sides to convert between pandas and xarray, allowing for more focused development effort. Xarray objects have a much richer model of dimensionality - if you were using Panels:
You need to create a new factory type for each dimensionality.
You can’t do math between NDPanels with different dimensionality.
Each dimension in a NDPanel has a name (e.g., ‘labels’, ‘items’, ‘major_axis’, etc.) but the dimension names refer to order, not their meaning. You can’t specify an operation as to be applied along the “time” axis.
You often have to manually convert collections of pandas arrays (Series, DataFrames, etc) to have the same number of dimensions. In contrast, this sort of data structure fits very naturally in an xarray
Dataset
.
You can read about switching from Panels to xarray here. pandas gets a lot of things right, but many science, engineering and complex analytics use cases need fully multi-dimensional data structures.
How do xarray data structures differ from those found in pandas?¶
The main distinguishing feature of xarray’s DataArray
over labeled arrays in
pandas is that dimensions can have names (e.g., “time”, “latitude”,
“longitude”). Names are much easier to keep track of than axis numbers, and
xarray uses dimension names for indexing, aggregation and broadcasting. Not only
can you write x.sel(time='2000-01-01')
and x.mean(dim='time')
, but
operations like x - x.mean(dim='time')
always work, no matter the order
of the “time” dimension. You never need to reshape arrays (e.g., with
np.newaxis
) to align them for arithmetic operations in xarray.
Why don’t aggregations return Python scalars?¶
Xarray tries hard to be self-consistent: operations on a DataArray
(resp.
Dataset
) return another DataArray
(resp. Dataset
) object. In
particular, operations returning scalar values (e.g. indexing or aggregations
like mean
or sum
applied to all axes) will also return xarray objects.
Unfortunately, this means we sometimes have to explicitly cast our results from xarray when using them in other libraries. As an illustration, the following code fragment
In [1]: arr = xr.DataArray([1, 2, 3])
In [2]: pd.Series({"x": arr[0], "mean": arr.mean(), "std": arr.std()})
Out[2]:
x <xarray.DataArray ()>\narray(1)
mean <xarray.DataArray ()>\narray(2.)
std <xarray.DataArray ()>\narray(0.81649658)
dtype: object
does not yield the pandas DataFrame we expected. We need to specify the type conversion ourselves:
In [3]: pd.Series({"x": arr[0], "mean": arr.mean(), "std": arr.std()}, dtype=float)
Out[3]:
x 1.000000
mean 2.000000
std 0.816497
dtype: float64
Alternatively, we could use the item
method or the float
constructor to
convert values one at a time
In [4]: pd.Series({"x": arr[0].item(), "mean": float(arr.mean())})
Out[4]:
x 1.0
mean 2.0
dtype: float64
What is your approach to metadata?¶
We are firm believers in the power of labeled data! In addition to dimensions
and coordinates, xarray supports arbitrary metadata in the form of global
(Dataset) and variable specific (DataArray) attributes (attrs
).
Automatic interpretation of labels is powerful but also reduces flexibility.
With xarray, we draw a firm line between labels that the library understands
(dims
and coords
) and labels for users and user code (attrs
). For
example, we do not automatically interpret and enforce units or CF
conventions. (An exception is serialization to and from netCDF files.)
An implication of this choice is that we do not propagate attrs
through
most operations unless explicitly flagged (some methods have a keep_attrs
option, and there is a global flag, accessible with xarray.set_options()
,
for setting this to be always True or False). Similarly, xarray does not check
for conflicts between attrs
when combining arrays and datasets, unless
explicitly requested with the option compat='identical'
. The guiding
principle is that metadata should not be allowed to get in the way.
What other projects leverage xarray?¶
See section Xarray related projects.
How should I cite xarray?¶
If you are using xarray and would like to cite it in academic publication, we would certainly appreciate it. We recommend two citations.
At a minimum, we recommend citing the xarray overview journal article, published in the Journal of Open Research Software.
Hoyer, S. & Hamman, J., (2017). xarray: N-D labeled Arrays and Datasets in Python. Journal of Open Research Software. 5(1), p.10. DOI: http://doi.org/10.5334/jors.148
Here’s an example of a BibTeX entry:
@article{hoyer2017xarray, title = {xarray: {N-D} labeled arrays and datasets in {Python}}, author = {Hoyer, S. and J. Hamman}, journal = {Journal of Open Research Software}, volume = {5}, number = {1}, year = {2017}, publisher = {Ubiquity Press}, doi = {10.5334/jors.148}, url = {http://doi.org/10.5334/jors.148} }You may also want to cite a specific version of the xarray package. We provide a Zenodo citation and DOI for this purpose:
What parts of xarray are considered public API?¶
As a rule, only functions/methods documented in our API reference are considered
part of xarray’s public API. Everything else (in particular, everything in
xarray.core
that is not also exposed in the top level xarray
namespace)
is considered a private implementation detail that may change at any time.
Objects that exist to facilitate xarray’s fluent interface on DataArray
and
Dataset
objects are a special case. For convenience, we document them in
the API docs, but only their methods and the DataArray
/Dataset
methods/properties to construct them (e.g., .plot()
, .groupby()
,
.str
) are considered public API. Constructors and other details of the
internal classes used to implemented them (i.e.,
xarray.plot.plotting._PlotMethods
, xarray.core.groupby.DataArrayGroupBy
,
xarray.core.accessor_str.StringAccessor
) are not.