pandasply: functional data manipulation for pandas¶
pandasply is a thin layer which makes it easier to manipulate data with pandas. In particular, it provides elegant, functional, chainable syntax in cases where pandas would require mutation, saved intermediate values, or other awkward constructions. In this way, it aims to move pandas closer to the “grammar of data manipulation” provided by the dplyr package for R.
For example, take the dplyr code below:
flights %>%
group_by(year, month, day) %>%
summarise(
arr = mean(arr_delay, na.rm = TRUE),
dep = mean(dep_delay, na.rm = TRUE)
) %>%
filter(arr > 30 & dep > 30)
The most common way to express this in pandas is probably:
grouped_flights = flights.groupby(['year', 'month', 'day'])
output = pd.DataFrame()
output['arr'] = grouped_flights.arr_delay.mean()
output['dep'] = grouped_flights.dep_delay.mean()
filtered_output = output[(output.arr > 30) & (output.dep > 30)]
pandasply lets you instead write:
(flights
.groupby(['year', 'month', 'day'])
.ply_select(
arr = X.arr_delay.mean(),
dep = X.dep_delay.mean())
.ply_where(X.arr > 30, X.dep > 30))
In our opinion, this pandasply code is cleaner, more expressive, more readable, more concise, and less errorprone than the original pandas code.
Explanatory notes on the pandasply code sample above:
 pandasply‘s methods (like
ply_select
andply_where
above) are attached directly to pandas objects and can be used immediately, without any wrapping or redirection. They start with aply_
prefix to distinguish them from builtin pandas methods.  pandasply‘s methods are named for (and modelled after) SQL’s operators. (But keep in mind that these operators will not always appear in the same order as they do in a SQL statement:
SELECT a FROM b WHERE c GROUP BY d
probably maps tob.ply_where(c).groupby(d).ply_select(a)
.)  pandasply includes a simple system for building “symbolic expressions” to provide as arguments to its methods.
X
above is an instance ofply.symbolic.Symbol
. Operations on this symbol produce larger compound symbolic expressions. Whenpandasply
receives a symbolic expression as an argument, it converts it into a function. So, for instance,X.arr > 30
in the above code could have instead been provided aslambda x: x.arr > 30
. Use of symbolic expressions allows thelambda x:
to be left off, resulting in less cluttered code.
Warning¶
pandasply is new, and in an experimental stage of its development. The API is not yet stable. Expect the unexpected.
(Pull requests are welcome. Feel free to contact us at pandasply@coursera.org.)
Using pandasply¶
Install pandasply with:
$ pip install pandasply
Typical use of pandasply starts with:
import pandas as pd
from pandas_ply import install_ply, X, sym_call
install_ply(pd)
After calling install_ply
, all pandas objects have pandasply‘s methods attached.
API reference¶
pandas extensions¶
This module contains the pandasply methods which are designed to be added to panda objects. The methods in this module should not be used directly. Instead, the function install_ply should be used to attach them to the pandas classes.

class
ply.methods.
PlyDataFrame
¶ The following methods are added to pandas.DataFrame:

ply_select
(*args, **kwargs)¶ Transform a dataframe by selecting old columns and new (computed) columns.
Analogous to SQL’s
SELECT
, or dplyr’sselect
/rename
/mutate
/transmute
.Parameters:  *args –
Each should be one of:
'*'
 says that all columns in the input dataframe should be included
'column_name'
 says that column_name in the input dataframe should be included
'column_name'
 says that column_name in the input dataframe should be excluded.
If any ‘column_name’ is present, then ‘*’ should be present, and if ‘*’ is present, no ‘column_name’ should be present. Columnincludes and columnexcludes should not overlap.
 **kwargs – Each argument name will be the name of a new column in the
output dataframe, with the column’s contents determined by the
argument contents. These contents can be given as a dataframe, a
function (taking the input dataframe as its single argument), or a
symbolic expression (taking the input dataframe as
Symbol(0)
). kwargprovided columns override argprovided columns.
Example
>>> flights.ply_select('*', ... gain = X.arr_delay  X.dep_delay, ... speed = X.distance / X.air_time * 60) [ original dataframe, with two new computed columns added ]
 *args –

ply_where
(*conditions)¶ Filter a dataframe/series to only include rows/entries satisfying a given set of conditions.
Analogous to SQL’s
WHERE
, or dplyr’sfilter
.Parameters: *conditions – Each should be a dataframe/series of booleans, a function returning such an object when run on the input dataframe, or a symbolic expression yielding such an object when evaluated with Symbol(0) mapped to the input dataframe. The input dataframe will be filtered by the AND of all the conditions. Example
>>> flights.ply_where(X.month == 1, X.day == 1) [ same result as `flights[(flights.month == 1) & (flights.day == 1)]` ]


class
ply.methods.
PlyDataFrameGroupBy
¶ The following methods are added to pandas.core.groupby.DataFrameGroupBy:

ply_select
(**kwargs)¶ Summarize a grouped dataframe or series.
Analogous to SQL’s
SELECT
(when aGROUP BY
is present), or dplyr’ssummarise
.Parameters: **kwargs – Each argument name will be the name of a new column in the output dataframe, with the column’s contents determined by the argument contents. These contents can be given as a dataframe, a function (taking the input grouped dataframe as its single argument), or a symbolic expression (taking the input grouped dataframe as Symbol(0)).


class
ply.methods.
PlySeries
¶ The following methods are added to pandas.Series:

ply_where
(*conditions)¶ Filter a dataframe/series to only include rows/entries satisfying a given set of conditions.
Analogous to SQL’s
WHERE
, or dplyr’sfilter
.Parameters: *conditions – Each should be a dataframe/series of booleans, a function returning such an object when run on the input dataframe, or a symbolic expression yielding such an object when evaluated with Symbol(0) mapped to the input dataframe. The input dataframe will be filtered by the AND of all the conditions. Example
>>> flights.ply_where(X.month == 1, X.day == 1) [ same result as `flights[(flights.month == 1) & (flights.day == 1)]` ]


class
ply.methods.
PlySeriesGroupBy
¶ The following methods are added to pandas.core.groupby.SeriesGroupBy:

ply_select
(**kwargs)¶ Summarize a grouped dataframe or series.
Analogous to SQL’s
SELECT
(when aGROUP BY
is present), or dplyr’ssummarise
.Parameters: **kwargs – Each argument name will be the name of a new column in the output dataframe, with the column’s contents determined by the argument contents. These contents can be given as a dataframe, a function (taking the input grouped dataframe as its single argument), or a symbolic expression (taking the input grouped dataframe as Symbol(0)).


ply.methods.
install_ply
(pandas_to_use)¶ Install pandasply onto the objects in a copy of pandas.
ply.symbolic¶
ply.symbolic is a simple system for building “symbolic expressions” to provide as arguments to pandasply‘s methods (in place of lambda expressions).

class
ply.symbolic.
Call
(func, args=[], kwargs={})¶ Bases:
ply.symbolic.Expression
Call(func, args, kwargs) is a symbolic expression representing the result of func(*args, **kwargs). (func, each member of the args iterable, and each value in the kwargs dictionary can themselves be symbolic).

_eval
(context, **options)¶


class
ply.symbolic.
Expression
¶ Bases:
object
Expression is the (abstract) base class for symbolic expressions. Symbolic expressions are encoded representations of Python expressions, kept on ice until you are ready to evaluate them. Operations on symbolic expressions (like my_expr.some_attr or my_expr(some_arg) or my_expr + 7) are automatically turned into symbolic representations thereof – nothing is actually done until the special evaluation method _eval is called.

__call__
(*args, **kwargs)¶ Construct a symbolic representation of self(*args, **kwargs).

__getattr__
(name)¶ Construct a symbolic representation of getattr(self, name).

_eval
(context, **options)¶ Evaluate a symbolic expression.
Parameters:  context – The context object for evaluation. Currently, this is a dictionary mapping symbol names to values,
 **options – Options for evaluation. Currently, the only option is log, which results in some debug output during evaluation if it is set to True.
Returns: anything


class
ply.symbolic.
GetAttr
(obj, name)¶ Bases:
ply.symbolic.Expression
GetAttr(obj, name) is a symbolic expression representing the result of getattr(obj, name). (obj and name can themselves be symbolic.)

_eval
(context, **options)¶


class
ply.symbolic.
Symbol
(name)¶ Bases:
ply.symbolic.Expression
Symbol(name) is an atomic symbolic expression, labelled with an arbitrary name.

_eval
(context, **options)¶


ply.symbolic.
X
¶ Symbol(name) is an atomic symbolic expression, labelled with an arbitrary name.

ply.symbolic.
_get_sym_magic_method
(name)¶

ply.symbolic.
eval_if_symbolic
(obj, context, **options)¶ Evaluate an object if it is a symbolic expression, or otherwise just returns it back.
Parameters:  obj – Either a symbolic expression, or anything else (in which case this is a noop).
 context – Passed as an argument to obj._eval if obj is symbolic.
 **options – Passed as arguments to obj._eval if obj is symbolic.
Returns: anything
Examples
>>> eval_if_symbolic(Symbol('x'), {'x': 10}) 10 >>> eval_if_symbolic(7, {'x': 10}) 7

ply.symbolic.
sym_call
(func, *args, **kwargs)¶ Construct a symbolic representation of func(*args, **kwargs).
This is necessary because func(symbolic) will not (ordinarily) know to construct a symbolic expression when it receives the symbolic expression symbolic as a parameter (if func is not itself symbolic). So instead, we write sym_call(func, symbolic).
Tip: If the main argument of the function is a (symbolic) DataFrame, then pandas’ pipe method takes care of this problem without sym_call. For instance, while np.sqrt(X) won’t work, X.pipe(np.sqrt) will.
Parameters:  func – Function to call on evaluation (can be symbolic).
 *args – Arguments to provide to func on evaluation (can be symbolic).
 **kwargs – Keyword arguments to provide to func on evaluation (can be symbolic).
Returns: ply.symbolic.Expression
Example
>>> sym_call(math.sqrt, Symbol('x'))._eval({'x': 16}) 4

ply.symbolic.
to_callable
(obj)¶ Turn an object into a callable.
Parameters: obj – This can be
 a symbolic expression, in which case the output callable evaluates the expression with symbols taking values from the callable’s arguments (listed arguments named according to their numerical index, keyword arguments named according to their string keys),
 a callable, in which case the output callable is just the input object, or
 anything else, in which case the output callable is a constant function which always returns the input object.
Returns: callable Examples
>>> to_callable(Symbol(0) + Symbol('x'))(3, x=4) 7 >>> to_callable(lambda x: x + 1)(10) 11 >>> to_callable(12)(3, x=4) 12