# data[].attributes.headline
Making pivot tables with pandas dataframes

# data[].attributes.subject
Making pivot tables with pandas dataframes

# data[].attributes.attachment_link.readable_text
Introduction

Most people likely have experience with pivot tables in Excel. Pandas provides a similar function called (appropriately enough) pivot_table . While it is exceedingly useful, I frequently find myself struggling to remember how to use the syntax to format the output for my needs. This article will focus on explaining the pandas pivot_table function and how to use it for your data analysis.

If you are not familiar with the concept, wikipedia explains it in high level terms. BTW, did you know that Microsoft trademarked PivotTable? Neither did I. Needless to say, I'll be talking about a pivot table not PivotTable!

As an added bonus, I've created a simple cheat sheet that summarizes the pivot_table. You can find it at the end of this post and I hope it serves as a useful reference. Let me know if it is helpful.

The Data

One of the challenges with using the panda's pivot_table is making sure you understand your data and what questions you are trying to answer with the pivot table. It is a seemingly simple function but can produce very powerful analysis very quickly.

In this scenario, I'm going to be tracking a sales pipeline (also called funnel). The basic problem is that some sales cycles are very long (think "enterprise software", capital equipment, etc.) and management wants to understand it in more detail throughout the year.

Typical questions include:

How much revenue is in the pipeline?

What products are in the pipeline?

Who has what products at what stage?

How likely are we to close deals by year end?

Many companies will have CRM tools or other software that sales uses to track the process. While they may have useful tools for analyzing the data, inevitably someone will export the data to Excel and use a PivotTable to summarize the data.

Using a panda's pivot table can be a good alternative because it is:

Quicker (once it is set up)

Self documenting (look at the code and you know what it does)

Easy to use to generate a report or email

More flexible because you can define custome aggregation functions

Read in the data

Let's set up our environment first.

If you want to follow along, you can download the Excel file.

Read in our sales funnel data into our DataFrame

For convenience sake, let's define the status column as a category and set the order we want to view.

This isn't strictly required but helps us keep the order we want as we work through analyzing the data.

Pivot the data

As we build up the pivot table, I think it's easiest to take it one step at a time. Add items and check each step to verify you are getting the results you expect. Don't be afraid to play with the order and the variables to see what presentation makes the most sense for your needs.

The simplest pivot table must have a dataframe and an index . In this case, let's use the Name as our index.

You can have multiple indexes as well. In fact, most of the pivot_table args can take multiple values via a list.

This is interesting but not particularly useful. What we probably want to do is look at this by Manager and Rep. It's easy enough to do by changing the index .

You can see that the pivot table is smart enough to start aggregating the data and summarizing it by grouping the reps with their managers. Now we start to get a glimpse of what a pivot table can do for us.

For this purpose, the Account and Quantity columns aren't really useful. Let's remove it by explicitly defining the columns we care about using the values field.

The price column automatically averages the data but we can do a count or a sum. Adding them is simple using aggfunc and np.sum .

aggfunc can take a list of functions. Let's try a mean using the numpy mean function and len to get a count.

If we want to see sales broken down by the products, the columns variable allows us to define one or more columns.

The NaN's are a bit distracting. If we want to remove them, we could use fill_value to set them to 0.

I think it would be useful to add the quantity as well. Add Quantity to the values list.

What's interesting is that you can move items to the index to get a different visual representation. Remove Product from the columns and add to the index .

For this data set, this representation makes more sense. Now, what if I want to see some totals? margins=True does that for us.

Let's move the analysis up a level and look at our pipeline at the manager level. Notice how the status is ordered based on our earlier category definition.

A really handy feature is the ability to pass a dictionary to the aggfunc so you can perform different functions on each of the values you select. This has a side-effect of making the labels a little cleaner.

You can provide a list of aggfunctions to apply to each value too:

It can look daunting to try to pull this all together at one time but as soon as you start playing with the data and slowly add the items, you can get a feel for how it works. My general rule of thumb is that once you use multiple grouby you should evaluate whether a pivot table is a useful approach.

Advanced Pivot Table Filtering

Once you have generated your data, it is in a DataFrame so you can filter on it using your standard DataFrame functions.

If you want to look at just one manager:

We can look at all of our pending and won deals.

This a poweful feature of the pivot_table so do not forget that you have the full power of pandas once you get your data into the pivot_table format you need.

The full notebook is available if you would like to save it as a reference.

Cheat Sheet

In order to try to summarize all of this, I have created a cheat sheet that I hope will help you remember how to use the pandas pivot_table . Take a look and let me know what you think.

Thanks and good luck with creating your own pivot tables.

# data[].attributes.attachment_link.description
Most people likely have experience with pivot tables in Excel. Pandas provides a similar function called (appropriately enough) pivot_table . While it is exceedingly useful, I frequently find myself struggling to remember how to use the syntax to format the output for my needs.

# data[].attributes.attachment_link.readable_text
An essential piece of analysis of large data is efficient summarization: computing aggregations like sum(), mean(), median(), min(), and max(), in which a single number gives insight into the nature of a potentially large dataset. In this section, we'll explore aggregations in Pandas, from simple operations akin to what we've seen on NumPy arrays, to more sophisticated operations based on the concept of a groupby.

Planets Data¶

Here we will use the Planets dataset, available via the Seaborn package (see Visualization With Seaborn). It gives information on planets that astronomers have discovered around other stars (known as extrasolar planets or exoplanets for short). It can be downloaded with a simple Seaborn command:

This has some details on the 1,000+ extrasolar planets discovered up to 2014.

Earlier, we explored some of the data aggregations available for NumPy arrays ("Aggregations: Min, Max, and Everything In Between"). As with a one-dimensional NumPy array, for a Pandas Series the aggregates return a single value:

Pandas Series and DataFrames include all of the common aggregates mentioned in Aggregations: Min, Max, and Everything In Between; in addition, there is a convenience method describe() that computes several common aggregates for each column and returns the result. Let's use this on the Planets data, for now dropping rows with missing values:

This can be a useful way to begin understanding the overall properties of a dataset. For example, we see in the year column that although exoplanets were discovered as far back as 1989, half of all known expolanets were not discovered until 2010 or after. This is largely thanks to the Kepler mission, which is a space-based telescope specifically designed for finding eclipsing planets around other stars.

GroupBy: Split, Apply, Combine¶

Simple aggregations can give you a flavor of your dataset, but often we would prefer to aggregate conditionally on some label or index: this is implemented in the so-called groupby operation. The name "group by" comes from a command in the SQL database language, but it is perhaps more illuminative to think of it in the terms first coined by Hadley Wickham of Rstats fame: split, apply, combine.

Split, apply, combine¶

A canonical example of this split-apply-combine operation, where the "apply" is a summation aggregation, is illustrated in this figure:

This makes clear what the groupby accomplishes:

The split step involves breaking up and grouping a DataFrame depending on the value of the specified key.

The apply step involves computing some function, usually an aggregate, transformation, or filtering, within the individual groups.

The combine step merges the results of these operations into an output array.

While this could certainly be done manually using some combination of the masking, aggregation, and merging commands covered earlier, an important realization is that the intermediate splits do not need to be explicitly instantiated. Rather, the GroupBy can (often) do this in a single pass over the data, updating the sum, mean, count, min, or other aggregate for each group along the way. The power of the GroupBy is that it abstracts away these steps: the user need not think about how the computation is done under the hood, but rather thinks about the operation as a whole.

As a concrete example, let's take a look at using Pandas for the computation shown in this diagram. We'll start by creating the input DataFrame:

The most basic split-apply-combine operation can be computed with the groupby() method of DataFrame s, passing the name of the desired key column:

Notice that what is returned is not a set of DataFrames, but a DataFrameGroupBy object. This object is where the magic is: you can think of it as a special view of the DataFrame, which is poised to dig into the groups but does no actual computation until the aggregation is applied. This "lazy evaluation" approach means that common aggregates can be implemented very efficiently in a way that is almost transparent to the user.

To produce a result, we can apply an aggregate to this DataFrameGroupBy object, which will perform the appropriate apply/combine steps to produce the desired result:

The sum() method is just one possibility here; you can apply virtually any common Pandas or NumPy aggregation function, as well as virtually any valid DataFrame operation, as we will see in the following discussion.

The GroupBy object¶

The GroupBy object is a very flexible abstraction. In many ways, you can simply treat it as if it's a collection of DataFrame s, and it does the difficult things under the hood. Let's see some examples using the Planets data.

Perhaps the most important operations made available by a GroupBy are aggregate, filter, transform, and apply. We'll discuss each of these more fully in"Aggregate, Filter, Transform, Apply", but before that let's introduce some of the other functionality that can be used with the basic GroupBy operation.

Column indexing¶

The GroupBy object supports column indexing in the same way as the DataFrame, and returns a modified GroupBy object. For example:

Here we've selected a particular Series group from the original DataFrame group by reference to its column name. As with the GroupBy object, no computation is done until we call some aggregate on the object:

This gives an idea of the general scale of orbital periods (in days) that each method is sensitive to.

Iteration over groups¶

The GroupBy object supports direct iteration over the groups, returning each group as a Series or DataFrame:

Dispatch methods¶

Through some Python class magic, any method not explicitly implemented by the GroupBy object will be passed through and called on the groups, whether they are DataFrame or Series objects. For example, you can use the describe() method of DataFrame s to perform a set of aggregations that describe each group in the data:

Looking at this table helps us to better understand the data: for example, the vast majority of planets have been discovered by the Radial Velocity and Transit methods, though the latter only became common (due to new, more accurate telescopes) in the last decade. The newest methods seem to be Transit Timing Variation and Orbital Brightness Modulation, which were not used to discover a new planet until 2011.

This is just one example of the utility of dispatch methods. Notice that they are applied to each individual group, and the results are then combined within GroupBy and returned. Again, any valid DataFrame/ Series method can be used on the corresponding GroupBy object, which allows for some very flexible and powerful operations!

Aggregate, filter, transform, apply¶

The preceding discussion focused on aggregation for the combine operation, but there are more options available. In particular, GroupBy objects have aggregate(), filter(), transform(), and apply() methods that efficiently implement a variety of useful operations before combining the grouped data.

For the purpose of the following subsections, we'll use this DataFrame:

Aggregation¶

We're now familiar with GroupBy aggregations with sum(), median(), and the like, but the aggregate() method allows for even more flexibility. It can take a string, a function, or a list thereof, and compute all the aggregates at once. Here is a quick example combining all these:

Another useful pattern is to pass a dictionary mapping column names to operations to be applied on that column:

The filter function should return a Boolean value specifying whether the group passes the filtering. Here because group A does not have a standard deviation greater than 4, it is dropped from the result.

The apply() method¶

The apply() method lets you apply an arbitrary function to the group results. The function should take a DataFrame, and return either a Pandas object (e.g., DataFrame, Series) or a scalar; the combine operation will be tailored to the type of output returned.

For example, here is an apply() that normalizes the first column by the sum of the second:

apply() within a GroupBy is quite flexible: the only criterion is that the function takes a DataFrame and returns a Pandas object or scalar; what you do in the middle is up to you!

Specifying the split key¶

In the simple examples presented before, we split the DataFrame on a single column name. This is just one of many options by which the groups can be defined, and we'll go through some other options for group specification here.

A list, array, series, or index providing the grouping keys¶

The key can be any series or list with a length matching that of the DataFrame. For example:

Grouping example¶

As an example of this, in a couple lines of Python code we can put all these together and count discovered planets by method and by decade:

This shows the power of combining many of the operations we've discussed up to this point when looking at realistic datasets. We immediately gain a coarse understanding of when and how planets have been discovered over the past several decades!

Here I would suggest digging into these few lines of code, and evaluating the individual steps to make sure you understand exactly what they are doing to the result. It's certainly a somewhat complicated example, but understanding these pieces will give you the means to similarly explore your own data.

# data[].attributes.attachment_link.description
An essential piece of analysis of large data is efficient summarization: computing aggregations like sum(), mean(), median(), min(), and max(), in which a single number gives insight into the nature of a potentially large dataset. In this section, we'll explore aggregations in Pandas, from simple operations akin to what we've seen on NumPy arrays, to more sophisticated operations based on the concept of a groupby.

# data[].attributes.attachment_link.title
Overview of Pandas Data Types - Practical Business Python

# data[].attributes.attachment_link.description
Introduction to pandas data types and how to convert data columns to correct dtypes.

# data[].attributes.attachment_link.description
itertools.accumulate(iterable[, func]) This function makes an iterator that returns the results of a function. Functions can be passed around very much like variables. The accumulate() function takes a function as an argument. It also takes an iterable. It returns the accumulated results. The results are themselves contained in an iterable.

# data[].attributes.attachment_link.readable_text
It has been called a "gem" and "pretty much the coolest thing ever," and if you have not heard of it, then you are missing out on one of the greatest corners of the Python 3 standard library: itertools.

A handful of excellent resources exist for learning what functions are available in the itertools module. The docs themselves are a great place to start. So is this post.

The thing about itertools, though, is that it is not enough to just know the definitions of the functions it contains. The real power lies in composing these functions to create fast, memory-efficient, and good-looking code.

This article takes a different approach. Rather than introducing itertools to you one function at a time, you will construct practical examples designed to encourage you to "think iteratively." In general, the examples will start simple and gradually increase in complexity.

A word of warning: this article is long and intended for the intermediate-to-advanced Python programmer. Before diving in, you should be confident using iterators and generators in Python 3, multiple assignment, and tuple unpacking. If you aren't, or if you need to brush up on your knowledge, consider checking out the following before reading on:

All set? Let's start the way any good journey should-with a question.

What Is Itertools and Why Should You Use It?

According to the itertools docs, it is a "module [that] implements a number of iterator building blocks inspired by constructs from APL, Haskell, and SML... Together, they form an 'iterator algebra' making it possible to construct specialized tools succinctly and efficiently in pure Python."

Loosely speaking, this means that the functions in itertools "operate" on iterators to produce more complex iterators. Consider, for example, the built-in zip() function, which takes any number of iterables as arguments and returns an iterator over tuples of their corresponding elements:

How, exactly, does zip() work?

[1, 2, 3] and ['a', 'b', 'c'], like all lists, are iterable, which means they can return their elements one at a time. Technically, any Python object that implements the .__iter__() or .__getitem__() methods is iterable. (See the Python 3 docs glossary for a more detailed explanation.)

The iter() built-in function, when called on an iterable, returns an iterator object for that iterable:

The map() built-in function is another "iterator operator" that, in its simplest form, applies a single-parameter function to each element of an iterable one element at a time:

Since iterators are iterable, you can compose zip() and map() to produce an iterator over combinations of elements in more than one iterable. For example, the following sums corresponding elements of two lists:

This is what is meant by the functions in itertools forming an "iterator algebra." itertools is best viewed as a collection of building blocks that can be combined to form specialized "data pipelines" like the one in the example above.

Historical Note: In Python 2, the built-in and functions do not return an iterator, but rather a list. To return an iterator, the and functions of itertools must be used. In Python 3, izip() and imap() have been removed from itertools and replaced the zip() and map() built-ins. So, in a way, if you have ever used zip() or map() in Python 3, you have already been using itertools!

There are two main reasons why such an "iterator algebra" is useful: improved memory efficiency (via lazy evaluation) and faster execuction time. To see this, consider the following problem:

Given a list of values inputs and a positive integer n, write a function that splits inputs into groups of length n. For simplicity, assume that the length of the input list is divisible by n. For example, if inputs = [1, 2, 3, 4, 5, 6] and n = 2, your function should return [(1, 2), (3, 4), (5, 6)].

Taking a naive approach, you might write something like this:

When you test it, you see that it works as expected:

What happens when you try to pass it a list with, say, 100 million elements? You will need a whole lot of available memory! Even if you have enough memory available, your program will hang for a while until the output list is populated.

To see this, store the following in a script called naive.py:

From the console, you can use the time command (on UNIX systems) to measure memory usage and CPU user time. Make sure you have at least 5GB of free memory before executing the following:

$ time -f "Memory used (kB): %M\nUser time (seconds): %U" python3 naive.py
Memory used (kB): 4551872
User time (seconds): 11.04

Note: On Ubuntu, you may need to run /usr/bin/time instead of time for the above example to work.

There's a lot going on in this little function, so let's break it down with a concrete example. The expression [iters(inputs)] * n creates a list of n references to the same iterator:

At this point, "both" iterators in iters start at 3, so when zip() pulls 3 from the "first" iterator, it gets 4 from the "second" to produce the tuple (3, 4). This process continues until zip() finally produces (9, 10) and "both" iterators in iters are exhausted:

Store the following in a file called better.py and run it with time from the console again:

$ time -f "Memory used (kB): %M\nUser time (seconds): %U" python3 better.py
Memory used (kB): 7224
User time (seconds): 2.48

That's a whopping 630 times less memory used than naive.py in less than a quarter of the time!

The grouper Recipe

The problem with better_grouper() is that it doesn't handle situations where the value passed to the second argument isn't a factor of the length of the iterable in the first argument:

The elements 9 and 10 are missing from the grouped output. This happens because zip() stops aggregating elements once the shortest iterable passed to it is exhausted. It would make more sense to return a third group containing 9 and 10.

Now you get a better result:

The grouper() function can be found in the Recipes section of the itertools docs. The recipes are an excellent source of inspiration for ways to use itertools to your advantage.

Note: From this point forward, the line import itertools as it will not be included at the beginning of examples. All itertools methods in code examples are prefaced with it. The module import is implied.

If you get a NameError: name 'itertools' is not defined or a NameError: name 'it' is not defined exception when running one of the examples in this tutorial you'll need to import the itertools module first.

Et tu, Brute Force?

Here's a common interview-style problem:

You have three $20 dollar bills, five $10 dollar bills, two $5 dollar bills, and five $1 dollar bills. How many ways can you make change for a $100 dollar bill?

To "brute force" this problem, you just start listing off the ways there are to choose one bill from your wallet, check whether any of these makes change for $100, then list the ways to pick two bills from your wallet, check again, and so on and so forth.

But you are a programmer, so naturally you want to automate this process.

First, create a list of the bills you have in your wallet:

A choice of k things from a set of n things is called a combination, and itertools has your back here. The itertools.combinations() function takes two arguments-an iterable inputs and a positive integer n-and produces an iterator over tuples of all combinations of n elements in inputs.

For example, to list the combinations of three bills in your wallet, just do:

To solve the problem, you can loop over the positive integers from 1 to len(bills), then check which combinations of each size add up to $100:

If you print out makes_100, you will notice there are a lot of repeated combinations. This makes sense because you can make change for $100 with three $20 dollar bills and four $10 bills, but combinations() does this with the first four $10 dollars bills in your wallet; the first, third, fourth and fifth $10 dollar bills; the first, second, fourth and fifth $10 bills; and so on.

To remove duplicates from makes_100, you can convert it to a set:

So, there are five ways to make change for a $100 bill with the bills you have in your wallet.

Here's a variation on the same problem:

How many ways are there to make change for a $100 bill using any number of $50, $20, $10, $5, and $1 dollar bills?

In this case, you don't have a pre-set collection of bills, so you need a way to generate all possible combinations using any number of bills. For this, you'll need the itertools.combinations_with_replacement() function.

For example:

Compare that to combinations():

Here's what the solution to the revised problem looks like:

In this case, you do not need to remove any duplicates since combinations_with_replacement() won't produce any:

If you run the above solution, you may notice that it takes a while for the output to display. That is because it has to process 96,560,645 combinations!

Another "brute force" itertools function is permutations(), which accepts a single iterable and produces all possible permutations (rearrangements) of its elements:

Any iterable of three elements will have six permutations, and the number of permutations of longer iterables grows extremely fast. In fact, an iterable of length n has n! permutations, where

To put this in perspective, here's a table of these numbers for n = 1 to n = 10:

The phenomenon of just a few inputs producing a large number of outcomes is called a combinatorial explosion and is something to keep in mind when working with combinations(), combinations_with_replacement(), and permutations().

It is usually best to avoid brute force algorithms, although there are times you may need to use one (for example, if the correctness of the algorithm is critical, or every possible outcome must be considered). In that case, itertools has you covered.

Section Recap

Let's review these functions before moving on:

itertools.combinations Example

itertools.combinations_with_replacement Example

itertools.permutations Example

Sequences of Numbers

With itertools, you can easily generate iterators over infinite sequences. In this section, you will explore numeric sequences, but the tools and techniques seen here are by no means limited to numbers.

Evens and Odds

For the first example, you will create a pair of iterators over even and odd integers without explicitly doing any arithmetic. Before diving in, let's look at an arithmetic solution using generators:

That is pretty straightforward, but with itertools you can do this much more compactly. The function you need is itertools.count(), which does exactly what it sounds like: it counts, starting by default with the number 0.

With count(), iterators over even and odd integers become literal one-liners:

Ever since Python 3.1, the count() function also accepts non-integer arguments:

You can even pass it negative numbers:

The example that made me realize the power of the infinite iterator was the following, which emulates the behavior of the built-in enumerate() function:

It is a simple example, but think about it: you just enumerated a list without a for loop and without knowing the length of the list ahead of time.

Recurrence Relations

A recurrence relation is a way of describing a sequence of numbers with a recursive formula. One of the best-known recurrence relations is the one that describes the Fibonacci sequence.

The Fibonacci sequence is the sequence 0, 1, 1, 2, 3, 5, 8, 13, .... It starts with 0 and 1, and each subsequent number in the sequence is the sum of the previous two. The numbers in this sequence are called the Fibonacci numbers. In mathematical notation, the recurrence relation describing the n-th Fibonacci number looks like this:

Note: If you search Google, you will find a host of implementations of these numbers in Python. You can find a recursive function that produces them in the Thinking Recursively in Python article here on Real Python.

It is common to see the Fibonacci sequence produced with a generator:

The recurrence relation describing the Fibonacci numbers is called a second order recurrence relation because, to calculate the next number in the sequence, you need to look back two numbers behind it.

In general, second order recurrence relations have the form:

Here, P, Q, and R are constants. To generate the sequence, you need two initial values. For the Fibonacci numbers, P = Q = 1, R = 0, and the initial values are 0 and 1.

As you might guess, a first order recurrence relation has the following form:

There are countless sequences of numbers that can be described by first and second order recurrence relations. For example, the positive integers can be described as a first order recurrence relation with P = Q = 1 and initial value 1. For the even integers, take P = 1 and Q = 2 with initial value 0.

In this section, you will construct functions for producing any sequence whose values can be described with a first or second order recurrence relation.

First Order Recurrence Relations

You've already seen how count() can generate the sequence of non-negative integers, the even integers, and the odd integers. You can also use it to generate the sequence 3n = 0, 3, 6, 9, 12, ... and 4n = 0, 4, 8, 12, 16, ....

In fact, count() can produce sequences of multiples of any number you wish. These sequences can be described with first-order recurrence relations. For example, to generate the sequence of multiples of some number n, just take P = 1, Q = n, and initial value 0.

Another easy example of a first-order recurrence relation is the constant sequence n, n, n, n, n..., where n is any value you'd like. For this sequence, set P = 1 and Q = 0 with initial value n. itertools provides an easy way to implement this sequence as well, with the repeat() function:

If you need a finite sequence of repeated values, you can set a stopping point by passing a positive integer as a second argument:

What may not be quite as obvious is that the sequence 1, -1, 1, -1, 1, -1, ... of alternating 1s and -1s can also be described by a first order recurrence relation. Just take P = -1, Q = 0, and initial value 1.

The goal of this section, though, is to produce a single function that can generate any first order recurrence relation-just pass it P, Q, and an initial value. One way to do this is with itertools.accumulate().

The accumulate() function takes two arguments-an iterable inputs and a binary function func (that is, a function with exactly two inputs)-and returns an iterator over accumulated results of applying func to elements of inputs. It is roughly equivalent to the following generator:

For example:

The first value in the iterator returned by accumulate() is always the first value in the input sequence. In the above example, this is 1-the first value in [1, 2, 3, 4, 5].

add(3, 3) = add(add(1, 2), 3) = 6

The fourth value produced by accumulate() is add(add(add(1, 2), 3), 4) = 10, and so on.

The second argument of accumulate() defaults to operator.add(), so the previous example can be simplified to:

Passing the built-in min() to accumulate() will keep track of a running minimum:

More complex functions can be passed to accumulate() with lambda expressions:

The order of the arguments in the binary function passed to accumulate() is important. The first argument is always the previously accumulated result and the second argument is always the next element of the input iterable. For example, consider the difference in output of the following expressions:

To model a recurrence relation, you can just ignore the second argument of the binary function passed to accumulate(). That is, given values p, q, and s, lambda x, _: p*s + q will return the value following x in the recurrence relation defined by sᵢ = Psᵢ₋₁ + Q.

In order for accumulate() to iterate over the resulting recurrence relation, you need to pass to it an infinite sequence with the right initial value. It doesn't matter what the rest of the values in the sequence are, as long as the initial value is the initial value of the recurrence relation. You can do this is with repeat():

Using first_order(), you can build the sequences from above as follows:

Second Order Recurrence Relations

Generating sequences described by second order recurrence relations, like the Fibonacci sequence, can be accomplished using a similar technique as the one used for first order recurrence relations.

The difference here is that you need to create an intermediate sequence of tuples that keep track of the previous two elements of the sequence, and then map() each of these tuples to their first component to get the final sequence.

Here's what it looks like:

Using second_order(), you can generate the Fibonacci sequence like this:

Other sequences can be easily generated by changing the values of p, q, and r. For example, the Pell numbers and the Lucas numbers can be generated as follows:

You can even generate the alternating Fibonacci numbers:

This is all really cool if you are a giant math nerd like I am, but step back for a second and compare second_order() to the fibs() generator from the beginning of this section. Which one is easier to understand?

This is a valuable lesson. The accumulate() function is a powerful tool to have in your toolkit, but there are times when using it could mean sacrificing clarity and readability.

Section Recap

You saw several itertools function in this section. Let's review those now.

itertools.count Example

count(start=0, step=1)
Return a count object whose .__next__() method returns consecutive values.

itertools.repeat Example

itertools.cycle Example

itertools accumulate Example

Alright, let's take a break from the math and have some fun with cards.

Dealing a Deck of Cards

Suppose you are building a Poker app. You'll need a deck of cards. You might start by defining a list of ranks (ace, king, queen, jack, 10, 9, and so on) and a list of suits (hearts, diamonds, clubs, and spades):

You could represent a card as a tuple whose first element is a rank and second element is a suit. A deck of cards would be a collection of such tuples. The deck should act like the real thing, so it makes sense to define a generator that yields cards one at a time and becomes exhausted once all the cards are dealt.

You could write this more compactly with a generator expression:

However, some might argue that this is actually more difficult to understand than the more explicit nested for loop.

It helps to view nested for loops from a mathematical standpoint-that is, as a Cartesian product of two or more iterables. In mathematics, the Cartesian product of two sets A and B is the set of all tuples of the form (a, b) where a is an element of A and b is an element of B.

The itertools.product() function is for exactly this situation. It takes any number of iterables as arguments and returns an iterator over tuples in the Cartesian product:

The product() function is by no means limited to two iterables. You can pass it as many as you like-they don't even have to all be of the same size! See if you can predict what product([1, 2, 3], ['a', 'b'], ['c']) is, then check your work by running it in the interpreter.

Warning: The product() function is another "brute force" function and can lead to a combinatorial explosion if you aren't careful.

Using product(), you can re-write the cards in a single line:

This is all fine and dandy, but any Poker app worth its salt better start with a shuffled deck:

Note: The random.shuffle() function uses the Fisher-Yates shuffle to shuffle a list (or any mutable sequence) in place . This algorithm is well-suited for shuffling cards because it produces an unbiased permutation-that is, all permutations of the iterable are equally likely to be returned by random.shuffle().

That said, you probably noticed that shuffle() creates a copy of its input deck in memory by calling list(deck). While this seemingly goes against the spirit of this article, this author is unaware of a good way to shuffle an iterator without making a copy.

As a courtesy to your users, you would like to give them the opportunity to cut the deck. If you imagine the cards being stacked neatly on a table, you have the user pick a number n and then remove the first n cards from the top of the stack and move them to the bottom.

If you know a thing or two about slicing, you might accomplish this like so:

Let's take a look at how those functions work.

While tee() is useful for creating independent iterators, it is important to understand a little bit about how it works under the hood. When you call tee() to create n independent iterators, each iterator is essentially working with its own FIFO queue.

When a value is extracted from one iterator, that value is appended to the queues for the other iterators. Thus, if one iterator is exhausted before the others, each remaining iterator will hold a copy of the entire iterable in memory. (You can find a Python function that emulates tee() in the itertools docs.)

The islice() function works much the same way as slicing a list or tuple. You pass it an iterable, a starting, and stopping point, and, just like slicing a list, the slice returned stops at the index just before the stopping point. You can optionally include a step value, as well. The biggest difference here is, of course, that islice() returns an iterator.

The last function you need is chain(). This function takes any number of iterables as arguments and "chains" them together. For example:

Now that you've got some additional firepower in your arsenal, you can re-write the cut() function to cut the deck of cards without making a full copy cards in memory:

Now that you have shuffled and cut the cards, it is time to deal some hands. You could write a function deal() that takes a deck, the number of hands, and the hand size as arguments and returns a tuple containing the specified number of hands.

You do not need any new itertools functions to write this function. See what you can come up with on your own before reading ahead.

Here's one solution:

What do you think the state of cards is now that you have dealt three hands of five cards?

The fifteen cards dealt are consumed from the cards iterator, which is exactly what you want. That way, as the game continues, the state of the cards iterator reflects the state of the deck in play.

Section Recap

Let's review the itertools functions you saw in this section.

itertools.product Example

itertools.tee Example

itertools.islice Example

islice(iterable, stop)islice(iterable, start, stop, step=1)

Return an iterator whose __next__() method returns selected values from an iterable. Works like a slice() on a list but returns an iterator.

itertools.chain Example

chain(*iterables)
Return a chain object whose __next__() method returns elements from the first iterable until it is exhausted, then elements from the next iterable, until all of the iterables are exhausted.

Intermission: Flattening A List of Lists

There's no reason the argument of chain.from_iterable() needs to be finite. You could emulate the behavior of cycle(), for example:

The chain.from_iterable() function is useful when you need to build an iterator over data that has been "chunked."

In the next section, you will see how to use itertools to do some data analysis on a large dataset. But you deserve a break for having stuck with it this far. Why not hydrate yourself and relax a bit? Maybe even play a little Star Trek: The Nth Iteration.

Back? Great! Let's do some data analysis.

Analyzing the S&P500

In this example, you will get your first taste of using itertools to manipulate a large dataset-in particular, the historical daily price data of the S&P500 index. A CSV file SP500.csv with this data can be found here (source: Yahoo Finance). The problem you'll tackle is this:

Determine the maximum daily gain, daily loss (in percent change), and the longest growth streak in the history of the S&P500.

To get a feel for what you're dealing with, here are the first ten rows of SP500.csv:

As you can see, the early data is limited. The data improves for later dates, and, as a whole, is sufficient for this example.

The strategy for solving this problem is as follows:

Read data from the CSV file and transform it into a sequence gains of daily percent changes using the "Adj Close" column.

Find the maximum and minimum values of the gains sequence, and the date on which they occur. (Note that it is possible that these values are attained on more than on date; in that case, the most recent date will suffice.)

Transform gains into a sequence growth_streaks of tuples of consecutive positive values in gains. Then determine the length of the longest tuple in growth_streaks and the beginning and ending dates of the streak. (It is possible that the maximum length is attained by more than one tuple in growth_streaks; in that case, the tuple with the most recent beginning and ending dates will suffice.)

The percent change between two values x and y is given by the following formula:

For each step in the analysis, it is necessary to compare values associated with dates. To facilitate these comparisons, you can subclass the namedtuple object from the collections module:

The DataPoint class has two attributes: date (a datetime.datetime instance) and value. The , and dunder methods are implemented so that the <=, <, and > boolean comparators can be used to compare the values of two DataPoint objects. This also allows the and built-in functions to be called with DataPoint arguments.

Note: If you are not familiar with namedtuple, check out this excellent resource. The namedtuple implementation for DataPoint is just one of many ways to build this data structure. For example, in Python 3.7 you could implement DataPoint as a data class. Check out our Ultimate Guide to Data Classes for more information.

The following reads the data from SP500.csv to a tuple of DataPoint objects:

The read_prices() generator opens SP500.csv and reads each row with a csv.DictReader() object. DictReader() returns each row as an OrderedDict whose keys are the column names from the header row of the CSV file.

Next, prices needs to be transformed to a sequence of daily percent changes:

The choice of storing the data in a tuple is intentional. Although you could point gains to an iterator, you will need to iterate over the data twice to find the minimum and maximum values.

Note: This example focuses on leveraging itertools for analyzing the S&P500 data. Those intent on working with a lot of time series financial data might also want to check out the Pandas library, which is well suited for such tasks.

Maximum Gain and Loss

To determine the maximum gain on any single day, you might do something like this:

You can simplify the for loop using the functools.reduce() function. This function accepts a binary function func and an iterable inputs as arguments, and "reduces" inputs to a single value by applying func cumulatively to pairs of objects in the iterable.

Using reduce(), you can get rid of the for loop altogether in the above example:

The above solution works, but it isn't equivalent to the for loop you had before. Do you see why? Suppose the data in your CSV file recorded a loss every single day. What would the value of max_gain be?

This is where itertools can help you out. The itertools.filterfalse() function takes two arguments: a function that returns True or False (called a predicate), and an iterable inputs. It returns an iterator over the elements in inputs for which the predicate returns False.

Here's a simple example:

What happens if there are never any gains? Consider the following:

>>> ft.reduce(max, it.filterfalse(lambda x: x <= 0, [-1, -2, -3]))
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: reduce() of empty sequence with no initial value

The reduce() function accepts an optional third argument for an initial value. Passing 0 to this third argument gets you the expected behavior:

Applying this to the S&P500 example:

Great! You've got it working just the way it should! Now, finding the maximum loss is easy:

Longest Growth Streak

The dropwhile() function does exactly the opposite. It returns an iterator beginning at the first element for which the predicate returns False:

In the following generator function, takewhile() and dropwhile() are composed to yield tuples of consecutive positive elements of a sequence:

You can use consecutive_positives() to get a generator that produces tuples of consecutive positive data points in gains:

Now you can use reduce() to extract the longest growth streak:

Putting the whole thing together, here's a full script that will read data from the SP500.csv file and print out the max gain/loss and longest growth streak:

Running the above script produces the following output:

Max gain: 11.58% on 2008-10-13
Max loss: -20.47% on 1987-10-19
Longest growth streak: 14 days (1971-03-26 to 1971-04-15)

Section Recap

In this section, you covered a lot of ground, but you only saw a few functions from itertools. Let's review those now.

itertools.filterfalse Example

filterfalse(pred, iterable)
Return those items of sequence for which pred(item) is false. If pred is None, return the items that are false.

itertools.takewhile Example

takewhile(pred, iterable)
Return successive entries from an iterable as long as pred evaluates to true for each entry.

itertools.dropwhile Example

dropwhile(pred, iterable)
Drop items from the iterable while pred(item) is true. Afterwards, return every element until the iterable is exhausted.

You are really starting to master this whole itertools thing! The community swim team would like to commission you for a small project.

Building Relay Teams From Swimmer Data

In this example, you will read data from a CSV file containing swimming event times for a community swim team from all of the swim meets over the course of a season. The goal is to determine which swimmers should be in the relay teams for each stroke next season.

Each stroke should have an "A" and a "B" relay team with four swimmers each. The "A" team should contain the four swimmers with the best times for the stroke and the "B" team the swimmers with the next four best times.

The data for this example can be found here. If you want to follow along, download it to your current working directory and save it as swimmers.csv.

Here are the first 10 rows of swimmers.csv:

The three times in each row represent the times recorded by three different stopwatches, and are given in MM:SS:mmmmmm format (minutes, seconds, microseconds). The accepted time for an event is the median of these three times, not the average.

Let's start by creating a subclass Event of the namedtuple object, just like we did in the SP500 example:

To read the data from the CSV into a tuple of Event objects, you can use the csv.DictReader object:

For example, the first row of the file (excluding the header row) is read into the following object:

Next, read_events() yields an Event object with the stroke, swimmer name, and median time (as a datetime.time object) returned by the _median() function, which calls statistics.median() on the list of times in the row.

Since each item in the list of times is read as a string by csv.DictReader(), _median() uses the datetime.datetime.strptime() classmethod to instantiate a time object from each string.

Finally, a tuple of Event objects is created:

The first five elements of events look like this:

Now that you've got the data into memory, what do you do with it? Here's the plan of attack:

Group the events by stroke.

For each stroke:

Group its events by swimmer name and determine the best time for each swimmer.

Order the swimmers by best time.

The first four swimmers make the "A" team for the stroke, and the next four swimmers make the "B" team.

Here's a simple groupby() example:

If no key is specified, groupby() defaults to grouping by "identity"-that is, aggregating identical elements in the iterable:

The object returned by groupby() is sort of like a dictionary in the sense that the iterators returned are associated with a key. However, unlike a dictionary, it won't allow you to access its values by key name:

>>> grouped_data[1]
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: 'itertools.groupby' object is not subscriptable

In fact, groupby() returns an iterator over tuples whose first components are keys and second components are iterators over the grouped data:

One thing to keep in mind with groupby() is that it isn't as smart as you might like. As groupby() traverses the data, it aggregates elements until an element with a different key is encountered, at which point it starts a new group:

Compare this to, say, the SQL GROUP BY command, which groups elements regardless of their order of appearance.

When working with groupby(), you need to sort your data on the same key that you would like to group by. Otherwise, you may get unexpected results. This is so common that it helps to write a utility function to take care of this for you:

Returning to the swimmers example, the first thing you need to do is create a for loop that iterates over the data in the events tuple grouped by stroke:

Next, you need to group the evts iterator by swimmer name inside of the above for loop:

Note that the best_times generator yields Event objects containing the best stroke time for each swimmer. To build the relay teams, you'll need to sort best_times by time and aggregate the result into groups of four. To aggregate the results, you can use the grouper() function from The grouper() recipe section and use islice() to grab the first two groups.

Now teams is an iterator over exactly two tuples representing the "A" and the "B" team for the stroke. The first component of each tuple is the letter "A" or "B", and the second component is an iterator over Event objects containing the swimmers in the team. You can now print the results:

Here's the full script:

If you run the above code, you'll get the following output:

Backstroke A: Sophia, Grace, Penelope, Addison
Backstroke B: Elizabeth, Audrey, Emily, Aria
Breaststroke A: Samantha, Avery, Layla, Zoe
Breaststroke B: Lillian, Aria, Ava, Alexa
Butterfly A: Audrey, Leah, Layla, Samantha
Butterfly B: Alexa, Zoey, Emma, Madison
Freestyle A: Aubrey, Emma, Olivia, Evelyn
Freestyle B: Elizabeth, Zoe, Addison, Madison

Where to Go From Here

If you have made it this far, congratulations! I hope you have enjoyed the journey.

itertools is a powerful module in the Python standard library, and an essential tool to have in your toolkit. With it, you can write faster and more memory efficient code that is often simpler and easier to read (although that is not always the case, as you saw in the section on second order recurrence relations).

If anything, though, itertools is a testament to the power of iterators and lazy evaluation. Even though you have seen many techniques, this article only scratches the surface.

So I guess this means your journey is only just beginning.

In fact, this article skipped two itertools functions: and . In my experience, these are two of the lesser used itertools functions, but I urge you to read their docs an experiment with your own use cases!

Here are a few places where you can find more examples of itertools in action (thanks to Brad Solomon for these fine suggestions):

Finally, for even more tools for constructing iterators, take a look at more-itertools.

Do you have any favorite itertools recipes/use-cases? We would love to hear about them in the comments!
We would like to thank our readers Putcher and Samir Aghayev for pointing out a couple of errors in the original version of this article.

# data[].attributes.attachment_link.title
Itertools in Python 3, By Example - Real Python

# data[].attributes.attachment_link.description
It has been called a "gem" and "pretty much the coolest thing ever," and if you have not heard of it, then you are missing out on one of the greatest corners of the Python 3 standard library: itertools. A handful of excellent resources exist for learning what functions are available in the itertools module.

# data[].attributes.attachment_link.readable_text
pandas. read_excel( io, sheet_name=0, header=0, names=None, index_col=None, parse_cols=None, usecols=None, squeeze=False, dtype=None, engine=None, converters=None, true_values=None, false_values=None, skiprows=None, nrows=None, na_values=None, keep_default_na=True, verbose=False, parse_dates=False, date_parser=None, thousands=None, comment=None, skip_footer=0, skipfooter=0, convert_float=True, mangle_dupe_cols=True, **kwds)

Read an Excel file into a pandas DataFrame.

Support both xls and xlsx file extensions from a local filesystem or URL. Support an option to read a single sheet or a list of sheets.

Parameters:

The string could be a URL. Valid URL schemes include http, ftp, s3, gcs, and file. For file URLs, a host is expected. For instance, a local file could be /path/to/workbook.xlsx.

Strings are used for sheet names. Integers are used in zero-indexed sheet positions. Lists of strings/integers are used to request multiple sheets. Specify None to get all sheets.

Available cases:

Row (0-indexed) to use for the column labels of the parsed DataFrame. If a list of integers is passed those row positions will be combined into a MultiIndex . Use None if there is no header.

List of column names to use. If file contains no header row, then you should explicitly pass header=None.

Column (0-indexed) to use as the row labels of the DataFrame. Pass None if there is no such column. If a list is passed, those columns will be combined into a MultiIndex . If a subset of data is selected with , index_col is based on the subset.

Return a subset of the columns. * If None, then parse all columns. * If int, then indicates last column to be parsed.

Deprecated since version 0.24.0: Pass in a list of int instead from 0 to usecols inclusive.

If str, then indicates comma separated list of Excel column letters and column ranges (e.g. "A:E" or "A,C,E:F"). Ranges are inclusive of both sides.

If list of int, then indicates list of column numbers to be parsed.

If list of string, then indicates list of column names to be parsed.

If callable, then evaluate each column name against it and parse the column if the callable returns .

If the parsed data only contains one column then return a Series.

Data type for data or columns. E.g. {'a': np.float64, 'b': np.int32} Use object to preserve data as stored in Excel and not interpret dtype. If converters are specified, they will be applied INSTEAD of dtype conversion.

If io is not a buffer or path, this must be set to identify io. Acceptable values are None or xlrd.

Dict of functions for converting values in certain columns. Keys can either be integers or column labels, values are functions that take one input argument, the Excel cell content, and return the transformed content.

Rows to skip at the beginning (0-indexed).

Additional strings to recognize as NA/NaN. If dict passed, specific per-column NA values. By default the following values are interpreted as NaN: '', '#N/A', '#N/A N/A', '#NA', '-1.#IND', '-1.#QNAN', '-NaN', '-nan', '1.#IND', '1.#QNAN', 'N/A', 'NA', 'NULL', 'NaN', 'n/a', 'nan', 'null'.

If na_values are specified and keep_default_na is False the default NaN values are overridden, otherwise they're appended to.

Indicate number of NA values placed in non-numeric columns.

The behavior is as follows:

bool. If True -> try parsing the index.

list of int or names. e.g. If [1, 2, 3] -> try parsing columns 1, 2, 3 each as a separate date column.

list of lists. e.g. If [[1, 3]] -> combine columns 1 and 3 and parse as a single date column.

dict, e.g. {{'foo' : [1, 3]}} -> parse columns 1, 3 as date and call result 'foo'

If a column or index contains an unparseable date, the entire column or index will be returned unaltered as an object data type. For non-standard datetime parsing, use pd.to_datetime after

Note: A fast-path exists for iso8601-formatted dates.

Function to use for converting a sequence of string columns to an array of datetime instances. The default uses dateutil.parser.parser to do the conversion. Pandas will try to call date_parser in three different ways, advancing to the next if an exception occurs: 1) Pass one or more arrays (as defined by parse_dates) as arguments; 2) concatenate (row-wise) the string values from the columns defined by parse_dates into a single array and pass that; and 3) call date_parser once for each row using one or more strings (corresponding to the columns defined by parse_dates) as arguments.

Thousands separator for parsing string columns to numeric. Note that this parameter is only necessary for columns stored as TEXT in Excel, any numeric columns will automatically be parsed, regardless of display format.

Comments out remainder of line. Pass a character or characters to this argument to indicate comments in the input file. Any data between the comment string and the end of the current line is ignored.

Alias of skipfooter.

Deprecated since version 0.23.0: Use skipfooter instead.

Rows at the end to skip (0-indexed).

Convert integral floats to int (i.e., 1.0 -> 1). If False, all numeric data will be read in as floats: Excel stores all numbers as floats internally.

Duplicate columns will be specified as 'X', 'X.1', ...'X.N', rather than 'X'...'X'. Passing in False will cause data to be overwritten if there are duplicate names in the columns.

Optional keyword arguments can be passed to TextFileReader .

Returns:

DataFrame or dict of DataFrames

DataFrame from the passed in Excel file. See notes in sheet_name argument for more information on when a dict of DataFrames is returned.

Examples

The file can be read using the file name as string or an open file object:

Index and header can be specified via the index_col and header arguments

Column types are inferred but can be explicitly specified

True, False, and NA values, and thousands separators have defaults, but can be explicitly specified, too. Supply the values you would like as strings or lists of strings!

Comment lines in the excel input file can be skipped using the comment kwarg

# data[].attributes.attachment_link.title
pandas.read_excel - pandas 0.24.2 documentation

# data[].attributes.attachment_link.description
pandas. read_excel( io, sheet_name=0, header=0, names=None, index_col=None, parse_cols=None, usecols=None, squeeze=False, dtype=None, engine=None, converters=None, true_values=None, false_values=None, skiprows=None, nrows=None, na_values=None, keep_default_na=True, verbose=False, parse_dates=False, date_parser=None, thousands=None, comment=None, skip_footer=0, skipfooter=0, convert_float=True, mangle_dupe_cols=True, **kwds) Read an Excel file into a pandas DataFrame. Support both xls and xlsx file extensions from a local filesystem or URL.

# data[].attributes.attachment_link.readable_text
Working with Pandas

Loading and Saving Data with Pandas

When you want to use Pandas for data analysis, you'll usually use it in one of three different ways:

Convert a Python's list, dictionary or Numpy array to a Pandas data frame

Open a local file using Pandas, usually a CSV file, but could also be a delimited text file (like TSV), Excel, etc

Open a remote file or database like a CSV or a JSONon a website through a URL or read from a SQL table/database

There are different commands to each of these options, but when you open a file, they would look like this:

pd.read_filetype()

As I mentioned before, there are different filetypes Pandas can work with, so you would replace "filetype" with the actual, well, filetype (like CSV). You would give the path, filename etc inside the parenthesis. Inside the parenthesis you can also pass different arguments that relate to how to open the file. There are numerous arguments and in order to know all you them, you would have to read the documentation (for example, the documentation for pd.read_csv() would contain all the arguments you can pass in this Pandas command).

In order to convert a certain Python object (dictionary, lists etc) the basic command is:

pd.DataFrame()

Inside the parenthesis you would specify the object(s) you're creating the data frame from. This command also has different arguments (clickable link).

You can also save a data frame you're working with/on to different kinds of files (like CSV, Excel, JSON and SQL tables). The general code for that is:

df.to_filetype(filename)

Viewing and Inspecting Data

Now that you've loaded your data, it's time to take a look. How does the data frame look? Running the name of the data frame would give you the entire table, but you can also get the first n rows with df.head(n) or the last n rows with df.tail(n). df.shape would give you the number of rows and columns. df.info() would give you the index, datatype and memory information. The command s.value_counts(dropna=False) would allow you to view unique values and counts for a series (like a column or a few columns). A very useful command is df.describe() which inputs summary statistics for numerical columns. It is also possible to get statistics on the entire data frame or a series (a column etc):

df.mean()Returns the mean of all columns

df.corr()Returns the correlation between columns in a data frame

df.count()Returns the number of non-null values in each data frame column

df.max()Returns the highest value in each column

df.min()Returns the lowest value in each column

df.median()Returns the median of each column

df.std()Returns the standard deviation of each column

Selection of Data

One of the things that is so much easier in Pandas is selecting the data you want in comparison to selecting a value from a list or a dictionary. You can select a column ( df[col]) and return column with label col as Series or a few columns ( df[[col1, col2]]) and returns columns as a new DataFrame. You can select by position ( s.iloc[0]), or by index ( s.loc['index_one']) . In order to select the first row you can use df.iloc[0,:] and in order to select the first element of the first column you would run df.iloc[0,0] . These can also be used in different combinations, so I hope it gives you an idea of the different selection and indexing you can perform in Pandas.

Filter, Sort and Groupby

You can use different conditions to filter columns. For example, df[df[year] > 1984] would give you only the column year is greater than 1984. You can use & (and) or | (or) to add different conditions to your filtering. This is also called boolean filtering.

It is possible to sort values in a certain column in an ascending order using df.sort_values(col1) ; and also in a descending order using df.sort_values(col2,ascending=False). Furthermore, it's possible to sort values by col1 in ascending order then col2 in descending order by using df.sort_values([col1,col2],ascending=[True,False]).

The last command in this section is groupby. It involves splitting the data into groups based on some criteria, applying a function to each group independently and combining the results into a data structure. df.groupby(col) returns a groupby object for values from one column while df.groupby([col1,col2]) returns a groupby object for values from multiple columns.

Data Cleaning

Data cleaning is a very important step in data analysis. For example, we always check for missing values in the data by running pd.isnull() which checks for null Values, and returns a boolean array (an array of true for missing values and false for non-missing values). In order to get a sum of null/missing values, run pd.isnull().sum(). pd.notnull() is the opposite of pd.isnull(). After you get a list of missing values you can get rid of them, or drop them by using df.dropna() to drop the rows or df.dropna(axis=1) to drop the columns. A different approach would be to fill the missing values with other values by using df.fillna(x) which fills the missing values with x (you can put there whatever you want) or s.fillna(s.mean()) to replace all null values with the mean (mean can be replaced with almost any function from the statistics section).

It is sometimes necessary to replace values with different values. For example, s.replace(1,'one') would replace all values equal to 1 with 'one'. It's possible to do it for multiple values: s.replace([1,3],['one','three']) would replace all 1 with 'one' and 3 with 'three'. You can also rename specific columns by running: df.rename(columns={'old_name': 'new_ name'}) or use df.set_index('column_one') to change the index of the data frame.

Join/Combine

The last set of basic Pandas commands are for joining or combining data frames or rows/columns. The three commands are:

df1.append(df2)- add the rows in df1 to the end of df2 (columns should be identical)

df.concat([df1, df2],axis=1) - add the columns in df1 to the end of df2 (rows should be identical)

df1.join(df2,on=col1,how='inner') - SQL-style join the columns in df1 with the columns on df2 where the rows for colhave identical values. how can be equal to one of: 'left', 'right', 'outer', 'inner'

# data[].attributes.attachment_link.title
A Quick Introduction to the "Pandas" Python Library

# data[].attributes.attachment_link.description
Loading and Saving Data with Pandas When you want to use Pandas for data analysis, you'll usually use it in one of three different ways: Convert a Python's list, dictionary or Numpy array to a Pandas data frame Open a local file using Pandas, usually a CSV file, but could

# data[].attributes.attachment_link.readable_text
The Python interpreter has a number of functions and types built into it that are always available. They are listed here in alphabetical order.

Return the absolute value of a number. The argument may be an integer, a floating point number, or an object implementing . If the argument is a complex number, its magnitude is returned.

Return if all elements of the iterable are true (or if the iterable is empty). Equivalent to:

Return if any element of the iterable is true. If the iterable is empty, return . Equivalent to:

As , return a string containing a printable representation of an object, but escape the non-ASCII characters in the string returned by using , , or escapes. This generates a string similar to that returned by in Python 2.

Convert an integer number to a binary string prefixed with "0b". The result is a valid Python expression. If x is not a Python object, it has to define an method that returns an integer. Some examples:

If the prefix "0b" is desired or not, you can use either of the following ways.

This function drops you into the debugger at the call site. Specifically, it calls , passing and straight through. By default, sys.breakpointhook() calls expecting no arguments. In this case, it is purely a convenience function so you don't have to explicitly import or type as much code to enter the debugger. However, can be set to some other function and will automatically call that, allowing you to drop into the debugger of choice. If is not accessible, this function will raise .

By default, the behavior of can be changed with the environment variable. See for usage details.

Raises an auditing event builtins.breakpoint with argument breakpointhook .

Return a new array of bytes. The class is a mutable sequence of integers in the range 0 <= x < 256. It has most of the usual methods of mutable sequences, described in Mutable Sequence Types , as well as most methods that the type has, see Bytes and Bytearray Operations .

The optional source parameter can be used to initialize the array in a few different ways:

If it is a string, you must also give the encoding (and optionally, errors) parameters; then converts the string to bytes using .

If it is an integer, the array will have that size and will be initialized with null bytes.

If it is an object conforming to the buffer interface , a read-only buffer of the object will be used to initialize the bytes array.

If it is an iterable, it must be an iterable of integers in the range , which are used as the initial contents of the array.

Without an argument, an array of size 0 is created.

See also Binary Sequence Types - bytes, bytearray, memoryview and Bytearray Objects .

Return if the object argument appears callable, if not. If this returns , it is still possible that a call fails, but if it is , calling object will never succeed. Note that classes are callable (calling a class returns a new instance); instances are callable if their class has a method.

New in version 3.2: This function was first removed in Python 3.0 and then brought back in Python 3.2.

Return the string representing a character whose Unicode code point is the integer i. For example, returns the string , while returns the string . This is the inverse of .

The valid range for the argument is from 0 through 1,114,111 (0x10FFFF in base 16). will be raised if i is outside that range.

Transform a method into a class method.

A class method receives the class as an implicit first argument, just like an instance method receives the instance. To declare a class method, use this idiom:

The @classmethod form is a function - see Function definitions for details.

A class method can be called either on the class (such as ) or on an instance (such as ). The instance is ignored except for its class. If a class method is called for a derived class, the derived class object is passed as the implied first argument.

Class methods are different than C++ or Java static methods. If you want those, see in this section. For more information on class methods, see The standard type hierarchy .

Compile the source into a code or AST object. Code objects can be executed by or . source can either be a normal string, a byte string, or an AST object. Refer to the module documentation for information on how to work with AST objects.

The filename argument should give the file from which the code was read; pass some recognizable value if it wasn't read from a file ( is commonly used).

The mode argument specifies what kind of code must be compiled; it can be if source consists of a sequence of statements, if it consists of a single expression, or if it consists of a single interactive statement (in the latter case, expression statements that evaluate to something other than will be printed).

The optional arguments flags and dont_inherit control which compiler options should be activated and which future features should be allowed. If neither is present (or both are zero) the code is compiled with the same flags that affect the code that is calling . If the flags argument is given and dont_inherit is not (or is zero) then the compiler options and the future statements specified by the flags argument are used in addition to those that would be used anyway. If dont_inherit is a non-zero integer then the flags argument is it - the flags (future features and compiler options) in the surrounding code are ignored.

Compiler options and future statements are specified by bits which can be bitwise ORed together to specify multiple options. The bitfield required to specify a given future feature can be found as the compiler_flag attribute on the instance in the module. Compiler flags can be found in module, with prefix.

The argument optimize specifies the optimization level of the compiler; the default value of selects the optimization level of the interpreter as given by options. Explicit levels are (no optimization; is true), (asserts are removed, is false) or (docstrings are removed too).

Raises an auditing event with arguments and . This event may also be raised by implicit compilation.

Note

When compiling a string with multi-line code in or mode, input must be terminated by at least one newline character. This is to facilitate detection of incomplete and complete statements in the module.

Warning

It is possible to crash the Python interpreter with a sufficiently large/complex string when compiling to an AST object due to stack depth limitations in Python's AST compiler.

Changed in version 3.2: Allowed use of Windows and Mac newlines. Also, input in mode does not have to end in a newline anymore. Added the optimize parameter.

Changed in version 3.5: Previously, was raised when null bytes were encountered in source.

Return a complex number with the value real + imag*1j or convert a string or number to a complex number. If the first parameter is a string, it will be interpreted as a complex number and the function must be called without a second parameter. The second parameter can never be a string. Each argument may be any numeric type (including complex). If imag is omitted, it defaults to zero and the constructor serves as a numeric conversion like and . If both arguments are omitted, returns .

The complex type is described in Numeric Types - int, float, complex .

Changed in version 3.6: Grouping digits with underscores as in code literals is allowed.

This is a relative of . The arguments are an object and a string. The string must be the name of one of the object's attributes. The function deletes the named attribute, provided the object allows it. For example, is equivalent to . name need not be a Python identifier (see ).

Take two (non-complex) numbers as arguments and return a pair of numbers consisting of their quotient and remainder when using integer division. With mixed operand types, the rules for binary arithmetic operators apply. For integers, the result is the same as . For floating point numbers the result is , where q is usually but may be 1 less than that. In any case is very close to a, if is non-zero it has the same sign as b, and .

Return an enumerate object. iterable must be a sequence, an , or some other object which supports iteration. The method of the iterator returned by returns a tuple containing a count (from start which defaults to 0) and the values obtained from iterating over iterable.

Equivalent to:

The arguments are a string and optional globals and locals. If provided, globals must be a dictionary. If provided, locals can be any mapping object.

The expression argument is parsed and evaluated as a Python expression (technically speaking, a condition list) using the globals and locals dictionaries as global and local namespace. If the globals dictionary is present and does not contain a value for the key , a reference to the dictionary of the built-in module is inserted under that key before expression is parsed. That way you can control what builtins are available to the executed code by inserting your own dictionary into globals before passing it to . If the locals dictionary is omitted it defaults to the globals dictionary. If both dictionaries are omitted, the expression is executed with the globals and locals in the environment where is called. Note, eval() does not have access to the nested scopes (non-locals) in the enclosing environment.

The return value is the result of the evaluated expression. Syntax errors are reported as exceptions. Example:

This function can also be used to execute arbitrary code objects (such as those created by ). In this case, pass a code object instead of a string. If the code object has been compiled with as the mode argument, 's return value will be .

If the given source is a string, then leading and trailing spaces and tabs are stripped.

Raises an auditing event with the code object as the argument. Code compilation events may also be raised.

This function supports dynamic execution of Python code. object must be either a string or a code object. If it is a string, the string is parsed as a suite of Python statements which is then executed (unless a syntax error occurs). 1 If it is a code object, it is simply executed. In all cases, the code that's executed is expected to be valid as file input (see the section in the Reference Manual). Be aware that the , , and statements may not be used outside of function definitions even within the context of code passed to the function. The return value is .

In all cases, if the optional parts are omitted, the code is executed in the current scope. If only globals is provided, it must be a dictionary (and not a subclass of dictionary), which will be used for both the global and the local variables. If globals and locals are given, they are used for the global and local variables, respectively. If provided, locals can be any mapping object. Remember that at the module level, globals and locals are the same dictionary. If exec gets two separate objects as globals and locals, the code will be executed as if it were embedded in a class definition.

If the globals dictionary does not contain a value for the key , a reference to the dictionary of the built-in module is inserted under that key. That way you can control what builtins are available to the executed code by inserting your own dictionary into globals before passing it to .

The closure argument specifies a closure-a tuple of cellvars. It's only valid when the object is a code object containing free variables. The length of the tuple must exactly match the number of free variables referenced by the code object.

Raises an auditing event with the code object as the argument. Code compilation events may also be raised.

Note

The built-in functions and return the current global and local dictionary, respectively, which may be useful to pass around for use as the second and third argument to .

Note

The default locals act as described for function below: modifications to the default locals dictionary should not be attempted. Pass an explicit locals dictionary if you need to see effects of the code on locals after function returns.

Construct an iterator from those elements of iterable for which function is true. iterable may be either a sequence, a container which supports iteration, or an iterator. If function is , the identity function is assumed, that is, all elements of iterable that are false are removed.

Note that filter(function, iterable) is equivalent to the generator expression if function is not and if function is .

See itertools.filterfalse() for the complementary function that returns elements of iterable for which function is false.

Return a floating point number constructed from a number or string x.

If the argument is a string, it should contain a decimal number, optionally preceded by a sign, and optionally embedded in whitespace. The optional sign may be or ; a sign has no effect on the value produced. The argument may also be a string representing a NaN (not-a-number), or positive or negative infinity. More precisely, the input must conform to the floatvalue production rule in the following grammar, after leading and trailing whitespace characters are removed:

Here is a Unicode decimal digit (character in the Unicode general category ). Case is not significant, so, for example, "inf", "Inf", "INFINITY", and "iNfINity" are all acceptable spellings for positive infinity.

If no argument is given, is returned.

Examples:

The float type is described in Numeric Types - int, float, complex .

Changed in version 3.6: Grouping digits with underscores as in code literals is allowed.

Convert a value to a "formatted" representation, as controlled by format_spec. The interpretation of format_spec will depend on the type of the value argument; however, there is a standard formatting syntax that is used by most built-in types: Format Specification Mini-Language .

The default format_spec is an empty string which usually gives the same effect as calling .

A call to format(value, format_spec) is translated to type(value).__format__(value, format_spec) which bypasses the instance dictionary when searching for the value's method. A exception is raised if the method search reaches and the format_spec is non-empty, or if either the format_spec or the return value are not strings.

Changed in version 3.4: object().__format__(format_spec) raises if format_spec is not an empty string.

Return a new object, optionally with elements taken from iterable. is a built-in class. See and Set Types - set, frozenset for documentation about this class.

Return the value of the named attribute of object. name must be a string. If the string is the name of one of the object's attributes, the result is the value of that attribute. For example, is equivalent to . If the named attribute does not exist, default is returned if provided, otherwise is raised. name need not be a Python identifier (see ).

Note

Since private name mangling happens at compilation time, one must manually mangle a private attribute's (attributes with two leading underscores) name in order to retrieve it with .

Return the dictionary implementing the current module namespace. For code within functions, this is set when the function is defined and remains the same regardless of where the function is called.

The arguments are an object and a string. The result is if the string is the name of one of the object's attributes, if not. (This is implemented by calling and seeing whether it raises an or not.)

Return the hash value of the object (if it has one). Hash values are integers. They are used to quickly compare dictionary keys during a dictionary lookup. Numeric values that compare equal have the same hash value (even if they are of different types, as is the case for 1 and 1.0).

Note

For objects with custom methods, note that truncates the return value based on the bit width of the host machine. See for details.

Convert an integer number to a lowercase hexadecimal string prefixed with "0x". If x is not a Python object, it has to define an method that returns an integer. Some examples:

If you want to convert an integer number to an uppercase or lower hexadecimal string with prefix or not, you can use either of the following ways:

Note

To obtain a hexadecimal string representation for a float, use the method.

If x is not a number or if base is given, then x must be a string, , or instance representing an integer in radix base. Optionally, the string can be preceded by or (with no space in between), have leading zeros, be surrounded by whitespace, and have single underscores interspersed between digits.

A base-n integer string contains digits, each representing a value from 0 to n-1. The values 0-9 can be represented by any Unicode decimal digit. The values 10-35 can be represented by to (or to ). The default base is 10. The allowed bases are 0 and 2-36. Base-2, -8, and -16 strings can be optionally prefixed with /, /, or /, as with integer literals in code. For base 0, the string is interpreted in a similar way to an integer literal in code , in that the actual base is 2, 8, 10, or 16 as determined by the prefix. Base 0 also disallows leading zeros: is not legal, while and are.

The integer type is described in Numeric Types - int, float, complex .

Changed in version 3.4: If base is not an instance of and the base object has a method, that method is called to obtain an integer for the base. Previous versions used instead of .

Changed in version 3.6: Grouping digits with underscores as in code literals is allowed.

Changed in version 3.11: The delegation to is deprecated.

Changed in version 3.11: string inputs and string representations can be limited to help avoid denial of service attacks. A is raised when the limit is exceeded while converting a string x to an or when converting an into a string would exceed the limit. See the integer string conversion length limitation documentation.

Return if the object argument is an instance of the classinfo argument, or of a (direct, indirect, or ) subclass thereof. If object is not an object of the given type, the function always returns . If classinfo is a tuple of type objects (or recursively, other such tuples) or a of multiple types, return if object is an instance of any of the types. If classinfo is not a type or tuple of types and such tuples, a exception is raised. may not be raised for an invalid type if an earlier check succeeds.

Return if class is a subclass (direct, indirect, or ) of classinfo. A class is considered a subclass of itself. classinfo may be a tuple of class objects (or recursively, other such tuples) or a , in which case return if class is a subclass of any entry in classinfo. In any other case, a exception is raised.

Return an object. The first argument is interpreted very differently depending on the presence of the second argument. Without a second argument, object must be a collection object which supports the protocol (the method), or it must support the sequence protocol (the method with integer arguments starting at ). If it does not support either of those protocols, is raised. If the second argument, sentinel, is given, then object must be a callable object. The iterator created in this case will call object with no arguments for each call to its method; if the value returned is equal to sentinel, will be raised, otherwise the value will be returned.

See also Iterator Types .

Return the length (the number of items) of an object. The argument may be a sequence (such as a string, bytes, tuple, list, or range) or a collection (such as a dictionary, set, or frozen set).

Update and return a dictionary representing the current local symbol table. Free variables are returned by when it is called in function blocks, but not in class blocks. Note that at the module level, and are the same dictionary.

Note

The contents of this dictionary should not be modified; changes may not affect the values of local and free variables used by the interpreter.

Return an iterator that applies function to every item of iterable, yielding the results. If additional iterables arguments are passed, function must take that many arguments and is applied to the items from all iterables in parallel. With multiple iterables, the iterator stops when the shortest iterable is exhausted. For cases where the function inputs are already arranged into argument tuples, see .

Return a "memory view" object created from the given argument. See Memory Views for more information.

Convert an integer number to an octal string prefixed with "0o". The result is a valid Python expression. If x is not a Python object, it has to define an method that returns an integer. For example:

If you want to convert an integer number to an octal string either with the prefix "0o" or not, you can use either of the following ways.

Open file and return a corresponding file object . If the file cannot be opened, an is raised. See Reading and Writing Files for more examples of how to use this function.

file is a path-like object giving the pathname (absolute or relative to the current working directory) of the file to be opened or an integer file descriptor of the file to be wrapped. (If a file descriptor is given, it is closed when the returned I/O object is closed unless closefd is set to .)

mode is an optional string that specifies the mode in which the file is opened. It defaults to which means open for reading in text mode. Other common values are for writing (truncating the file if it already exists), for exclusive creation, and for appending (which on some Unix systems, means that all writes append to the end of the file regardless of the current seek position). In text mode, if encoding is not specified the encoding used is platform-dependent: is called to get the current locale encoding. (For reading and writing raw bytes use binary mode and leave encoding unspecified.) The available modes are:

As mentioned in the , Python distinguishes between binary and text I/O. Files opened in binary mode (including in the mode argument) return contents as objects without any decoding. In text mode (the default, or when is included in the mode argument), the contents of the file are returned as , the bytes having been first decoded using a platform-dependent encoding or using the specified encoding if given.

Note

Python doesn't depend on the underlying operating system's notion of text files; all the processing is done by Python itself, and is therefore platform-independent.

buffering is an optional integer used to set the buffering policy. Pass 0 to switch buffering off (only allowed in binary mode), 1 to select line buffering (only usable when writing in text mode), and an integer > 1 to indicate the size in bytes of a fixed-size chunk buffer. Note that specifying a buffer size this way applies for binary buffered I/O, but TextIOWrapper (i.e., files opened with ) would have another buffering. To disable buffering in TextIOWrapper , consider using the write_through flag forio.TextIOWrapper.reconfigure() . When no buffering argument is given, the default buffering policy works as follows:

Binary files are buffered in fixed-size chunks; the size of the buffer is chosen using a heuristic trying to determine the underlying device's "block size" and falling back on . On many systems, the buffer will typically be 4096 or 8192 bytes long.

"Interactive" text files (files for which returns ) use line buffering. Other text files use the policy described above for binary files.

encoding is the name of the encoding used to decode or encode the file. This should only be used in text mode. The default encoding is platform dependent (whatever returns), but any text encoding supported by Python can be used. See the module for the list of supported encodings.

errors is an optional string that specifies how encoding and decoding errors are to be handled-this cannot be used in binary mode. A variety of standard error handlers are available (listed under Error Handlers ), though any error handling name that has been registered with is also valid. The standard names include:

to raise a exception if there is an encoding error. The default value of has the same effect.

ignores errors. Note that ignoring encoding errors can lead to data loss.

causes a replacement marker (such as ) to be inserted where there is malformed data.

'surrogateescape' will represent any incorrect bytes as low surrogate code units ranging from U+DC80 to U+DCFF. These surrogate code units will then be turned back into the same bytes when the surrogateescape error handler is used when writing data. This is useful for processing files in an unknown encoding.

'xmlcharrefreplace' is only supported when writing to a file. Characters not supported by the encoding are replaced with the appropriate XML character reference .

'backslashreplace' replaces malformed data by Python's backslashed escape sequences.

'namereplace' (also only supported when writing) replaces unsupported characters with escape sequences.

When reading input from the stream, if newline is , universal newlines mode is enabled. Lines in the input can end in , , or , and these are translated into before being returned to the caller. If it is , universal newlines mode is enabled, but line endings are returned to the caller untranslated. If it has any of the other legal values, input lines are only terminated by the given string, and the line ending is returned to the caller untranslated.

When writing output to the stream, if newline is , any characters written are translated to the system default line separator, . If newline is or , no translation takes place. If newline is any of the other legal values, any characters written are translated to the given string.

If closefd is and a file descriptor rather than a filename was given, the underlying file descriptor will be kept open when the file is closed. If a filename is given closefd must be (the default); otherwise, an error will be raised.

A custom opener can be used by passing a callable as opener. The underlying file descriptor for the file object is then obtained by calling opener with ( file, flags). opener must return an open file descriptor (passing as opener results in functionality similar to passing ).

The newly created file is non-inheritable .

The following example uses the parameter of the function to open a file relative to a given directory:

The type of file object returned by the function depends on the mode. When is used to open a file in a text mode (, , , , etc.), it returns a subclass of (specifically ). When used to open a file in a binary mode with buffering, the returned class is a subclass of . The exact class varies: in read binary mode, it returns an ; in write binary and append binary modes, it returns an , and in read/write mode, it returns an . When buffering is disabled, the raw stream, a subclass of , , is returned.

The and arguments may have been modified or inferred from the original call.

Changed in version 3.5:

If the system call is interrupted and the signal handler does not raise an exception, the function now retries the system call instead of raising an exception (see PEP 475 for the rationale).

The 'namereplace' error handler was added.

Changed in version 3.11: The mode has been removed.

Given a string representing one Unicode character, return an integer representing the Unicode code point of that character. For example, returns the integer and (Euro sign) returns . This is the inverse of .

Print objects to the text stream file, separated by sep and followed by end. sep, end, file, and flush, if present, must be given as keyword arguments.

All non-keyword arguments are converted to strings like does and written to the stream, separated by sep and followed by end. Both sep and end must be strings; they can also be , which means to use the default values. If no objects are given, will just write end.

The file argument must be an object with a write(string) method; if it is not present or , will be used. Since printed arguments are converted to text strings, cannot be used with binary mode file objects. For these, use instead.

Output buffering is usually determined by file. However, if flush is true, the stream is forcibly flushed.

Return a property attribute.

fget is a function for getting an attribute value. fset is a function for setting an attribute value. fdel is a function for deleting an attribute value. And doc creates a docstring for the attribute.

A typical use is to define a managed attribute :

If c is an instance of C, will invoke the getter, will invoke the setter, and the deleter.

If given, doc will be the docstring of the property attribute. Otherwise, the property will copy fget 's docstring (if it exists). This makes it possible to create read-only properties easily using as a :

The decorator turns the method into a "getter" for a read-only attribute with the same name, and it sets the docstring for voltage to "Get the current voltage."

This code is exactly equivalent to the first example. Be sure to give the additional functions the same name as the original property ( in this case.)

Changed in version 3.5: The docstrings of property objects are now writeable.

Rather than being a function, is actually an immutable sequence type, as documented in and Sequence Types - list, tuple, range .

Return a string containing a printable representation of an object. For many types, this function makes an attempt to return a string that would yield an object with the same value when passed to ; otherwise, the representation is a string enclosed in angle brackets that contains the name of the type of the object together with additional information often including the name and address of the object. A class can control what this function returns for its instances by defining a method. If is not accessible, this function will raise .

Return a reverse . seq must be an object which has a method or supports the sequence protocol (the method and the method with integer arguments starting at ).

Return number rounded to ndigits precision after the decimal point. If ndigits is omitted or is , it returns the nearest integer to its input.

For the built-in types supporting , values are rounded to the closest multiple of 10 to the power minus ndigits; if two multiples are equally close, rounding is done toward the even choice (so, for example, both and are , and is ). Any integer value is valid for ndigits (positive, zero, or negative). The return value is an integer if ndigits is omitted or . Otherwise, the return value has the same type as number.

Note

The behavior of for floats can be surprising: for example, gives instead of the expected . This is not a bug: it's a result of the fact that most decimal fractions can't be represented exactly as a float. See Floating Point Arithmetic: Issues and Limitations for more information.

This is the counterpart of . The arguments are an object, a string, and an arbitrary value. The string may name an existing attribute or a new attribute. The function assigns the value to the attribute, provided the object allows it. For example, is equivalent to .

name need not be a Python identifier as defined in Identifiers and keywords unless the object chooses to enforce that, for example in a custom or via . An attribute whose name is not an identifier will not be accessible using the dot notation, but is accessible through etc..

Note

Since private name mangling happens at compilation time, one must manually mangle a private attribute's (attributes with two leading underscores) name in order to set it with .

Return a object representing the set of indices specified by . The start and step arguments default to . Slice objects have read-only data attributes , , and which merely return the argument values (or their default). They have no other explicit functionality; however, they are used by NumPy and other third-party packages. Slice objects are also generated when extended indexing syntax is used. For example: a[start:stop:step] or . See for an alternate version that returns an iterator.

Return a new sorted list from the items in iterable.

Has two optional arguments which must be specified as keyword arguments.

key specifies a function of one argument that is used to extract a comparison key from each element in iterable (for example, key=str.lower ). The default value is (compare the elements directly).

reverse is a boolean value. If set to , then the list elements are sorted as if each comparison were reversed.

Use to convert an old-style cmp function to a key function.

The sort algorithm uses only comparisons between items. While defining an method will suffice for sorting, PEP 8 recommends that all six rich comparisons be implemented. This will help avoid bugs when using the same data with other ordering tools such as that rely on a different underlying method. Implementing all six comparisons also helps avoid confusion for mixed type comparisons which can call reflected the method.

For sorting examples and a brief sorting tutorial, see Sorting HOW TO .

Transform a method into a static method.

A static method does not receive an implicit first argument. To declare a static method, use this idiom:

The @staticmethod form is a function - see Function definitions for details.

Like all decorators, it is also possible to call staticmethod as a regular function and do something with its result. This is needed in some cases where you need a reference to a function from a class body and you want to avoid the automatic transformation to instance method. For these cases, use this idiom:

For more information on static methods, see The standard type hierarchy .

Changed in version 3.10: Static methods now inherit the method attributes (, , , and __annotations__ ), have a new attribute, and are now callable as regular functions.

Return a proxy object that delegates method calls to a parent or sibling class of type. This is useful for accessing inherited methods that have been overridden in a class.

The object_or_type determines the method resolution order to be searched. The search starts from the class right after the type.

The attribute of the object_or_type lists the method resolution search order used by both and . The attribute is dynamic and can change whenever the inheritance hierarchy is updated.

If the second argument is omitted, the super object returned is unbound. If the second argument is an object, must be true. If the second argument is a type, must be true (this is useful for classmethods).

There are two typical use cases for super. In a class hierarchy with single inheritance, super can be used to refer to parent classes without naming them explicitly, thus making the code more maintainable. This use closely parallels the use of super in other programming languages.

The second use case is to support cooperative multiple inheritance in a dynamic execution environment. This use case is unique to Python and is not found in statically compiled languages or languages that only support single inheritance. This makes it possible to implement "diamond diagrams" where multiple base classes implement the same method. Good design dictates that such implementations have the same calling signature in every case (because the order of calls is determined at runtime, because that order adapts to changes in the class hierarchy, and because that order can include sibling classes that are unknown prior to runtime).

For both use cases, a typical superclass call looks like this:

In addition to method lookups, also works for attribute lookups. One possible use case for this is calling descriptors in a parent or sibling class.

Note that is implemented as part of the binding process for explicit dotted attribute lookups such as super().__getitem__(name) . It does so by implementing its own__getattribute__() method for searching classes in a predictable order that supports cooperative multiple inheritance. Accordingly, is undefined for implicit lookups using statements or operators such as .

For practical suggestions on how to design cooperative classes using , see guide to using super().

With one argument, return the type of an object. The return value is a type object and generally the same object as returned by .

With three arguments, return a new type object. This is essentially a dynamic form of the statement. The name string is the class name and becomes the attribute. The bases tuple contains the base classes and becomes the attribute; if empty, , the ultimate base of all classes, is added. The dict dictionary contains attribute and method definitions for the class body; it may be copied or wrapped before becoming the attribute. The following two statements create identical objects:

Keyword arguments provided to the three argument form are passed to the appropriate metaclass machinery (usually ) in the same way that keywords in a class definition (besides metaclass) would.

See also Customizing class creation .

Changed in version 3.6: Subclasses of which don't override may no longer use the one-argument form to get the type of an object.

Iterate over several iterables in parallel, producing tuples with an item from each one.

Example:

More formally: returns an iterator of tuples, where the i-th tuple contains the i-th element from each of the argument iterables.

Another way to think of is that it turns rows into columns, and columns into rows. This is similar to transposing a matrix.

is lazy: The elements won't be processed until the iterable is iterated on, e.g. by a loop or by wrapping in a .

By default, stops when the shortest iterable is exhausted. It will ignore the remaining items in the longer iterables, cutting off the result to the length of the shortest iterable:

is often used in cases where the iterables are assumed to be of equal length. In such cases, it's recommended to use the strict=True option. Its output is the same as regular :

Unlike the default behavior, it raises a if one iterable is exhausted before the others:

Without the strict=True argument, any bug that results in iterables of different lengths will be silenced, possibly manifesting as a hard-to-find bug in another part of the program.

Shorter iterables can be padded with a constant value to make all the iterables have the same length. This is done by .

Tips and tricks:

The left-to-right evaluation order of the iterables is guaranteed. This makes possible an idiom for clustering a data series into n-length groups using . This repeats the same iterator times so that each output tuple has the result of calls to the iterator. This has the effect of dividing the input into n-length chunks.

Changed in version 3.10: Added the argument.

This function is invoked by the statement. It can be replaced (by importing the module and assigning to builtins.__import__ ) in order to change semantics of the statement, but doing so is strongly discouraged as it is usually simpler to use import hooks (seePEP 302) to attain the same goals and does not cause issues with code which assumes the default import implementation is in use. Direct use of is also discouraged in favor ofimportlib.import_module() .

The function imports the module name, potentially using the given globals and locals to determine how to interpret the name in a package context. The fromlist gives the names of objects or submodules that should be imported from the module given by name. The standard implementation does not use its locals argument at all and uses its globals only to determine the package context of the statement.

level specifies whether to use absolute or relative imports. (the default) means only perform absolute imports. Positive values for level indicate the number of parent directories to search relative to the directory of the module calling (seePEP 328 for the details).

When the name variable is of the form package.module , normally, the top-level package (the name up till the first dot) is returned, not the module named by name. However, when a non-empty fromlist argument is given, the module named by name is returned.

For example, the statement results in bytecode resembling the following code:

The statement results in this call:

On the other hand, the statement results in

Here, the module is returned from . From this object, the names to import are retrieved and assigned to their respective names.

Footnotes

# data[].attributes.attachment_link.description
The Python interpreter has a number of functions and types built into it that are always available. They are listed here in alphabetical order.,,,, Built-in Functions,,, A, abs(), aiter(), all(), a...

# data[].attributes.attachment_link.readable_text
In this lesson, you will write Python code in Jupyter Notebook to import text data (.txt. and .csv files) into numpy arrays. You will also write Python to download the datasets (.txt. and .csv files) needed for the numpy array lessons.

Learning Objectives

After completing this hands-on exercise, you will be able to:

Explain the differences between plain text and comma delimited files

Write Python code to download data using URLs

Write Python code to import data from text files (.txt) into numpy arrays

What You Need

Be sure that you have completed the previous lessons on Python Variables and Python Lists.

Be sure that you have a subdirectory called data under your earth-analytics-bootcamp directory. For help with this task, please see the challenge for the lesson on Intro to Shell.

The code below is available in the ea-bootcamp-day-4 repository that you cloned to earth-analytics-bootcamp under your home directory.

Text Files

Scientific data can come in a variety of file formats and types. In this course, you will work with data stored in plain text files (.txt) and comma-delimited text files (.csv).

Plain Text Files

Plain text files simply list out the values on separate lines without any symbols or delimiters to indicate separate values. For example, data for the average monthly precipitation data for Boulder, CO can be stored as a plain text file (.txt), with a separate line for each month's value.

Due to their simplicity, text files (.txt) can be very useful for collecting very large datasets that are all the same type of observation or data type.

CSV Files

Unlike plain-text files which simply list out the values on separate lines without any symbols or delimiters, comma delimited (CSV) files use commas (or some other delimiter like tab spaces or semi-colons) to indicate separate values.

This means that CSV files can easily support multiple rows and columns of related data. For example, data for the monthly precipitation for Boulder, CO for the years 2002 and 2013 can be stored together in a comma delimited (.csv) file.

In this lesson and the next lesson, you will use data from:

a .txt file containing the average monthly precipitation data for Boulder, CO

a .csv file containing the monthly precipitation for Boulder, CO for the years 2002 and 2013

Begin Writing Your Code

From previous lessons, you know that you always begin your Python code by importing the necessary packages and checking the working directory.

Import Packages

In this lesson, you will use the os package along with some new packages:

1. numpy with the alias np: to create and work with data as numpy arrays

2. urllib: to download the datasets for this lesson

Set Working Directory

Remember that you can check the current working directory using os.getcwd(). You can also set the current working directory using another useful function os.chdir().

Download Data Using URLs

You can use the urllib package to download data from online sources such as Figshare.com, where the datasets for this lesson are published.

To use urllib, you provide parameter values for url as well as filename for the downloaded file.

For this lesson, you will download the .txt files for average monthly precipitation for Boulder, CO as well as the month names from the Earth Lab Figshare.com repository.

Note that you do not have to provide the full path for filename because it is relative to the current working directory that you set using os.chdir().

Into which directory were your files downloaded? Open the directory on your computer to see your downloaded files.

Import Text Data Into Numpy Arrays

Numeric Data

You can create new numpy arrays by importing data from files, such as text files. You can import these data using the loadtxt() function from numpy, which you imported as np.

For both .txt and .csv files, you need to specify a value for the parameter called fname for the file name (e.g. np.loadtxt(fname = "filename.txt")). Be sure to update the path for the file to your home directory.

Recall that that you can use the print() function to see the values stored in a variable (e.g. print(variablename)).

You can also use the type() function to check the type of data structure (e.g. type(variablename)) and see that avg_monthly_precip is a numpy array.

Text String Data

In addition to numeric data, you can also import text strings to numpy arrays using the genfromtxt() function from numpy. You need to specify a parameter value for filename as well as for the data type as dtype='str'.

Again, you can check the type and the data in your new numpy array.

Congratulations! You have now learned how to create numpy arrays by importing numeric and text string data from text files.

Optional Challenge

Test your Python skills to:

1. Download a .csv file containing the monthly precipitation for Boulder, CO for the years 2002 and 2013 (monthly-precip-2002-2013.csv)from https://ndownloader.figshare.com/files/12707792.
Be sure to assign a useful variable name that is short but indicative of what it contains (e.g. precip_2002_2013).

2. Import the data from this .csv file into a numpy array.
Note that for .csv files, you need to specify another parameter in addition to filename. You need to provide a value for delimiter, which indicates the symbol that is used to separate the values (e.g. delimiter = ",").

3. Print the data type of your new numpy array as well as its contents.
Note that you can add a line of code before print(variablename) to display the values in the numpy array as floats, rather than scientific notation (np.set_printoptions(suppress=True)).

4. Print the data contained in avg_monthly_precip, and compare it to your new numpy array. Do you notice any differences in the structure of the data between these two numpy arrays?

# data[].attributes.attachment_link.description
In this lesson, you will write Python code in Jupyter Notebook to import text data (.txt. and .csv files) into numpy arrays. You will also write Python to download the datasets (.txt. and .csv files) needed for the numpy array lessons.

# data[].attributes.attachment_link.description
This reference manual details functions, modules, and objects included in NumPy, describing what they are and what they do. For learning how to use NumPy, see also NumPy User Guide .